commit aa079d4dc93b5a72935e1616a0431a838adae8a1 Author: Andrew Myers Date: Thu Jan 6 12:26:52 2022 -0500 Add plotfile diff to neighbor particles test. (#2562) Tests/Particles/NeighborParticles/main.cpp commit 98a66afcf4162b5d3539642dd2cd868d661976be Author: Andrew Myers Date: Thu Jan 6 12:24:38 2022 -0500 Add option not to remove particles with negative ids when calling Redistribute. (#2561) This is useful for WarpX when interacting particles with EB walls. The default behavior of Redistribute is not changed by this PR. Src/Particle/AMReX_ParticleContainerI.H Src/Particle/AMReX_ParticleUtil.H Src/Particle/AMReX_Particles.H Tests/Particles/Redistribute/main.cpp commit c34082833c563ff9dab469fb610164ae57dd9a88 Author: Candace Gilet Date: Thu Jan 6 11:49:34 2022 -0500 Update FillPatchTwoLevels for single component face-centered data (#2539) Allow the non-array version of FillPatchTwoLevels to operate on face-centered data and use the valid fine values on the coarse-fine interface during the interpolation to ghost cells. This also changes the FaceLinear Interpolater to use the valid fine data on the coarse-fine interface in the interpolation to ghost cells. Src/AmrCore/AMReX_FillPatchUtil_I.H Src/AmrCore/AMReX_Interp_2D_C.H Src/AmrCore/AMReX_Interp_3D_C.H Src/AmrCore/AMReX_Interp_C.H Src/AmrCore/AMReX_Interpolater.H Src/AmrCore/AMReX_Interpolater.cpp commit 5726371bbb1a9fabb68640db67f3482d78fc33c7 Author: Weiqun Zhang Date: Wed Jan 5 10:06:05 2022 -0800 Kernel fusing in FabArray Comm (#2559) Note that amrex::Add has been moved from AMReX_FabArrayUtility.H to AMReX_FabArray.H so that it can be used by AMReX_FabArrayCommI.H. Src/Base/AMReX_FabArray.H Src/Base/AMReX_FabArrayCommI.H Src/Base/AMReX_FabArrayUtility.H commit dfee5b74765e4ec7f0bc28830514168595c1d148 Author: Weiqun Zhang Date: Tue Jan 4 14:09:35 2022 -0800 Use MF ParallelFor in error tagging (#2558) Src/AmrCore/AMReX_ErrorList.cpp commit 458679dbff56a7ca18d8bbf3994a1a172fa2e5c3 Author: Weiqun Zhang Date: Mon Jan 3 11:50:26 2022 -0800 iMultiFab I/O (#2495) * iMultiFab I/O Add functions for writing and reading iMultiFab. * Update Src/Base/AMReX_VisMF.H Co-authored-by: Andrew Myers Co-authored-by: Andrew Myers Src/Base/AMReX_FArrayBox.H Src/Base/AMReX_FArrayBox.cpp Src/Base/AMReX_IArrayBox.H Src/Base/AMReX_IArrayBox.cpp Src/Base/AMReX_IntConv.H Src/Base/AMReX_IntConv.cpp Src/Base/AMReX_NFiles.H Src/Base/AMReX_NFiles.cpp Src/Base/AMReX_VectorIO.H Src/Base/AMReX_VectorIO.cpp Src/Base/AMReX_VisMF.H Src/Base/AMReX_VisMF.cpp Src/Base/AMReX_VisMFBuffer.H Src/Base/CMakeLists.txt Src/Base/Make.package Src/Extern/HDF5/AMReX_PlotFileUtilHDF5.cpp commit dd75ba5ec7276a494e61b5d53226c88be80b5299 Author: Weiqun Zhang Date: Mon Jan 3 09:39:35 2022 -0800 Embed more EB data in EBFArrayBox (#2550) * Embed more EB data in EBFArrayBox Before this, we only had access to EBCellFlag given an EBFArrayBox. This adds access to more EB data via EBFArrayBox. This could be useful for functions with Fab parameters (e.g., AmrLevel's derive functions). * Mark EBFArrayBox(Arena*) explicit Src/EB/AMReX_EBFArrayBox.H Src/EB/AMReX_EBFArrayBox.cpp Src/EB/AMReX_EBFabFactory.cpp Src/EB/AMReX_MultiCutFab.H Src/EB/AMReX_MultiCutFab.cpp commit d36950d8589cad77558519c00f9fa0f46015d5b5 Author: Weiqun Zhang Date: Mon Jan 3 08:36:20 2022 -0800 CpuBndryFuncFab: Face Data (#2545) For completeness, fab_filfc is implemented for CpuBndryFuncFab. However, one should use GpuBndryFuncFab that works for both GPU and CPU builds. Src/Base/AMReX_FilCC_1D_C.H Src/Base/AMReX_FilCC_2D_C.H Src/Base/AMReX_FilCC_3D_C.H Src/Base/AMReX_FilFC_1D_C.H Src/Base/AMReX_FilFC_2D_C.H Src/Base/AMReX_FilFC_3D_C.H Src/Base/AMReX_FilFC_C.H Src/Base/AMReX_FilFC_C.cpp Src/Base/AMReX_PhysBCFunct.cpp Src/Base/CMakeLists.txt Src/Base/Make.package commit a73ce6759c914bdf15c44e58e839a744b5878be3 Author: Weiqun Zhang Date: Sun Jan 2 14:55:14 2022 -0800 Fix bug in Parser (#2555) Src/Base/Parser/AMReX_IParser_Y.cpp Src/Base/Parser/AMReX_Parser_Y.cpp commit 39c9cb6221959a8761bebf3b372f97dc65635ff3 Author: Weiqun Zhang Date: Sat Jan 1 14:03:53 2022 -0800 Update CHANGES for 22.01 (#2554) CHANGES commit 7c606b85c6bc22e1321727ed6692dd1e591e2b72 Author: Max Katz Date: Thu Dec 30 21:24:14 2021 -0800 Allow CUDA compilation on Perlmutter if only PrgEnv-nvidia is loaded (#2553) In this scenario, there is not a CUDA module loaded by default; nvcc is provided as part of the NVIDIA HPC SDK. In the Cray PE the environment variable defined pointing to the top level of the HPC SDK is `NVIDIA_PATH` so we can use this to indicate to the make system that we should have CUDA support. Tools/GNUMake/sites/Make.nersc commit bcdb6ca488826e10cf28691b0ed2ecc776753d55 Author: kngott Date: Thu Dec 23 19:52:37 2021 -0500 Add Gatherv for USE_MPI=FALSE. (#2549) This adds a Gatherv implementation for when USE_MPI=FALSE. Src/Base/AMReX_ParallelDescriptor.H commit c5f0685554aaff7588d71a535050121dbc8c393f Author: kngott Date: Thu Dec 23 19:51:26 2021 -0500 Resize vector inside GatherLayoutDataToVector (#2548) The output vector of GatherLayoutDataToVector doesn't resize the resulting vector; the user has to pass a minimally sized vector themselves. This PR resizes the result vector inside the function. Src/Base/AMReX_ParallelDescriptor.H commit 29853f0b30252deabb343e0a919b77b480bbba60 Author: kngott Date: Thu Dec 23 15:09:32 2021 -0500 Add missing FillBoundary_nowait (#2546) The `IntVect nghost` function signature doesn't have a matching `nowait` call. This PR adds it. Src/Base/AMReX_FabArray.H commit 24e25124cc179ee7e5aed1a06c3b9800a156595d Author: kngott Date: Thu Dec 23 15:06:33 2021 -0500 Remove unused variable. (#2547) Src/Base/AMReX_FabArrayCommI.H commit 85284ea15a3c5ca7d6d8e3f1045ac1e5a48c6a19 Author: Weiqun Zhang Date: Tue Dec 21 09:48:05 2021 -0800 Fix compilation for 1D build with hypre (#2544) Src/LinearSolvers/MLMG/AMReX_MLEBNodeFDLaplacian.H Src/LinearSolvers/MLMG/AMReX_MLEBNodeFDLaplacian.cpp commit 77ccf474ba9c4123f96be0b56ef08df1985af24a Author: Michael Zingale Date: Tue Dec 21 12:47:40 2021 -0500 add a tool to compute the volume integral of a plotfile field (#2541) Tools/Plotfile/fvolumesum.cpp commit df6893521551ab14b9d87f9610786a42eafcd9d9 Author: Jean Luca Bez Date: Tue Dec 21 08:44:38 2021 -0800 Update read path for HDF5 benchmark (#2542) Update path that was not used after including the directory for read Tests/HDF5Benchmark/main.cpp commit a38ccbc90843d32cf34beb7d89c60d95af3383d3 Author: Don E. Willcox Date: Mon Dec 20 12:26:52 2021 -0800 Eliminate compiler warnings for time integration classes. (#2540) Src/Base/AMReX_FEIntegrator.H Src/Base/AMReX_IntegratorBase.H Src/Base/AMReX_RKIntegrator.H Src/Base/AMReX_TimeIntegrator.H commit e217e4a50525d4f973e1a5ba3c2510c4814cf745 Author: Weiqun Zhang Date: Mon Dec 20 11:50:42 2021 -0800 Update MLEBNodeFDLaplacian (#2538) * Update MLEBNodeFDLaplacian * Make it work for non-EB builds too. * Add option to solve in RZ coordinates. In that case, sigma is always one. * Assert that for 2d RZ with the lo-r boundary at r = 0, the domain BC is set to Neumann. Src/LinearSolvers/CMakeLists.txt Src/LinearSolvers/MLMG/AMReX_MLEBNodeFDLap_1D_K.H Src/LinearSolvers/MLMG/AMReX_MLEBNodeFDLap_2D_K.H Src/LinearSolvers/MLMG/AMReX_MLEBNodeFDLap_K.H Src/LinearSolvers/MLMG/AMReX_MLEBNodeFDLaplacian.H Src/LinearSolvers/MLMG/AMReX_MLEBNodeFDLaplacian.cpp Src/LinearSolvers/MLMG/Make.package commit 134d33e4b7f0c9999ab9cfc2f1c129c47be31cee Author: Andy Nonaka Date: Tue Dec 14 15:38:46 2021 -0800 fix WritePlotfileToASCII compilation (#2537) Co-authored-by: Jean M. Sexton Tools/Postprocessing/C_Src/Make.package commit c6978ac1efb1afbe8625d1cc48e0c7ae68a648fc Author: Houjun Tang Date: Tue Dec 14 12:12:17 2021 -0800 Add missing const (#2536) Src/Extern/HDF5/AMReX_PlotFileUtilHDF5.cpp commit 9373709e34b23add981551d5446bc4810fd3b688 Author: Jean M. Sexton Date: Fri Dec 10 22:24:55 2021 -0800 Fix assert to use IntVect (#2531) Src/Base/AMReX_MultiFabUtil.H commit 8a1f7651a4723574a276f072a963cc5eb2e7dacc Author: Jean M. Sexton Date: Fri Dec 10 21:04:19 2021 -0800 Update NVector_MultiFab functionality from ATPESC-codes and add to Src/Extern/SUNDIALS (#2505) Adds NVector_MultiFab from https://github.com/AMReX-Codes/ATPESC-codes Updates NVector MultiFab wrapper for the GPU, and adds a NormHelper function Adds a check for SUNDIALS_FOUND in the CMake setup of SUNDIALS as a ThirdPartyLibrary Src/Base/AMReX_MultiFabUtil.H Src/Extern/SUNDIALS/AMReX_NVector_MultiFab.H Src/Extern/SUNDIALS/AMReX_NVector_MultiFab.cpp Src/Extern/SUNDIALS/AMReX_Sundials.H Src/Extern/SUNDIALS/CMakeLists.txt Src/Extern/SUNDIALS/Make.package Tools/CMake/AMReXThirdPartyLibraries.cmake commit 55afaaf5945ad4b55483a3fe623f50d1768d9be6 Author: Erik Date: Fri Dec 10 20:13:48 2021 -0500 Projections migrated to AMReX-Hydro (#2530) Projections have been moved to AMReX-Hydro. This PR removes them from AMReX. Docs/sphinx_documentation/source/LinearSolvers.rst GNUmakefile.in Src/LinearSolvers/CMakeLists.txt Src/LinearSolvers/Projections/AMReX_MacProjector.H Src/LinearSolvers/Projections/AMReX_MacProjector.cpp Src/LinearSolvers/Projections/AMReX_NodalProjector.H Src/LinearSolvers/Projections/AMReX_NodalProjector.cpp Src/LinearSolvers/Projections/Make.package Tests/LinearSolvers/MAC_Projection_EB/CMakeLists.txt Tests/LinearSolvers/MAC_Projection_EB/GNUmakefile Tests/LinearSolvers/MAC_Projection_EB/Make.package Tests/LinearSolvers/MAC_Projection_EB/README Tests/LinearSolvers/MAC_Projection_EB/inputs_3d Tests/LinearSolvers/MAC_Projection_EB/main.cpp Tools/RegressionTesting/AMReX-cuda-tests.ini Tools/RegressionTesting/AMReX-hip-tests.ini Tools/RegressionTesting/AMReX-tests.ini commit 50f8e4314be75003610378d159134960ad27e64b Author: Andrew Myers Date: Fri Dec 10 19:13:07 2021 -0500 Add another overload for MakeSimilarDM. (#2528) This one works with an input `ba` and `dm` instead of a `MultiFab.` Src/Base/AMReX_DistributionMapping.H Src/Base/AMReX_DistributionMapping.cpp commit dbbac7b9649ce7feecdd80d7e46b97438fb51173 Author: Don E. Willcox Date: Fri Dec 10 12:45:28 2021 -0800 Adds `get` functions to the TimeIntegrator classes (#2529) * Add get_rhs and get_post_update functions to allow user code to retrieve std::functions for the RHS and post_update. Src/Base/AMReX_IntegratorBase.H Src/Base/AMReX_TimeIntegrator.H commit afe587b071d9f1a0f2b95125d8d3204a01cf8967 Author: Weiqun Zhang Date: Fri Dec 10 09:47:11 2021 -0800 Fix a bug in Robin BC (#2477) * Fix a bug in Robin BC When the a scalar is zero but A coefficient is not, there was a bug in the handling of Robin BC. We should reset the A coefficient to zero first before modifying it for Robin BC because we need to modify the a scalar to non-zero. Closes #2476 * Fix a bug in the detection of singularity. Src/LinearSolvers/MLMG/AMReX_MLABecLaplacian.H Src/LinearSolvers/MLMG/AMReX_MLABecLaplacian.cpp commit 4b695b7b3fe17962bb2645b0b37959adbf38e6ce Author: Erik Date: Fri Dec 10 10:54:25 2021 -0500 Doc refine grid layout (#2527) * CMake file for AmrLevel-SingleVortex * move new test to its own folder * message to indicate where build is coming from * Starting to convert Prob.f90 to cpp. * Convert Prob.f90 to Prob.cpp, runs. Testing next. * remove Adv_F.H from CMake file * Forgot to add the Prob.H file * delete unneeded ctest section * Switch AmrLevel for AmrCore b/c level not WIN32 compatible. * remove WIP comments * remove unneeded changes * Remove unneeded files. * fix words and remove comments * trying to add CUDA to Cmake WIP * some changes to the CMake file, nothing concrete * Add CUDA to CMake file. * add refine_grid_layout to sphinx docs * more add refine_grid_layout to sphinx docs Docs/sphinx_documentation/source/InputsLoadBalancing.rst commit 8d5e756a5a58b1ce08ea4ff69ad8d8e2802e8eb9 Author: Weiqun Zhang Date: Thu Dec 9 16:37:00 2021 -0800 Add ParallelCopyToGhost and fix a bug in FillPatch (#2523) * Add ParallelCopyToGhost function that copies data to ghost cells only. * There was a bug in the Array version of FillPatch for Face data. The issue is, if the destination is the same FabArray as one of the source FabArrays on the fine level, some valid data on valid box faces are overwritten with the interpolated values. This is now fixed with the use of ParallelCopyToGhost in that case. Src/AmrCore/AMReX_FillPatchUtil_I.H Src/Base/AMReX_FabArray.H Src/Base/AMReX_FabArrayBase.H Src/Base/AMReX_FabArrayBase.cpp Src/Base/AMReX_FabArrayCommI.H commit fe08a6914b069739332405eec45a8ca79be57ac0 Author: Axel Huebl Date: Thu Dec 9 16:36:32 2021 -0800 CMake 3.22+: Policy CMP0127 (#2525) Fix a warning with CMake 3.22+. ~~We use use sometimes ;-connected AND syntax in `cmake_dependent_option`, so we need to update some of those options once we require CMake 3.22+ only.~~ We use simple syntax in `cmake_dependent_option`, so we are compatible with the extended syntax in CMake 3.22+ https://cmake.org/cmake/help/v3.22/policy/CMP0127.html First seen on Perlmutter. CMakeLists.txt commit 6b3937d41be4756291747fca2f90cdd50c5afe43 Author: Andrew Myers Date: Thu Dec 9 19:35:45 2021 -0500 Add function for make a dmap that is similar to an input MultiFab. (#2526) This was originally a PR to WarpX, but it makes more sense for it to live in AMReX: https://github.com/ECP-WarpX/WarpX/pull/2640 Src/Base/AMReX_DistributionMapping.H Src/Base/AMReX_DistributionMapping.cpp commit fba4eba1e10f87fc2113d476766e568800c81815 Author: Erik Date: Thu Dec 9 12:16:23 2021 -0500 Specify dimensions that can be refined with `refine_grid_layout` (#2516) Adds a pp variable that can be included in the input file to specify which dimensions can be refined with `refine_grid_layout`. Co-authored-by: Weiqun Zhang Src/AmrCore/AMReX_AmrMesh.H Src/AmrCore/AMReX_AmrMesh.cpp commit e88a72a7c8572ab94a078ddf6736d840bc1dbc6c Author: darylbond Date: Fri Dec 10 02:02:39 2021 +1000 avoid divide by zero in EB (#2519) Add a small number to the variable `apnorm` in the 2D EB code to avoid a divide by zero. Src/EB/AMReX_EB2_2D_C.cpp commit d52e45350f77bb70f051913f794cc205a3c76e97 Author: Luca Fedeli Date: Thu Dec 9 17:01:55 2021 +0100 Fix issue concerning Fujitsu compiler in AMReX_BLBackTrace.H (#2517) The line `#pragma omp threadprivate(bt_stack)` in `AMReX_BLBackTrace.H` can't be compiled with Fujitsu compiler (as well as with Intel compiler and Cray compiler). This PR excludes this line when the Fujitsu compiler is used. Src/Base/AMReX_BLBackTrace.H commit d2bdb9482d43b029da86203c2e6d1580b2f9c742 Author: Erik Date: Thu Dec 9 10:22:27 2021 -0500 Interpolater doxygen (#2524) Co-authored-by: atmyers Src/Particle/AMReX_ParticleInterpolators.H commit 3a7a24d05a94545596cf706a188cf5983f4d0975 Author: Weiqun Zhang Date: Wed Dec 8 15:44:37 2021 -0800 DPCPP: AMREX_INTEL_ARCH (#2459) * DPCPP: AMREX_INTEL_ARCH When AOT is on, set the target to AMREX_INTEL_ARCH, which is required if the SYCL backend is used and AOT is on. * update documentation * Apply suggestions from code review Docs/sphinx_documentation/source/GPU.rst Tools/CMake/AMReXOptions.cmake Tools/CMake/AMReXSYCL.cmake commit f39e091d160cd5c0dcbcf8dc7cc00e3b28e956af Author: Andrew Myers Date: Wed Dec 8 12:27:30 2021 -0500 Doxygen comments for the new interpolator classes (#2521) Src/Particle/AMReX_ParticleInterpolators.H commit acf8694f2b2676a9e0479f3245760663fe46dd3b Author: Andrew Myers Date: Wed Dec 8 11:56:18 2021 -0500 Fix ParticleMesh test after #2475 (#2520) Remove some inadvertent changes to the ParticleMesh test. Tests/Particles/ParticleMesh/main.cpp commit 6780dbf281570c9354ed742ca9c9c4545a0b65d4 Author: asalmgren Date: Tue Dec 7 17:05:31 2021 -0800 Generalize problem size specification (#2518) * Add the option to specify geometry.prob_extent instead of geometry.prob_lo and geometry.prob_hi * add documentation of prob_extent * setting prob_hi should be inside if test * update docs Docs/sphinx_documentation/source/Basics.rst Docs/sphinx_documentation/source/InputsProblemDefinition.rst Src/Base/AMReX_Geometry.cpp commit 84c7864b66247fabdcd0e12665994ea2edd94759 Author: Axel Huebl Date: Tue Dec 7 09:37:43 2021 -0800 CI: Run 2D and 1D Tests (#2475) Co-authored-by: atmyers .github/workflows/gcc.yml Src/Particle/AMReX_ParticleInterpolators.H Src/Particle/CMakeLists.txt Src/Particle/Make.package Tests/AsyncOut/multifab/main.cpp Tests/LinearSolvers/NodalPoisson/CMakeLists.txt Tests/MultiBlock/Advection/CMakeLists.txt Tests/MultiBlock/Advection/main.cpp Tests/MultiBlock/IndexType/CMakeLists.txt Tests/MultiBlock/IndexType/main.cpp Tests/Particles/AssignDensity/main.cpp Tests/Particles/AsyncIO/main.cpp Tests/Particles/GhostsAndVirtuals/CMakeLists.txt Tests/Particles/InitFromAscii/CMakeLists.txt Tests/Particles/NeighborParticles/CheckPair.H Tests/Particles/NeighborParticles/MDParticleContainer.cpp Tests/Particles/ParallelContext/main.cpp Tests/Particles/ParticleIterator/main.cpp Tests/Particles/ParticleMesh/main.cpp Tests/Particles/ParticleMeshMultiLevel/main.cpp Tests/Particles/ParticleMeshMultiLevel/trilinear_deposition_K.H Tests/Particles/ParticleReduce/main.cpp Tests/Particles/ParticleTransformations/main.cpp commit 60fe729fe2ba65ebffc88c0af18743c254d3992c Author: Jean M. Sexton Date: Fri Dec 3 15:52:18 2021 -0800 Use specific targets for make uninstall (#2471) This updates the `make uninstall` rule which is used with configure to more specifically remove targets which were installed. It also adds a dependency file removal for the `make cleanconfig` rule. This removes the assumption that the directory set from `./configure --prefix=/my/example/directory` is empty, and therefore all files should be removed with `make uninstall` Co-authored-by: Weiqun Zhang GNUmakefile.in Tools/GNUMake/Make.defs Tools/GNUMake/Make.rules commit e96ffdead46310595dfa491b1e42ae208017de1d Author: Don E. Willcox Date: Fri Dec 3 14:58:21 2021 -0800 Move Basic Time Integration into AMReX (#2503) This PR moves the latest versions of a basic general explicit time integrator into AMReX. We are already using this time integrator for the Emu (https://github.com/AMReX-Astro/Emu), ET-Integration (https://github.com/amrex-codes/ET-integration), and ERF (https://github.com/erf-model/ERF) codes and this will let us consolidate the code and let other codes can benefit from it. The time integrator includes both forward Euler and several explicit Runge-Kutta methods and example code with input parameter explanations are supplied in a new documentation webpage labeled "Time Integration." Co-authored-by: Jean M. Sexton Co-authored-by: Weiqun Zhang Docs/sphinx_documentation/source/TimeIntegration_Chapter.rst Docs/sphinx_documentation/source/index.rst Src/Base/AMReX_FEIntegrator.H Src/Base/AMReX_IntegratorBase.H Src/Base/AMReX_RKIntegrator.H Src/Base/AMReX_TimeIntegrator.H Src/Base/CMakeLists.txt Src/Base/Make.package commit b6d3d1cced21648126e9465ad88ec3f6cd8b093d Author: Andrew Myers Date: Fri Dec 3 17:44:17 2021 -0500 allow the use of SuperParticle in amrex::ParticleToMesh (#2514) Src/Particle/AMReX_ParticleMesh.H Src/Particle/AMReX_ParticleUtil.H Tests/Particles/ParticleMesh/main.cpp commit f5c1f0371aa7ec2c53053907f856e0b64a6bc623 Author: Nathan X. Marshak <5407450+n8xm@users.noreply.github.com> Date: Fri Dec 3 15:04:01 2021 -0700 Workaround to fix amrex_particles_to_vtp.py samplefiles (#2508) ## Summary The sample input files for `amrex_paticles_to_vtp.py` were modified in order to stop the script from crashing. Based on comparison against output from the [Electrostatic PIC tutorial](https://amrex-codes.github.io/amrex/tutorials_html/Particles_Tutorial.html#electrostaticpic), I believe that the sample input files are in an outdated format. ([electrostatic_pic_particles50000.txt](https://github.com/AMReX-Codes/amrex/files/7637608/electrostatic_pic_particles50000.txt) is attached for reference.) ## Additional background `amrex_particles_to_vtp.py` skips the first four lines after the first line (containing the number of particles) of the input file. Then, the number of lines that are read is equal to exactly the number of particles specified on the first line. The sample input files under the `samplefiles/` directory do not contain these four "skipped" lines, and as a result, not only does `amrex_particles_to_vtp.py` skip four lines of valid particle data, the script crashes because it tries to read past the end of the file. (In other words, the script assumes that the file is four lines longer than it really is.) As a workaround, I've added four "dummy" lines to each file. This is NOT an ideal solution because real particle output would not contain my four "dummy" lines, and instead there would be some other information there (i.e. I believe the number of extra real and int components in the particle data/attributes). However, my workaround is adequate for the purpose of demonstrating correct usage of `amrex_particles_to_vtp.py`. I have confirmed that the resulting `.vtp` files produce sensible-looking results in ParaView. (See attached screenshots below for reference.) `particles00000`: ![particles00000](https://user-images.githubusercontent.com/5407450/144328022-71b847a8-d78a-4837-81dc-ea8d1cafd682.gif) `particles00001`: ![particles00001](https://user-images.githubusercontent.com/5407450/144328021-34077dda-fe6b-40b0-8849-4265bb4539f9.gif) `particles00002`: ![particles00002](https://user-images.githubusercontent.com/5407450/144328017-504b1158-b38c-49e9-924b-957ecc9a0fbd.gif) Co-authored-by: Nathan Marshak Tools/Py_util/amrex_particles_to_vtp/samplefiles/particles00000 Tools/Py_util/amrex_particles_to_vtp/samplefiles/particles00001 Tools/Py_util/amrex_particles_to_vtp/samplefiles/particles00002 commit 71b6368a8f3d807ce0c70d38c7ef3791bd5c431d Author: Andrew Myers Date: Fri Dec 3 15:21:36 2021 -0500 Fix AppleClang CI by ignoring unused variables in GccPlacater (#2513) Src/Base/AMReX.H Src/Base/AMReX_MultiFab.H commit e2af96c28d4e50ecac8476cbb82f048f8e82efe5 Author: Weiqun Zhang Date: Fri Dec 3 08:08:29 2021 -0800 ANSI Escape Codes (#2509) Add commonly used ANSI escape codes to AMReX_ANSIEscCode.H. They can be used to print colored output. For example, amrex::Print() << Font::Bold << FGColor::Red << "Hello world from AMReX\n" << ResetDisplay << Font::Faint << FGColor::BrightCyan << Font::Underline << "Hello world again from AMReX " << ResetDisplay << BGColor::White << FGColor::Black << amrex::Version() << ResetDisplay << std::endl; Src/Base/AMReX_ANSIEscCode.H Src/Base/AMReX_Print.H Src/Base/CMakeLists.txt Src/Base/Make.package commit 2c307a8276ff0ad5f7636544ac7c191ffd6f6365 Author: Andrew Myers Date: Thu Dec 2 13:25:26 2021 -0500 Fix misleading documentation for ParticleTile. (#2500) Co-authored-by: etpalmer63 Docs/sphinx_documentation/source/Particle.rst commit 6396050dde3f4fb08ef65173a52350c7b33adca9 Author: Weiqun Zhang Date: Wed Dec 1 10:22:25 2021 -0800 Update CHANGES for 21.12 (#2507) CHANGES commit ef9a37d6a14a704d59f105945c203da717595cb5 Author: Weiqun Zhang Date: Tue Nov 30 18:51:24 2021 -0800 AMREX_RELEASE_NUMBER (#2448) Add a new macro `AMREX_RELEASE_NUMBER`. It is a six digits number. The first two are for the year, and the next two are for the month. The last two are usually 00, unless a patch other than the monthly release is released. If the AMReX source is a git version that's not the release version, the last release number will be used. For git version obtained with git --describe, we define something like #define AMREX_GIT_VERSION "21.10-65-gfbb2edc95551-dirty" #define AMREX_RELEASE_NUMBER 211000 For git version obtained from the CHANGES file, we define something like #define AMREX_GIT_VERSION "21.10" #define AMREX_RELEASE_NUMBER 211000 or #define AMREX_GIT_VERSION "21.10.1" #define AMREX_RELEASE_NUMBER 2110001 or #define AMREX_GIT_VERSION "21.10.11" #define AMREX_RELEASE_NUMBER 211011 Tools/CMake/AMReXConfig.cmake.in Tools/CMake/AMReXSetDefines.cmake Tools/CMake/AMReXUtils.cmake Tools/CMake/AMReX_Config.H.in Tools/GNUMake/Make.defs commit 1d50bfc1c1da79614c853f9823efdf33f9003cc5 Author: Andrew Myers Date: Tue Nov 30 16:53:45 2021 -0500 Add a function for detecting whether amrex::Initialize has been called or not. (#2501) Src/Base/AMReX.H Src/Base/AMReX.cpp commit 062a07100e364b25d29a966092fd9e9200d4f2b0 Author: Weiqun Zhang Date: Mon Nov 29 18:17:47 2021 -0800 Fix fcompare's --abs_tol (#2504) There is a bug that ignores the absolute errors of components that do not have the biggest relative error when --abs_tol is used. It results in something like below fcompare.gnu.ex --abs_tol 1.e-12 plt00010 plt00002 variable name absolute error relative error (||A - B||) (||A - B||/||A||) ---------------------------------------------------------------------------- level = 0 density 0 0 pressure 0 0 theta 0 0 x_velocity 4.39990436e-05 2.749038203e-06 y_velocity 4.818762188e-05 9.964503154e-06 z_velocity 4.70120484e-19 0.7986458289 PLOTFILE AGREE to specified tolerances: absolute = 1e-12 relative = 0 In this commit, this is fixed by applying the test to each component separately. A component passes if its relative error or absolute error satisfies the respective tolerance. The whole comparison passes if all components pass. Tools/Plotfile/fcompare.cpp commit d62f0d27c907324065fe4968b6dfbbb98d41fd64 Author: Andrew Myers Date: Mon Nov 29 14:13:58 2021 -0500 Fix HIP CI (#2502) Something was going wrong with the "computed default" cmake argument on this action, so this removes it. .github/workflows/hip.yml commit d32658dbbccf1e5abd82cb971bb9fec8fd87eeb1 Author: Houjun Tang Date: Mon Nov 29 07:08:22 2021 -0800 Fix an issue when some MPI ranks have no box at all (#2494) Src/Extern/HDF5/AMReX_PlotFileUtilHDF5.cpp Tests/HDF5Benchmark/main.cpp commit 591c7856b7c876ec9f1f3a5db56ce652059c7c8b Author: Weiqun Zhang Date: Wed Nov 24 07:25:57 2021 -0800 Signed distance function in 2D (#2490) Function for computing signed distance function in 2D. Src/Base/AMReX_Scan.H Src/EB/AMReX_EB_utils.H Src/EB/AMReX_EB_utils.cpp commit 30185629176596663f9d2ee21bfd97ae4d9546b2 Author: hengjiew <86926839+hengjiew@users.noreply.github.com> Date: Mon Nov 22 14:20:15 2021 -0600 Reduce ghost particles in particle communication. (#2480) * Remove duplicated particle communication when boxes are not perfectly aligned * Add a function to select the ghost particles that are neighbors to the real particles And allow `updateNeighbors()` to only communicate these ghost neighbor particles. These two steps significant reduce the particle communication time when particles are dense. * Fix CI issues. * Fix coding style issues. Co-authored-by: Hengjie Wang Src/Particle/AMReX_NeighborParticles.H Src/Particle/AMReX_NeighborParticlesGPUImpl.H Src/Particle/AMReX_NeighborParticlesI.H commit 29584112a13ab086dc64edfdc99a217431ed8db7 Author: Andrew Myers Date: Sun Nov 21 10:43:12 2021 -0800 Assign initial value to prevent (likely spurious) -Wmaybe_unitialized compiler warning. (#2484) Src/Particle/AMReX_Particle.H commit 80156569892316cf413b072c3bc4a63abff28309 Author: Weiqun Zhang Date: Sun Nov 21 10:42:43 2021 -0800 Fix a corner case in fixing small cells. (#2489) During the iteration of fixing small cells, the edge types and level sets could become inconsistent if the fixed small cells are too close to periodic boundaries. The function that updates the edge types did not take that into account. This commit fixes that. Src/EB/AMReX_EB2_GeometryShop.H commit 96ba8b11befcd5361fdf70b58a9fb6e6eb5776ca Author: Candace Gilet Date: Wed Nov 17 18:39:52 2021 -0500 Fix out-of-date comment. (#2483) Src/Base/AMReX_ParmParse.H commit bf86e02eb5048c201b7014c4efa567ed3f49a1d8 Author: Erik Date: Wed Nov 17 18:39:19 2021 -0500 Remove HeatEqn Tutorial and link to new location in amrex-tutorials (#2482) Remove the Heat Equation Tutorial from the AMReX user's guide. Add link to AMReX tutorials in its place. Note, the tutorial already exists in amrex-tutorials. Docs/sphinx_documentation/source/Basics.rst commit 62774a08663f6552a5d6265200f6d52e18df824b Author: Erik Date: Wed Nov 17 18:37:48 2021 -0500 Combine debugging sections into a single chapter. (#2481) This change combines the debugging and Gpu debugging sections into a single debugging chapter. Docs/sphinx_documentation/source/Basics.rst Docs/sphinx_documentation/source/Debugging.rst Docs/sphinx_documentation/source/GPU.rst Docs/sphinx_documentation/source/index.rst commit a2db70cb3cb5e74ee692ed1b568727dd66aa145d Author: David Grote Date: Wed Nov 17 15:35:50 2021 -0800 Added special case for Mac when using rpath (#2479) Tools/GNUMake/Make.defs commit 88b868099018c191e7d3ea424d3e3420b826dd69 Author: Weiqun Zhang Date: Fri Nov 12 12:42:17 2021 -0800 Fix Advection_AmrLevel test (#2474) There is no need to use pinned memory in initData anymore after #2433. Tests/Amr/Advection_AmrLevel/Source/AmrLevelAdv.cpp commit 6ff068d5d04218520fb1408b01ea734b5bca40e6 Author: Erik Date: Fri Nov 12 11:19:48 2021 -0500 Advection_AmrLevel Test: Convert Prob.f90 to Prob.cpp (#2433) Converts the only Fortran file in the AmrLevel test to cpp. Tests/Amr/Advection_AmrLevel/CMakeLists.txt Tests/Amr/Advection_AmrLevel/Exec/Make.Adv Tests/Amr/Advection_AmrLevel/Exec/SingleVortex/Make.package Tests/Amr/Advection_AmrLevel/Exec/SingleVortex/Prob.H Tests/Amr/Advection_AmrLevel/Exec/SingleVortex/Prob.cpp Tests/Amr/Advection_AmrLevel/Exec/SingleVortex/Prob.f90 Tests/Amr/Advection_AmrLevel/Exec/UniformVelocity/Make.package Tests/Amr/Advection_AmrLevel/Exec/UniformVelocity/Prob.H Tests/Amr/Advection_AmrLevel/Exec/UniformVelocity/Prob.cpp Tests/Amr/Advection_AmrLevel/Exec/UniformVelocity/Prob.f90 Tests/Amr/Advection_AmrLevel/README Tests/Amr/Advection_AmrLevel/Source/Adv_F.H Tests/Amr/Advection_AmrLevel/Source/AmrLevelAdv.cpp Tests/Amr/Advection_AmrLevel/Source/Make.package commit 37d1af793b1ec9c20d4d9ba1c176b5936c3e01a3 Author: Weiqun Zhang Date: Thu Nov 11 16:53:31 2021 -0800 Fix multi-level EB nodal solver solvability fix (#2473) For the multi-level EB nodal solver with a singular matrix, we restrict the fine level RHS to the coarse level, and fix the coarse level's solvability by making the total sum of RHS is zero. The restriction is done with the stencil instead of simply geometric restriction because we need to take the EB into account. However, the previous way is incorrect if the sigma coefficient is not a constant, because the stencil includes sigma. But the restriction should be based on EB information only. In this commit, we fix this by constructing a special constant sigma stencil. We use this new stencil for the restriction of RHS, and for consistence the residual as well. Because we used to use the original stencil for the restriction of residual, this commit will change the answer at the tolerance level for multi-level problems. Src/LinearSolvers/MLMG/AMReX_MLNodeLaplacian.H Src/LinearSolvers/MLMG/AMReX_MLNodeLaplacian.cpp Src/LinearSolvers/MLMG/AMReX_MLNodeLaplacian_sten.cpp Src/LinearSolvers/MLMG/AMReX_MLNodeLaplacian_sync.cpp commit a2245756e64c1f414f35b64330fec99a9b87df0c Author: Weiqun Zhang Date: Mon Nov 8 12:36:09 2021 -0800 Disable warning on atomicAddNoRet in HIP (#2466) It seems that we may continue using the deprecated atomicAddNoRet in the near future. This disables the compiler warning. Src/Base/AMReX_GpuAtomic.H commit 798240684d9696fb00392688e3c216e21596b921 Author: Weiqun Zhang Date: Mon Nov 8 08:38:53 2021 -0800 DPCPP: workaround for the scan issue (#2462) A multi-pass algorithm is implemented. Extensive testing on early hardware shows it works for int, but there are still issues with long. Src/Base/AMReX_Scan.H commit c49b1834aa488e123cb379681577a8cf65fed68e Author: Weiqun Zhang Date: Mon Nov 8 08:28:56 2021 -0800 DPCPP: AOT in GNU Make (#2463) If INTEL_ARCH or AMREX_INTEL_ARCH is defined, use it to set target when AOT is on. Tools/GNUMake/comps/dpcpp.mak commit 3d29cc3234a319f5721da38635f37b3570da43c7 Author: Jean Luca Bez Date: Sun Nov 7 14:17:56 2021 -0800 Include optional directory to HDF5 Test Benchmark (#2461) Tests/HDF5Benchmark/inputs Tests/HDF5Benchmark/main.cpp commit 2fc0e4446aa466247de8b7efae11903cfd3b5c37 Author: Pastuzo <40230415+DUGKS@users.noreply.github.com> Date: Sun Nov 7 02:17:17 2021 +0800 Implement HOEXTRAPCC for more than single ghost cell (#2456) Implement HOEXTRAPCC for N ghost cells, N > 1. Previous HOEXTRAPCC works only for N = 1. Src/Base/AMReX_FilCC_1D_C.H Src/Base/AMReX_FilCC_2D_C.H Src/Base/AMReX_FilCC_3D_C.H Src/Base/AMReX_FilCC_C.cpp Src/Base/AMReX_filcc_mod.F90 commit d4669a1ce4c7abb8c15af7829b6d4f56075589a9 Author: Erik Date: Fri Nov 5 17:00:06 2021 -0400 Spack Smoke Test (#2460) Tests/SpackSmokeTest/CMakeLists.txt commit 39332fbb07478fe685c9917006df0fef1957280b Author: Weiqun Zhang Date: Fri Nov 5 10:19:11 2021 -0700 Support fmod in Parser (#2395) * Support fmod in Parser * Sneak in some minor changes Docs/sphinx_documentation/source/AmrCore.rst Docs/sphinx_documentation/source/Basics.rst Src/Base/Parser/AMReX_Parser_Y.H Src/Base/Parser/AMReX_Parser_Y.cpp Src/Base/Parser/amrex_parser.l Src/Base/Parser/amrex_parser.lex.cpp commit b0b7900500bc9879aefd8b649caf0346976bea50 Author: Weiqun Zhang Date: Wed Nov 3 16:25:11 2021 -0700 Fix a bug in BoxArray for SyncRegister (#2458) A bug was introduced recently in #2452. I forgot that code was used by SyncRegister in IAMR. Src/Base/AMReX_BoxArray.H commit 8023fd148ee0f23ae83127b8d0515c1323c194de Author: Scot Breitenfeld Date: Wed Nov 3 12:16:53 2021 -0500 Redo of PR #2450 (#2457) Src/Extern/HDF5/AMReX_PlotFileUtilHDF5.cpp Src/Extern/HDF5/AMReX_WriteBinaryParticleDataHDF5.H commit 2ec7459dd689d1425ce45a5977f26ae7e3c9facb Author: Jean M. Sexton Date: Tue Nov 2 16:04:36 2021 -0700 HIP: ftrapv causes issues with DEBUG linking (#2455) Tools/GNUMake/comps/hip.mak commit d491ebe1c1aca7dc53558e64af24526888b79abe Author: Weiqun Zhang Date: Tue Nov 2 15:04:14 2021 -0700 DPCPP: Enable early optimizations (#2454) Early optimizations were disabled after beta9 was released to work around a compiler bug. The issue has been resolved. So we enable it again. Tools/CMake/AMReXSYCL.cmake Tools/GNUMake/comps/dpcpp.mak commit 7555541ab3c43f2c46b26b25e42a40d73fdf3656 Author: Houjun Tang Date: Tue Nov 2 14:15:07 2021 -0700 New HDF5 schema with separate datasets for different variables (#2432) Src/Base/AMReX_PlotFileUtil.H Src/Base/AMReX_PlotFileUtil.cpp Src/CMakeLists.txt Src/Extern/HDF5/AMReX_ParticleHDF5.H Src/Extern/HDF5/AMReX_ParticleUtilHDF5.H Src/Extern/HDF5/AMReX_ParticlesHDF5.H Src/Extern/HDF5/AMReX_PlotFileUtilHDF5.H Src/Extern/HDF5/AMReX_PlotFileUtilHDF5.cpp Src/Extern/HDF5/AMReX_WriteBinaryParticleDataHDF5.H Src/Extern/HDF5/CMakeLists.txt Src/Extern/HDF5/Make.package Src/Particle/AMReX_ParticleUtil.cpp Src/Particle/AMReX_Particles.H Src/Particle/AMReX_WriteBinaryParticleData.H Src/Particle/CMakeLists.txt Src/Particle/Make.package Tests/HDF5Benchmark/GNUmakefile Tests/HDF5Benchmark/inputs Tests/HDF5Benchmark/main.cpp Tools/CMake/AMReXConfig.cmake.in Tools/GNUMake/packages/Make.hdf5 commit aca6926e0a85a0fad4087bd47016edad0885252f Author: Weiqun Zhang Date: Mon Nov 1 15:15:35 2021 -0700 InterpFaceRegister (#2452) This is a new class for interpolating from coarse to fine at the coarse/fine boundary faces. It also provides masks that can be used to test if a face is at coarse/fine boundary. Src/AmrCore/AMReX_AmrCoreFwd.H Src/AmrCore/AMReX_InterpFaceReg_1D_C.H Src/AmrCore/AMReX_InterpFaceReg_2D_C.H Src/AmrCore/AMReX_InterpFaceReg_3D_C.H Src/AmrCore/AMReX_InterpFaceReg_C.H Src/AmrCore/AMReX_InterpFaceRegister.H Src/AmrCore/AMReX_InterpFaceRegister.cpp Src/AmrCore/CMakeLists.txt Src/AmrCore/Make.package Src/Base/AMReX_BoxArray.H commit 3add1f2824c5a4f33e77abc612da8a52b3f5131b Author: Weiqun Zhang Date: Mon Nov 1 13:59:36 2021 -0700 Fix WarpX EB solver's grad phi computation (#2453) Must set the Dirichlet mask on covered nodes to a negative value, because the function that computes the gradient relies on the information to detect EB. This is a bug introduced recently in #2421. Src/LinearSolvers/MLMG/AMReX_MLEBNodeFDLaplacian.cpp commit 500ab4fb3ea7a755e0e0b29c7c021495fc0707b5 Author: Weiqun Zhang Date: Mon Nov 1 10:03:45 2021 -0700 Update CHANGES for 21.11 (#2451) CHANGES commit 2cfe8354f6c6a00c7a6bb2bcefd822d59fd65518 Author: Scot Breitenfeld Date: Sat Oct 30 00:05:44 2021 -0500 Set to never write fill values to the dataset for HDF5 output (#2450) Src/Base/AMReX_PlotFileUtil.cpp Src/Particle/AMReX_WriteBinaryParticleData.H commit 6e9788139d8e1401ed304fc56e526a1bdb315b81 Author: Weiqun Zhang Date: Thu Oct 28 13:36:13 2021 -0700 Fix a new bug in GNU Make (#2447) CXX should not be set to mpiicpc if it's not an MPI build. This is a new bug introduced by me in #2435. Tools/GNUMake/sites/Make.unknown commit 210cb0e7d77d1334d009246a4cb403ae463df1dc Author: Weiqun Zhang Date: Thu Oct 28 12:54:03 2021 -0700 Lower Hypre minimum version (#2446) Lower Hypre minimum version to 2.20. This will allow users to use the provided installation on summit. The ILU smoother that requires 2.21 is still available for hypre >= 2.21. Src/Extern/HYPRE/AMReX_HypreIJIface.cpp Tools/CMake/AMReXConfig.cmake.in Tools/CMake/AMReXThirdPartyLibraries.cmake commit 6c30ddf8fb0d1a4373b013517d43734ec4110385 Author: Axel Huebl Date: Thu Oct 28 11:28:13 2021 -0700 ROCm 4.5: Dev Packages (#2445) * ROCm 4.5: Devel Packages In ROCm 4.5, libraries were split into binary and `-devel` packages, as often done in Debian/Ubuntu packaging. We need the `-devel` packages for headers. Release notes: https://rocmdocs.amd.com/en/latest/Current_Release_Notes/Current-Release-Notes.html * Robust: Try while rollout is ongoing * What if the docs lie? Try `-dev` * Clean Up .github/workflows/dependencies/dependencies_hip.sh commit 138503ba8476c977de97628bc5bdbdd5fbb00356 Author: Weiqun Zhang Date: Thu Oct 28 08:44:46 2021 -0700 Fix use-after-free (#2443) The parallel copy meta-data cache is store in std::multimap. A CPC* might be stored multiple times in the multimap, and we need to delete the pointers before clearing the multimap. The previous way of avoiding double free was incorrect, because it had a use-after-free. In this commit, we fix it by pushing a unique copy of the pointers to a vector first while iterating the multimap. This bug only manifests if a user's code has a memory leak of MultiFab/FabArray. Src/Base/AMReX_FabArrayBase.cpp commit 08a068a638a124c7dd9e25f28a5ac02ec59fe4ae Author: Jean M. Sexton Date: Wed Oct 27 15:04:42 2021 -0700 Update MPI config checking to include Intel MPI wrappers for DPCPP (#2435) * Update MPI config checking to assume first Intel MPI wrappers, and then others * move the mpicpc check out of the usual MPI check Co-authored-by: Weiqun Zhang Tools/GNUMake/Make.defs Tools/GNUMake/sites/Make.unknown commit 8c8004c07a24dc1e20e5ac6c8e6bc07fdb0ef5a7 Author: Andrew Myers Date: Wed Oct 27 13:53:17 2021 -0700 Mark NeighborParticles as a 'selfTest' (#2442) Tools/RegressionTesting/AMReX-cuda-tests.ini Tools/RegressionTesting/AMReX-hip-tests.ini commit 68e2913f4783d3dc5fe9bc140bbaca06ff8c07d9 Author: Weiqun Zhang Date: Wed Oct 27 12:14:41 2021 -0700 GNU Make: Fix rpath when LIBRARY_LOCATION is empty. (#2441) Because of #2439, empty LIBRARY_LOCATION would result in -Xlinker -rpath='' We would like to avoid that. Tools/GNUMake/Make.defs commit 58d04c6195c0f2c47119315b37d5e99f69fe0898 Author: Weiqun Zhang Date: Wed Oct 27 12:14:09 2021 -0700 Add EBCellFlagFab::getNumRegularCells, getNumCutCells and getNumCoveredCells (#2422) Give a Box, these functions will return the number of regular, cut and covered cells, respectively. Internally, the results are cached. Src/EB/AMReX_EBCellFlag.H Src/EB/AMReX_EBCellFlag.cpp commit 4fa06516928ad07736cb537ddb58d8c4ecba5a4e Author: Neïl Zaim <49716072+NeilZaim@users.noreply.github.com> Date: Wed Oct 27 20:51:17 2021 +0200 ExclusiveSum on GPU: return 0 when size of pointer is 0 (#2440) Src/Base/AMReX_Scan.H commit 94b4764f098603c51bda652d78ea0dff8bee3635 Author: Weiqun Zhang Date: Wed Oct 27 10:40:56 2021 -0700 Multi-component overset solver (#2429) Modify cell-centered overset solver's constructor to allow for multiple components. Src/LinearSolvers/MLMG/AMReX_MLABecLaplacian.H Src/LinearSolvers/MLMG/AMReX_MLABecLaplacian.cpp commit 67e9aadcace3572ca840d8d276d67501663ead8b Author: Weiqun Zhang Date: Wed Oct 27 10:38:30 2021 -0700 Use -rpath=dir instead of -rpath dir (#2439) The latest dpcpp compiler does not like the latter. Tools/GNUMake/Make.defs commit 0a98df6746fcdbb51f133e1905b08d9ae75e99e9 Author: Weiqun Zhang Date: Tue Oct 26 21:13:49 2021 -0700 DPCPP: Use -g1 for non-debug build (#2438) Because `-g` makes link extremely slow, we have disabled that in GNU make since oneAPI beta6. It turns out we could still use `-g1` to get some basic debug information without slowing down link too much. Tools/GNUMake/comps/dpcpp.mak commit ce057cf7c225badbbae7f964e81f1e67745238a4 Author: Weiqun Zhang Date: Tue Oct 26 17:31:49 2021 -0700 DPCPP: Fix warning on tautological-constant-compare (#2436) Disable warning on tautological constant compare that has appeared since 2021.04. .github/workflows/intel.yml Tools/CMake/AMReXSYCL.cmake Tools/GNUMake/comps/dpcpp.mak commit 6b178313793544957f7d0c23ae7999cacead69bd Author: Weiqun Zhang Date: Tue Oct 26 17:24:03 2021 -0700 Update Tools/Postprocessing/C_Src (#2437) Tools/Postprocessing/C_Src/GNUmakefile Tools/Postprocessing/C_Src/HorizontalAvg.cpp Tools/Postprocessing/C_Src/PtwisePltTransform.cpp Tools/Postprocessing/C_Src/particle_compare.cpp commit 39da1abd86b46c64e098c43c3123fcff6c58742d Author: Axel Huebl Date: Mon Oct 25 18:02:00 2021 -0700 Tests: Safe-Guard Module Dirs (#2434) Make sure mod_file dirs are per executable. Avoid potential race conditions if people use the `setup_test` in loops in funny ways. Tests/CMakeLists.txt commit af8429793bc50cf7d5914104c05ed825dd0948c8 Author: Weiqun Zhang Date: Mon Oct 25 10:11:33 2021 -0700 Remove deprecated BaseUmap (#2425) Src/Base/AMReX_BaseUmap.H Src/Base/AMReX_BaseUmap_f.H Src/Base/AMReX_BaseUmap_nd.f90 Src/Base/Make.package commit 652af10ae992e2cf5c1812e0908d9b88ac22126c Author: Weiqun Zhang Date: Mon Oct 25 10:04:32 2021 -0700 Update WarpX EB solver (#2421) Change the coarsening strategy for much better performance. Src/EB/AMReX_EB2_2D_C.H Src/LinearSolvers/MLMG/AMReX_MLEBNodeFDLap_2D_K.H Src/LinearSolvers/MLMG/AMReX_MLEBNodeFDLap_3D_K.H Src/LinearSolvers/MLMG/AMReX_MLEBNodeFDLaplacian.H Src/LinearSolvers/MLMG/AMReX_MLEBNodeFDLaplacian.cpp Src/LinearSolvers/MLMG/AMReX_MLLinOp.H Src/LinearSolvers/MLMG/AMReX_MLMG.cpp commit 614b9ebbbf04eb5b4532029ff4a7ae40e6663042 Author: Weiqun Zhang Date: Fri Oct 22 14:59:39 2021 -0700 MLNodeLaplacian: Fix max coarsening level for enclosed EB (#2431) Where the domain is fully enclosed by EB, we need to make sure that it does not coarsen too much that there are no valid nodes left. Src/LinearSolvers/MLMG/AMReX_MLNodeLaplacian.H Src/LinearSolvers/MLMG/AMReX_MLNodeLaplacian.cpp Src/LinearSolvers/MLMG/AMReX_MLNodeLaplacian_sten.cpp commit 6562da035155d88663c6c98de5f9575c750a3be9 Author: Andrew Myers Date: Fri Oct 22 12:07:49 2021 -0700 Fix two IO overflow bugs uncovered by NyX. (#2417) Co-authored-by: Jean M. Sexton Src/Base/AMReX_FabConv.cpp Src/Particle/AMReX_WriteBinaryParticleData.H commit a707944d287a320e184768d85b5bb05b020f5545 Author: Weiqun Zhang Date: Fri Oct 22 12:03:19 2021 -0700 Cleanup of amrdata (#2430) * Avoid using std and amrex in amrdata headers. * Fix some warnings. Src/Extern/amrdata/AMReX_AmrData.H Src/Extern/amrdata/AMReX_AmrData.cpp Src/Extern/amrdata/AMReX_DataServices.H Src/Extern/amrdata/AMReX_DataServices.cpp Src/Extern/amrdata/AMReX_WritePlotFile.cpp Src/Extern/amrdata/AMReX_XYPlotDataList.H Src/Extern/amrdata/AMReX_XYPlotDataList.cpp commit 2d02c17651b81c3bb7d90a45c81265fdb1706c1b Author: Erik Date: Thu Oct 21 19:30:51 2021 -0400 Set options to allow other Doxygen docs to link to AMReX. (#2424) * Set options to allow other Doxygen docs to link to AMReX. * move amrex xml tag file to folder for uploading to webpage. Docs/Doxygen/doxygen.conf build_docs.sh commit f75e129df118cf448bc763d9eafd67c3cbb74825 Author: Weiqun Zhang Date: Thu Oct 21 15:57:51 2021 -0700 Remove DArena and EArena (#2426) DArena that uses the buddy memory allocation strategy and EArena that uses the best fit strategy are experimental features we explored in the past. But they have never been used because they do not work as well as the first bit based CArena. There is no reason to keep them anymore. Src/Base/AMReX_Arena.cpp Src/Base/AMReX_DArena.H Src/Base/AMReX_DArena.cpp Src/Base/AMReX_EArena.H Src/Base/AMReX_EArena.cpp Src/Base/CMakeLists.txt Src/Base/Make.package commit 8a1a3023b6f0bbc715935fcabbcddbfe9ff95861 Author: Andrew Myers Date: Thu Oct 21 15:55:51 2021 -0700 Add some docstrings for the particle tile access functions. (#2423) Src/Particle/AMReX_Particles.H commit 7c88a2c139eada6601401ac23e6f0b778fab9a9c Author: Weiqun Zhang Date: Thu Oct 21 14:20:57 2021 -0700 Update MLNodeTensorLap for WarpX (#2416) Change the stencil of the linear operator so that L=D*G. This could make the solution satisfy div E = rho up to solver tolerance. Previously, the stencil was similar to that of the nodal solver for approximate projection Src/EB/AMReX_EB2_2D_C.cpp Src/LinearSolvers/MLMG/AMReX_MLNodeLinOp.H Src/LinearSolvers/MLMG/AMReX_MLNodeTensorLap_1D_K.H Src/LinearSolvers/MLMG/AMReX_MLNodeTensorLap_2D_K.H Src/LinearSolvers/MLMG/AMReX_MLNodeTensorLap_3D_K.H Src/LinearSolvers/MLMG/AMReX_MLNodeTensorLaplacian.H Src/LinearSolvers/MLMG/AMReX_MLNodeTensorLaplacian.cpp Tests/LinearSolvers/NodeTensorLap/MyTest.cpp commit db792398f6da8158026527759d97b5b882d1f22b Author: Andrew Myers Date: Wed Oct 20 12:30:36 2021 -0700 Restore RedistributeLocal() call to InitParticles in redistribute test (removed in #2414) (#2420) Tests/Particles/Redistribute/main.cpp commit cdc1052e9d5f518aa864c501a410ed6fbae56afe Author: Andrew Myers Date: Tue Oct 19 12:20:38 2021 -0700 Revert changes to Redistribute test (follow on to #2414) (#2415) Tests/Particles/Redistribute/inputs Tests/Particles/Redistribute/inputs.rt.cuda Tests/Particles/Redistribute/main.cpp commit a9d758ab6dc019203ae467c185553f3dc99b4e8c Author: Andrew Myers Date: Tue Oct 19 10:51:16 2021 -0700 Reduce peak memory usage when sorting particles. (#2414) Src/Particle/AMReX_ArrayOfStructs.H Src/Particle/AMReX_ParticleContainerBase.H Src/Particle/AMReX_ParticleContainerBase.cpp Src/Particle/AMReX_ParticleContainerI.H Tests/Particles/Redistribute/inputs Tests/Particles/Redistribute/inputs.rt.cuda Tests/Particles/Redistribute/main.cpp commit 130f7304328ed1c26a72cea24dd3e96bc9426a58 Author: Candace Gilet Date: Tue Oct 19 12:24:45 2021 -0400 Add link to 'small cell problem' section that got moved to (#2412) Add link to 'small cell problem' section that got moved to AMReX-Hydro's documentation. Also add more details on EB data. Docs/sphinx_documentation/source/EB.rst Docs/sphinx_documentation/source/EB_Chapter.rst Docs/sphinx_documentation/source/conf.py commit ff747d901ea2127ed01d6053f1d80e9a45bcd381 Author: Roelof Groenewald <40245517+roelof-groenewald@users.noreply.github.com> Date: Tue Oct 19 09:23:26 2021 -0700 Docs: MLMG solver convergence criteria (#2413) expanded the documentation with regards to convergence criteria for the MLMG solver Docs/sphinx_documentation/source/LinearSolvers.rst commit 691d9bf3fb64343a3a929d12f1463fde165ec969 Author: Wileam Y. Phan <50928756+wyphan@users.noreply.github.com> Date: Tue Oct 19 12:22:38 2021 -0400 Extend CellQuadratic Interpolater to 3D (#2409) Co-authored-by: Weiqun Zhang Src/AmrCore/AMReX_Interpolater.cpp Src/AmrCore/AMReX_MFInterp_2D_C.H Src/AmrCore/AMReX_MFInterp_3D_C.H Src/AmrCore/AMReX_MFInterp_C.H commit 09a0ec56378f9f6a5f68f3ac025318af746432d5 Author: Wileam Y. Phan <50928756+wyphan@users.noreply.github.com> Date: Mon Oct 18 11:22:55 2021 -0400 Update documentation on Interpolaters (#2408) Docs/sphinx_documentation/source/AmrCore.rst commit be5294eefb2017cb0c5273054207ea1cb6d77807 Author: Weiqun Zhang Date: Thu Oct 14 16:13:05 2021 -0700 Fix 2D bug in EB2::SplineIF (#2407) Src/EB/AMReX_EB2_IF_Spline.H commit 92b4e5bc5d00b1f8c11c0eedfeaeb99ebbe38bbe Author: Weiqun Zhang Date: Thu Oct 14 16:12:34 2021 -0700 Bump up Hypre minimum version to 2.21.0 (#2406) Tools/CMake/AMReXConfig.cmake.in Tools/CMake/AMReXThirdPartyLibraries.cmake commit fb72b08c1914a5b1f8529dcf101c059d613ee4a0 Author: Candace Gilet Date: Thu Oct 14 19:12:12 2021 -0400 Cleanup Tools/Postprocessing/C_src (#2398) Fix memory, parsing for help, and add add automatic determination of whether f90 sources need to be included. Tools/Postprocessing/C_Src/GNUmakefile Tools/Postprocessing/C_Src/IntegrateComp.cpp Tools/Postprocessing/C_Src/PlotfileToMatLab.cpp Tools/Postprocessing/C_Src/PlotfileToTurb.cpp Tools/Postprocessing/C_Src/PtwisePltTransform.cpp commit d5a3334ba58a15a74fdb55da0eabf643a7cd0f9d Author: Wileam Y. Phan <50928756+wyphan@users.noreply.github.com> Date: Thu Oct 14 19:11:22 2021 -0400 Port CellQuadratic to C++ with GPU support (#2369) ## Summary The CellQuadratic interpolater has been ported from Fortran to C++ with GPU support. The errors for the CPU-only version on local are in the ~1e-15 range for AmrLevel SingleVortex 2D, ~1e-15 for IAMR 2D bubble Cartesian, and ~1e-12 for IAMR 2D bubble RZ. The errors for the GPU-enabled version on Gigan are in the ~1e-14 range for AmrLevel SingleVortex2D, ~1e-10 for IAMR 2D bubble Cartesian, and ~1e-9 for IAMR 2D bubble RZ. ## Additional background This port is heavily influenced by the CellConservativeLinear interpolater. It reuses two of the same slope kernels. With this PR, _all_ of the interpolaters now work on the GPU! Also, thanks to @cgilet for helping with debugging. Src/AmrCore/AMReX_INTERP_1D.F90 Src/AmrCore/AMReX_INTERP_2D.F90 Src/AmrCore/AMReX_INTERP_3D.F90 Src/AmrCore/AMReX_INTERP_F.H Src/AmrCore/AMReX_Interpolater.H Src/AmrCore/AMReX_Interpolater.cpp Src/AmrCore/AMReX_MFInterp_2D_C.H Src/AmrCore/CMakeLists.txt Src/AmrCore/Make.package commit d8e6ec544c25233483bf1dc0b5aa263a9a9df730 Author: Michael J. Brazell Date: Thu Oct 14 11:03:35 2021 -0700 added bamg option to set restriction type (#2402) 0 is P transpose and 1,2 are approximate ideal restriction AIR Src/Extern/HYPRE/AMReX_HypreIJIface.cpp commit 06b30c1eaff02952e9d656da01dd090ba5b5a540 Author: Stefan Vater Date: Thu Oct 14 19:59:57 2021 +0200 Bugfix/amr mesh constructor (#1155) These resizes must consider that the vectors in AmrInfo argument might be empty. Co-authored-by: Weiqun Zhang Src/AmrCore/AMReX_AmrMesh.cpp commit 5ae9ea7e67c347d81377ff722f12720a8c5377fc Author: Weiqun Zhang Date: Thu Oct 14 10:11:40 2021 -0700 Fix dot product mask in EB nodal solver (#2404) We should set the dot product mask on all Dirichlet nodes to zero so that they do not contribute to the sum used in fixing singular systems. Src/LinearSolvers/MLMG/AMReX_MLNodeLaplacian_sten.cpp commit a81dd4512658e05c220b358324cf0d67aef13e19 Author: Candace Gilet Date: Thu Oct 14 11:38:00 2021 -0400 Move Tools/C_util/Convergence/WritePlotFile to Src/Extern/amrdata. (#2397) This contains WritePlotFile functions that take an AmrData object as a parameter, and thus Src/Extern/amrdata is the most logical place to put it when codes in Tools/C_util/Convergence and Tools/Postprocessing/C_src use it. Src/Extern/amrdata/AMReX_WritePlotFile.H Src/Extern/amrdata/AMReX_WritePlotFile.cpp Src/Extern/amrdata/CMakeLists.txt Src/Extern/amrdata/Make.package Tools/C_util/Convergence/ComputeAmrDataNorms.cpp Tools/C_util/Convergence/DiffSameDomainRefined.cpp Tools/C_util/Convergence/DiffSameDomainRefinedComposite.cpp Tools/C_util/Convergence/DiffSameDomainRefinedFD.cpp Tools/C_util/Convergence/DiffSameDomainRefinedStag.cpp Tools/C_util/Convergence/DiffSameGrid.cpp Tools/C_util/Convergence/DiffSameGrid2.cpp Tools/C_util/Convergence/DiffSameGridRefined.cpp Tools/C_util/Convergence/DiffUniform.cpp Tools/C_util/Convergence/Make.package Tools/C_util/Convergence/PltFileNormB.cpp Tools/C_util/Convergence/RichardsonConvergenceTest.cpp Tools/Postprocessing/C_Src/PlotfileToTurb.cpp Tools/Postprocessing/C_Src/PtwisePltTransform.cpp commit 2ec5203f43050dcac8a8d8a7a99670e54ef0a87a Author: Candace Gilet Date: Thu Oct 14 11:35:27 2021 -0400 Move info on sharing the command line into the ParmParse section. (#2391) Co-authored-by: etpalmer63 Docs/sphinx_documentation/source/Basics.rst commit 92af41f8a966b252f260c0f3f32d41178844ac50 Author: Weiqun Zhang Date: Thu Oct 14 08:28:51 2021 -0700 Arena Document and PrintUsage (#2403) * Arena Document and PrintUsage * Document arena initial allocation size and release threshold. * Print the number of allocations in PrintUsage. * Update GPU.rst minor spellings Co-authored-by: Erik Docs/sphinx_documentation/source/GPU.rst Src/Base/AMReX_CArena.cpp commit b60a773d6c87f708a736634f6281da16e094e4cb Author: Weiqun Zhang Date: Tue Oct 12 16:03:46 2021 -0700 BArena: Make a static BArena object (#2401) Instead of creating multiple BArena objects, make a static BArena object that is local to a new function The_BArena() in an anonymous namespace. The motivation for this is to properly release the memory allocated by BArena after amrex::Finalize() is called. This is follow-up to #2389. Src/Base/AMReX_Arena.cpp commit 247b41e17aa83f653d848e77f7269e9d43bf4f73 Author: Candace Gilet Date: Tue Oct 12 16:14:54 2021 -0400 Fix inaccurate comment about AmrLevel::writePlotFile, it is no longer pure virtual (#2399) Src/Amr/AMReX_AmrLevel.H commit 48a2776d78ec2a4e2909a875fc7eab25973a81f6 Author: Erik Date: Tue Oct 12 16:09:50 2021 -0400 fix dead link. small reword (#2400) Docs/sphinx_documentation/source/index.rst commit a162b7c15518b6bd5ceb7f8fefa574e6ee3c87d5 Author: Candace Gilet Date: Tue Oct 12 13:23:19 2021 -0400 Make fcompare print usage and exit if only one plotfile is provided. (#2388) Tools/Plotfile/fcompare.cpp commit 4914d5e42c2ee02c80645722b2d336e4dba6b4b3 Author: Weiqun Zhang Date: Mon Oct 11 18:58:13 2021 -0700 Fix compilation of Tools/Postprocessing/C_Src (#2396) Broken by #2390. Tools/Postprocessing/C_Src/Make.package commit 93788e33e689b77a14290580eca0cf9f25be6523 Author: Weiqun Zhang Date: Fri Oct 8 14:54:08 2021 -0700 Add Arena::PrintUsageToFiles for debugging (#2392) Src/Base/AMReX_Arena.H Src/Base/AMReX_Arena.cpp Src/Base/AMReX_CArena.H Src/Base/AMReX_CArena.cpp commit b5cc0323c87d5caa9c3ed7dc9ac639cc0d3fb012 Author: Candace Gilet Date: Fri Oct 8 16:51:29 2021 -0400 Update c util (#2390) This PR cleans out old, unused code from amrex/Tools/C_util, and ensures that what's there compiles and runs. Tools/C_util/AmrDeriveTecplot/AmrDeriveTecplot.cpp Tools/C_util/AmrDeriveTecplot/GNUmakefile Tools/C_util/AppendToPlotFile.H Tools/C_util/AppendToPlotFile.cpp Tools/C_util/AugmentPlotfile/AugmentPlotfile.cpp Tools/C_util/AugmentPlotfile/GNUmakefile Tools/C_util/Convergence/Add.cpp Tools/C_util/Convergence/ComparePlotfiles.cpp Tools/C_util/Convergence/DebugDump.H Tools/C_util/Convergence/DebugOut.H Tools/C_util/Convergence/DebugOut.cpp Tools/C_util/Convergence/DiffFab.cpp Tools/C_util/Convergence/DiffSameDomainRefined.cpp Tools/C_util/Convergence/DiffSameDomainRefinedComposite.cpp Tools/C_util/Convergence/DiffSameDomainRefinedFD.cpp Tools/C_util/Convergence/DiffSameDomainRefinedStag.cpp Tools/C_util/Convergence/DiffSameGrid.cpp Tools/C_util/Convergence/DiffSameGrid2.cpp Tools/C_util/Convergence/DiffSameGridRefined.cpp Tools/C_util/Convergence/DiffUniform.cpp Tools/C_util/Convergence/GNUmakefile Tools/C_util/Convergence/Make.package Tools/C_util/Convergence/PltFileNorm.cpp Tools/C_util/Convergence/PltFileNormB.cpp Tools/C_util/Convergence/PltFileScalConvRate.cpp Tools/C_util/Convergence/RichardsonConvergenceTest.cpp Tools/C_util/Convergence/WritePlotFile.H Tools/C_util/Convergence/WritePlotFile.cpp Tools/C_util/DiffMultiFab/diffmultifab.cpp Tools/C_util/Make.package Tools/C_util/README Tools/C_util/Statistics/AVGDOWN_2D.F Tools/C_util/Statistics/AVGDOWN_3D.F Tools/C_util/Statistics/AVGDOWN_F.H Tools/C_util/Statistics/ComputeAmrDataStat.H Tools/C_util/Statistics/ComputeAmrDataStat.cpp Tools/C_util/Statistics/GNUmakefile Tools/C_util/Statistics/Make.package Tools/C_util/Statistics/PltFileFluxAve.H Tools/C_util/Statistics/PltFileFluxAve.cpp Tools/C_util/Statistics/PltFileList.cpp Tools/C_util/Statistics/PltFileStat.cpp Tools/C_util/Statistics/PltFileXAve.H Tools/C_util/Statistics/PltFileXAve.cpp Tools/C_util/TV_TempWrite.H Tools/C_util/ViewMF/GNUmakefile Tools/C_util/ViewMF/MFNorm.H Tools/C_util/ViewMF/MFNorm.cpp Tools/C_util/ViewMF/Make.package Tools/C_util/ViewMF/checkMFghostcells.cpp Tools/C_util/ViewMF/main.cpp Tools/C_util/ViewMF/mfMinMax.cpp Tools/C_util/ViewMF/viewMF.cpp Tools/C_util/ViewMF/viewMFcol.cpp Tools/C_util/ViewMF/viewMFdiff.cpp Tools/C_util/ViewMF/viewMFdiffHardWire.cpp Tools/C_util/dbgTools/GNUmakefile Tools/C_util/dbgTools/Make.package Tools/C_util/dbgTools/crsGrids.cpp Tools/C_util/dbgTools/intersectGrids.cpp commit d35f4c70bd536425fe49acacb7194983802ea10e Author: PaulMullowney <60452402+PaulMullowney@users.noreply.github.com> Date: Fri Oct 8 09:37:32 2021 -0600 Adding ILU smoothing parameters for Hypre interface. (#2382) Src/Extern/HYPRE/AMReX_HypreIJIface.cpp commit f07f8be1206a4fb7106a24ed611cc5dfcfe5da16 Author: Weiqun Zhang Date: Thu Oct 7 18:33:25 2021 -0700 MLEBNodeFDLaplacian Coarsening (#2386) Limit the maximum coarsening level so that EB objects do not disappear due to coarsening. The solver would fail to converge if the coarsening causes EB objects to disappear entirely. Src/LinearSolvers/MLMG/AMReX_MLEBNodeFDLap_2D_K.H Src/LinearSolvers/MLMG/AMReX_MLEBNodeFDLap_3D_K.H Src/LinearSolvers/MLMG/AMReX_MLEBNodeFDLaplacian.cpp Src/LinearSolvers/MLMG/AMReX_MLLinOp.H Src/LinearSolvers/MLMG/AMReX_MLLinOp.cpp Src/LinearSolvers/MLMG/AMReX_MLNodeLinOp.H Src/LinearSolvers/MLMG/AMReX_MLNodeLinOp.cpp commit b815c734bf01dd6844c55919ddedb4e9541f48ea Author: Weiqun Zhang Date: Thu Oct 7 16:55:50 2021 -0700 Avoid free after amrex::Finalize (#2389) Previously calling Arena::free after amrex::Finalize resulted in segfault because the Arena objects have been deleted. It will cause the following code to crash in the end. int main (int argc, char* argv[]) { amrex::Initialize(argc, argv); amrex::MultiFab(...); amrex::Finalize(); } One way to fix it is to add a scope to the function body between Initialize and Finalize so that the object inside will go out of scope before amrex::Finalize is called. In this commit, we implement a way of avoiding this kind of errors even if one forgets to free everything allocated by Arenas before Finalize. Src/Base/AMReX_Arena.cpp commit c64e719c0a9a5459d2349ef9ebe3a5b8b0e49b41 Author: Candace Gilet Date: Thu Oct 7 17:50:24 2021 -0400 Command line parsing updates. (#2387) Fix command line parsing inconsistency: allow argv[1] to be of the form "arg1 = my_arg", i.e. with spaces, since later arguments allow this. Also, if argv[1] starts with '-', do not use ParmParse. This will prevent "-h" or "--help" from creating errors within ParmParse. Application code can then parse the command line to handle these cases, if desired. Src/Base/AMReX.cpp commit 691e24555db99daa19deeea8da03f951f52a2578 Author: Andrew Myers Date: Wed Oct 6 16:35:44 2021 -0700 These assertions are not needed / wrong after PR #2376 (#2384) Src/Particle/AMReX_ParticleCommunication.H commit f368906eea12d8d68f7f240c17c0cafdfaf9cd1f Author: Andrew Myers Date: Wed Oct 6 15:27:30 2021 -0700 Workaround for nvcc bug that (sometimes) incorrectly complains about unused TinyProfiler variables. (#2383) Src/Base/AMReX_BLProfiler.H commit 3eb5c9edc9d1c62e2bb059e5304edf10559eae1b Author: Andrew Myers Date: Wed Oct 6 13:34:21 2021 -0700 Add inputs for redistribute test that uses a message type other than char. (#2381) Tests/Particles/Redistribute/inputs.rt.cuda.big commit ee975b2973a8e3dfaf5856ea2525832e118e0c98 Author: Neïl Zaim <49716072+NeilZaim@users.noreply.github.com> Date: Wed Oct 6 19:04:28 2021 +0200 Add floor and ceil to parser (#2380) I wanted to use the `floor` function in the parser for a WarpX automated test that I'm developing, so I've added `floor` and `ceil` in the parser. Hopefully I've done it right (I've tested compiling and using with WarpX on my computer and it looks like both functions are working well). Docs/sphinx_documentation/source/Basics.rst Src/Base/Parser/AMReX_Parser_Y.H Src/Base/Parser/AMReX_Parser_Y.cpp Src/Base/Parser/amrex_parser.l Src/Base/Parser/amrex_parser.lex.cpp commit f0a8fc785ff798cac6bdfe4913ad0fa5db988dfe Author: Axel Huebl Date: Tue Oct 5 10:16:10 2021 -0700 CI: Groups Include Branch (#2379) On pushes to mainline, if mainline contains multiple branches, these branches cancelled each other. This should fix this. .github/workflows/clang.yml .github/workflows/cuda.yml .github/workflows/docs.yml .github/workflows/gcc.yml .github/workflows/hip.yml .github/workflows/intel.yml .github/workflows/macos.yml .github/workflows/sensei.yml .github/workflows/style.yml .github/workflows/windows.yml commit 64c9d92c0f4680fc4a0cc6fb48f98ee17771d5e4 Author: Andrew Myers Date: Fri Oct 1 19:58:08 2021 -0700 Fix bug in the particle communication routines when the number of chars in the message is > MAXINT. (#2376) I believe this bug was introduced in PR#1569. Src/Particle/AMReX_ParticleCommunication.H commit a7dd9da02211a88efb3c758ef7f8b806204798e1 Author: Erik Date: Fri Oct 1 19:18:13 2021 -0400 Add short decription for AMReX_TESTING flag (#2375) 1. Add the short description, "build for testing --sets MutliFab initial data to NaN" to the CMake build flags table. 2. Two other small wording changes. Co-authored-by: Weiqun Zhang Docs/sphinx_documentation/source/BuildingAMReX.rst commit 91a9fae04bc2746bec48222731ac9198f3743798 Author: Weiqun Zhang Date: Fri Oct 1 15:31:33 2021 -0700 ROCm now supports __managed__ (#2377) Src/Base/AMReX_GpuQualifiers.H commit e52bccc4d9b95a791284a3e0c00fe27793cc15dd Author: Kevin Z. Zhu <86268612+KZhu-ME@users.noreply.github.com> Date: Fri Oct 1 14:30:50 2021 -0700 Fix read unitialized memory issue in MLMG solver (#2373) ## Summary Running valgrind on one of our failing circleCi tests revealed that the MLMG solver read allocated but uninitialized memory. Setting `m_phi_eb` to 0 after it is allocated fixes this issue. Attached is the valgrind logs, Line 17275 shows where valgrind detected reading of uninitialized memory. [vgout.txt](https://github.com/AMReX-Codes/amrex/files/7269235/vgout.txt) ## Additional background This was found due to one of our warpx circleCI test failing after a fetch and merge with development. This fix did not solve that issue though. Src/LinearSolvers/MLMG/AMReX_MLEBNodeFDLaplacian.H commit b99a8433d3d5b363957b0391f5487ca95ea9e780 Author: Weiqun Zhang Date: Fri Oct 1 11:59:50 2021 -0700 Support Neumann BC in MLEBNodeFDLaplacian (#2371) Src/LinearSolvers/MLMG/AMReX_MLEBNodeFDLaplacian.H Src/LinearSolvers/MLMG/AMReX_MLEBNodeFDLaplacian.cpp commit 3ede71fa7835699784e0fca632980d89d7c23ee5 Author: Weiqun Zhang Date: Fri Oct 1 11:55:41 2021 -0700 Reset GPU RNG Seed (#2374) Add an optional parameter to amrex::ResetRandomSeed to allow the users to reset the GPU RNG seed. Slip in a typo fix. Docs/sphinx_documentation/source/GPU.rst Src/Base/AMReX_Random.H Src/Base/AMReX_Random.cpp commit 98a4e2338b2aef2718981b552de2d3b5db409c14 Author: Andrew Myers Date: Fri Oct 1 11:55:18 2021 -0700 Remove managed memory from neighbor particle test. (#2370) Tests/Particles/NeighborParticles/MDParticleContainer.cpp commit 1211ea30b520eb270e99d63fb80fb1d94ca8e45a Author: PaulMullowney <60452402+PaulMullowney@users.noreply.github.com> Date: Fri Oct 1 12:49:46 2021 -0600 NodeLaplacian Updates (#2332) In this branch, I make the number of smoothing steps a user defined parameter. Previously it was fixed at 4 for GPUs and 2 for CPUs. I also modified the NodalPoisson test to 1) use this new parameter 2) include additional parameters that control plot file dumping and the number of trials of the test to be run. Src/LinearSolvers/MLMG/AMReX_MLNodeLaplacian.H Src/LinearSolvers/MLMG/AMReX_MLNodeLaplacian_misc.cpp Src/LinearSolvers/MLMG/AMReX_MLNodeLinOp.H Tests/LinearSolvers/NodalPoisson/MyTest.H Tests/LinearSolvers/NodalPoisson/MyTest.cpp Tests/LinearSolvers/NodalPoisson/main.cpp commit 64ab8faa89e663cb912da9ad03a058c609e95447 Author: Weiqun Zhang Date: Fri Oct 1 07:58:07 2021 -0700 Update CHANGES for 21.10 (#2372) CHANGES commit 6f2dc1d06ee7d3f0024a9b8480ea8dde0d5fd644 Author: Weiqun Zhang Date: Wed Sep 29 15:37:17 2021 -0700 Update for oneapi 2021.4.0 (#2368) There are some namespace changes and deprecated functions in the new release. .github/workflows/intel.yml Src/Base/AMReX_FBI.H Src/Base/AMReX_GpuPrint.H Src/Base/AMReX_GpuQualifiers.H Src/Base/AMReX_GpuReduce.H Src/Base/AMReX_ParmParse.H Src/Base/AMReX_Scan.H commit b11479d29b987b35089918d40747de72f994e117 Author: Axel Huebl Date: Wed Sep 29 15:09:36 2021 -0700 CI: NVHPC for CUDA (#2066) .github/workflows/cuda.yml .github/workflows/dependencies/dependencies_nvhpc21-9.sh Src/Base/AMReX_BLBackTrace.H Src/Base/AMReX_Extension.H Tools/CMake/AMReXGenerateConfigHeader.cmake commit 3eb5af68d209d056c20464c70d9abd9d0641e61a Author: Erik Date: Wed Sep 29 15:32:29 2021 -0400 fix -B grad(phi) typo to -beta grad(phi) (#2367) Fix a typo in the rst documents for the linear solvers. Docs/sphinx_documentation/source/LinearSolvers.rst commit ba6211f431bb594e7fc20478e9309fb88e79acc9 Author: Wileam Y. Phan <50928756+wyphan@users.noreply.github.com> Date: Wed Sep 29 14:49:54 2021 -0400 CI: Change HIP repo line to reflect latest changes (#2366) This updates the repo line for HIP to the latest version as documented on https://rocmdocs.amd.com/en/latest/Installation_Guide/Installation-Guide.html#installing-a-rocm-package-from-a-debian-repository Notably, `xenial` is now `ubuntu`. .github/workflows/dependencies/dependencies_hip.sh commit 8bf9816cf6f2a50cddad288deab1e8e72ef99e76 Author: Wileam Y. Phan <50928756+wyphan@users.noreply.github.com> Date: Wed Sep 29 14:09:41 2021 -0400 Port CellConservativeQuartic to C++ with GPU support (#2362) CellConservativeQuartic interpolater has been ported to C++ with GPU support. Errors for CPU-only version on local with AmrLevel SingleVortex 2D and 3D are exactly 0. Errors for GPU-enabled version on Gigan with AmrLevel SingleVortex 2D and 3D are on the order of 1e-14. Src/AmrCore/AMReX_INTERP_1D.F90 Src/AmrCore/AMReX_INTERP_2D.F90 Src/AmrCore/AMReX_INTERP_3D.F90 Src/AmrCore/AMReX_INTERP_F.H Src/AmrCore/AMReX_Interp_1D_C.H Src/AmrCore/AMReX_Interp_2D_C.H Src/AmrCore/AMReX_Interp_3D_C.H Src/AmrCore/AMReX_Interpolater.H Src/AmrCore/AMReX_Interpolater.cpp commit b219a9cbd456863b95fe78a949339ba94fce65ab Author: Axel Huebl Date: Wed Sep 29 10:13:37 2021 -0700 nvc++ w/ OMP: work-around atomic capture (#2365) ## Summary Internal compiler error in this region with NVHPC 21.9 when compiling for OpenMP (host-only). ## Additional background Nvidia bug report: 3390723. Src/Base/AMReX_FabArrayUtility.H commit 514320464dacb9677ae1b84a4aee25ca6669c22c Author: Erik Date: Wed Sep 29 11:45:33 2021 -0400 Gpu Comments to Doxygen Docs (#2354) Added some comments about `Gpu::setLaunchRegion`, `Gpu::synchronize()` and `Gpu::streamSynchronize()` from the new and previous Gpu Debugging write-up. Converted some comments to Doxygen comments. Src/Base/AMReX_GpuControl.H Src/Base/AMReX_GpuDevice.H commit 5932c85cd650eac65c04e09ca0163fe29de5421b Author: Corey Wetterer-Nelson <78513275+c-wetterer-nelson@users.noreply.github.com> Date: Wed Sep 29 00:03:00 2021 -0400 fix AmrMeshParticle cmake issue (#2364) Src/Extern/SENSEI/CMakeLists.txt commit 9b58b442cc7091e5cf18375112d60a2f3d1a1bee Author: Axel Huebl Date: Tue Sep 28 18:42:35 2021 -0700 CI: Split Linux (#2363) The CI list became too long in the Linux section. This is cumbersome if one of the tests fails due to network/runner issues, because then all need to be restarted. Split into: - Linux GCC - Linux Clang - SENSEI .github/workflows/clang.yml .github/workflows/gcc.yml .github/workflows/sensei.yml commit 3006bba653ea8d93b58e779bdf796f3bd07e9bfd Author: Weiqun Zhang Date: Tue Sep 28 14:16:16 2021 -0700 Fix bug introduced in #2327 in DPCPP backend (#2361) Src/Base/AMReX_Reduce.H commit 9256409e2aae8148ef13afcf8316a72f12f1f3b8 Author: Max Katz Date: Tue Sep 28 09:06:16 2021 -0700 Remove hardcoded CUDA version for NVHPC (#2360) Tools/GNUMake/comps/nvhpc.mak commit 26fb18b44481a76d22f182e5955e23d5dbeeef29 Author: Wileam Y. Phan <50928756+wyphan@users.noreply.github.com> Date: Mon Sep 27 17:27:26 2021 -0400 Port CellConservativeProtected::protect() to C++ with GPU support (#2347) CellConservativeProtected interpolater has been ported to C++ with GPU support. For the CPU-only version on local, the errors are exactly 0. For the GPU-enabled version on Gigan, the errors are in ~1e-11 range for IAMR 2D bubble test case, and ~1e-6 range for IAMR 3D Poiseuille test case. Co-authored-by: Weiqun Zhang Src/AmrCore/AMReX_INTERP_2D.F90 Src/AmrCore/AMReX_INTERP_3D.F90 Src/AmrCore/AMReX_INTERP_F.H Src/AmrCore/AMReX_Interp_2D_C.H Src/AmrCore/AMReX_Interp_3D_C.H Src/AmrCore/AMReX_Interpolater.H Src/AmrCore/AMReX_Interpolater.cpp commit 54dc681046f8c3f19034358fe0d45b72d13a9863 Author: Axel Huebl Date: Fri Sep 24 11:25:59 2021 -0700 CMake CUDA Debug: -G by Default (#2355) ## Summary In the past, this often did not compile at all, was very sensitive to further set options, or compiled super slowly; in some cases, such as recursive function usage, apps need to increase `cudaLimitStackSize` in order to not stack overflow with device debug symbols (this costs some extra DRAM). Nonetheless, for CUDA approx. 11.0+, we see the opposite: we have very slow Debug builds with CUDA if we do not activate -G for some `LinearSolvers/MLMG` objects: `AMReX_MLNodeLaplacian.cpp` & `AMReX_MLNodeLaplacian_sten.cpp`. Thus, we default-on now to -G in Debug builds. - [x] testing: now `LinearSolvers/MLMG/AMReX_MLTensorOp.cpp` compiles a bit longer (1-2min), but that's better than the other two TUs above without `-G` in Debug ## Additional background Thanks to @WeiqunZhang and @maxpkatz to looking into this with me. Docs/sphinx_documentation/source/GPU.rst Tools/CMake/AMReXCUDAOptions.cmake Tools/CMake/AMReXParallelBackends.cmake commit 66c90801c63a0c682ae59f0af3d2ebb4cb57f637 Author: kngott Date: Fri Sep 24 10:17:42 2021 -0700 OverlapSync nowait and finish. (#2346) Src/Base/AMReX_FabArray.H Src/Base/AMReX_FabArrayUtility.H Src/Base/AMReX_MultiFab.H Src/Base/AMReX_MultiFab.cpp commit 777a2927f0e4da2f398d54914a5bb580c8a305a4 Author: Andrew Myers Date: Thu Sep 23 14:48:49 2021 -0700 Add another overload for PC::Checkpoint (#2353) Src/Particle/AMReX_ParticleIO.H Src/Particle/AMReX_Particles.H commit 8c79eec68331b7e2122517e776a20cf77dcd04fc Author: Candace Gilet Date: Thu Sep 23 14:13:56 2021 -0400 Add curly brackets between amrex::Initialize and amrex::Finalize. (#2352) This prevents segmentation faults that can occur if amrex objects (like MultiFabs) are still in scope when Finalize is called. Tools/C_util/Convergence/Add.cpp Tools/C_util/Convergence/ComparePlotfiles.cpp Tools/C_util/Convergence/DiffFab.cpp Tools/C_util/Convergence/DiffSameDomainRefined.cpp Tools/C_util/Convergence/DiffSameDomainRefinedComposite.cpp Tools/C_util/Convergence/DiffSameDomainRefinedFD.cpp Tools/C_util/Convergence/DiffSameDomainRefinedStag.cpp Tools/C_util/Convergence/DiffSameGrid.cpp Tools/C_util/Convergence/DiffSameGrid2.cpp Tools/C_util/Convergence/DiffSameGridRefined.cpp Tools/C_util/Convergence/DiffUniform.cpp Tools/C_util/Convergence/PltFileNorm.cpp Tools/C_util/Convergence/PltFileNormB.cpp Tools/C_util/Convergence/PltFileScalConvRate.cpp Tools/C_util/Convergence/RichardsonConvergenceTest.cpp commit 368a808c57ca3dbff40c3aa496e79fdf46c730d7 Author: asalmgren Date: Tue Sep 21 12:42:11 2021 -0700 Call setLevelBC from getFluxes in case we have set up the projector class but not yet called project (#2345) Src/LinearSolvers/Projections/AMReX_MacProjector.cpp commit a6c76f123cee34ec57cfb4f11fdf53389da3c640 Author: Andrew Myers Date: Mon Sep 20 18:15:35 2021 -0700 Function to atomically set particle ids (#2344) Src/Particle/AMReX_Particle.H commit 5d6a01f90ca606ac7a46879102db744041414b8f Author: Erik Date: Mon Sep 20 13:41:41 2021 -0400 Faq part one (#2340) Docs/sphinx_documentation/source/Faq.rst Docs/sphinx_documentation/source/index.rst commit 205c9b9ef6996b34138e170985b4d79d028bd57e Author: Weiqun Zhang Date: Mon Sep 20 10:41:16 2021 -0700 Fix CUDA compilation for EBTensor test (#2342) Tests/LinearSolvers/EBTensor/MyTest.H commit 8e0a48ea2e068111e171b8649fa8d71294bfcf7c Author: Weiqun Zhang Date: Mon Sep 20 10:33:21 2021 -0700 Remove macro: CRSEGRNDOMP and AMREX_CRSEGRNDOMP (#2339) The macro was added for backward compatibility when BoxLib first transitioned to omp over tiles. This can be removed now because We no longer have any Fab level functions containing OpenMP parallel regions. Src/Amr/AMReX_AmrLevel.cpp Src/Amr/AMReX_StateData.cpp Tests/EB/CNS/Exec/Make.CNS Tests/EB_CNS/Exec/Make.CNS Tests/GPU/CNS/Exec/Make.CNS commit 3cff89900f8f6c21e02f8daceacffb4fda4b728a Author: Andrew Myers Date: Mon Sep 20 10:21:41 2021 -0700 Force turning off particle tiling for GPU on Advection_AmrLevel test (#2343) This enables the same inputs file to be used for GPU / non-GPU runs. Tests/Amr/Advection_AmrLevel/Source/main.cpp commit d568a0f9bb3ac324ad52b263240ddda03906ef43 Author: Andrew Myers Date: Mon Sep 20 10:20:58 2021 -0700 Don't rely on mananaged memory in PC::InitOnePerCell (#2337) Src/Particle/AMReX_ParticleInit.H commit 350a3efc2c6408efcadb6aaf0a334944af51098b Author: Michael Kieburtz Date: Mon Sep 20 08:26:21 2021 -0700 TinyProfile parser that saves data in a machine-readable format (#2335) This script parses the saved stdout output from a simulation run and parses the TinyProfile portion at the end. It saves this output in a JSON file that can be interpreted by Hatchet (https://github.com/hatchet/hatchet). Tools/TinyProfileParser/profileparser.py commit 677bc0990c7dea0cd32e3b8ab07664791c1681e1 Author: Weiqun Zhang Date: Sat Sep 18 12:12:19 2021 -0700 Various fixes (#2338) * AMReX_EB_LeastSquares_2D_K.H: fix warning * Tests/Amr/Advection_AmrLevel: should use pinned memory on host * Tests/AsyncOut/multifab: avoid managed memory * Tests/EB_CNS: fix warning on extra semicolon * Tests/GPU/AnyOf: avoid touching managed memory on host * Tests/LinearSolvers/CellOverset: fix compilation * Tests/LinearSolvers/EBTensor: initialize data on device * Tests/Parser: print max rel. error * Tests/Particles/AsyncIO: fix typo in inputs * Tests/Particles/DenseBins: do not test serial version for dpcpp * Tests/Particles/GhostsAndVirtuals: do not do randomperbox test for dpcpp Src/EB/AMReX_EB_LeastSquares_2D_K.H Tests/Amr/Advection_AmrLevel/Exec/UniformVelocity/face_velocity_2d_K.H Tests/Amr/Advection_AmrLevel/Exec/UniformVelocity/face_velocity_3d_K.H Tests/AsyncOut/multifab/main.cpp Tests/EB_CNS/Source/CNS.H Tests/GPU/AnyOf/main.cpp Tests/LinearSolvers/CellOverset/MyTest.cpp Tests/LinearSolvers/EBTensor/MyTest.cpp Tests/LinearSolvers/EBTensor/MyTest_2D_K.H Tests/LinearSolvers/EBTensor/MyTest_3D_K.H Tests/Parser/main.cpp Tests/Particles/AsyncIO/inputs Tests/Particles/DenseBins/main.cpp Tests/Particles/GhostsAndVirtuals/main.cpp commit f47f884a453dfb76c372f1cf064e5f1e7df5be33 Author: Weiqun Zhang Date: Fri Sep 17 09:38:44 2021 -0700 Add documentation on reduction using ParallelFor. (#2333) Co-authored-by: Wileam Y. Phan <50928756+wyphan@users.noreply.github.com> Docs/sphinx_documentation/source/GPU.rst commit 57703d6b624c7b03ff4bd282c3f07ca9aa5bb0fe Author: Jean M. Sexton Date: Fri Sep 17 09:37:45 2021 -0700 Update GPU Debugging basics section (#2330) * Update GPU Debugging basics section * Update Cuda-specific section to include new tools * Added Rocm specific commands * Added some information from Intel debugging webpages * Added links in comments Co-authored-by: etpalmer63 Docs/sphinx_documentation/source/GPU.rst commit 69b69c2b281c6f1b7b9b78a5aa81effebf0e3f2f Author: Weiqun Zhang Date: Mon Sep 13 16:21:41 2021 -0700 Fix return type of MLEBNodeFDLaplacian::setEBDirichlet (#2329) Src/LinearSolvers/MLMG/AMReX_MLEBNodeFDLaplacian.H commit 3d5b7d2e99d635960d031d6ad2aa07c9472be2e1 Author: Weiqun Zhang Date: Mon Sep 13 10:27:22 2021 -0700 ParReduce (#2327) ## Summary * Add ParReduce functions for MultiFab/FabArray and 1D iteration space. * The implementation of ParReduce(MF) uses new ReduceOps::eval functions that launch one GPU kernel for the whole MultiFab/MultiFab. * Use the new ParReduce functions in MultiFab/FabArray reduction functions. ## Additional background The name `ParallelReduce` has been used for the namespace containing MPI reduce wrappers. So we have to use a different name. Src/Base/AMReX_FabArray.H Src/Base/AMReX_FabArrayUtility.H Src/Base/AMReX_GpuTypes.H Src/Base/AMReX_MultiFab.H Src/Base/AMReX_MultiFab.cpp Src/Base/AMReX_ParReduce.H Src/Base/AMReX_Reduce.H Src/Base/AMReX_Tuple.H Src/Base/AMReX_iMultiFab.cpp Src/Base/CMakeLists.txt Src/Base/Make.package Src/LinearSolvers/MLMG/AMReX_MLCGSolver.cpp commit 18a57c000e608d9e31054e84eceb799fb2789a4b Author: kngott Date: Mon Sep 13 08:08:44 2021 -0700 Update Profiling API docs. (#2303) Based on previous discussion that TinyProfiler's REGION wasn't well documented, this adds in the discussion about the various methods and macros available for instrumentation. Co-authored-by: etpalmer63 Docs/sphinx_documentation/source/AMReX_Profiling_Tools.rst commit 177b78eec251582646050db125af007117e97aae Author: Weiqun Zhang Date: Sun Sep 12 10:05:03 2021 -0700 Remove the MLNodeLinOp test (#2328) The tutorial has been moved to amrex-tutorial. Because of recent changes, the test no longer compiles. Tests/LinearSolvers/MultiComponent/GNUmakefile Tests/LinearSolvers/MultiComponent/MCNodalLinOp.H Tests/LinearSolvers/MultiComponent/MCNodalLinOp.cpp Tests/LinearSolvers/MultiComponent/Make.package Tests/LinearSolvers/MultiComponent/inputs Tests/LinearSolvers/MultiComponent/main.cpp Tools/RegressionTesting/AMReX-tests.ini commit c1c57379871fec43b3512eab09b12950673e1ae6 Author: Weiqun Zhang Date: Sun Sep 12 08:58:33 2021 -0700 Remove some versions of ParallelFor(MF) (#2326) Remove ParallelFor(MF) functions that only take `int ncomp`, but not `IntVect nghost` to avoid potential bugs. The versions we have kept are, * ParallelFor(MF const&, ...); // valid region only * ParallelFor(MF const&, IntVect const& nghost, ...); // valid+nghost * ParallelFor(MF const&, IntVect const& nghost, int ncomp, ...); // valid+nghost and loop over component The version that is removed is * ParallelFor(MF const&, int ncomp, ...); This avoids bugs like below ParallelFor(mf, mf.nGrow(), ...); // should use mf.nGrowVect() Here, `mf.nGrow()` returns `int` and therefore would be treated as the number of components in the removed version. Src/Base/AMReX_MFParallelFor.H Src/Base/AMReX_MultiFabUtil.cpp Src/LinearSolvers/MLMG/AMReX_MLABecLaplacian.cpp Src/LinearSolvers/MLMG/AMReX_MLCellLinOp.cpp commit 1c5608e05856e8690f968900c88924043addf025 Author: Erik Date: Sun Sep 12 00:19:15 2021 -0400 Amrvis doc revisions (#2305) Docs/sphinx_documentation/source/Visualization.rst commit b49296b837bcdd411c9d79f2abf07de18abc5c46 Author: Brandon Runnels Date: Fri Sep 10 14:03:10 2021 -0600 Update MLMG solver to allow variable number of ghost nodes for variable refinement ratios (#2319) Src/LinearSolvers/MLMG/AMReX_MLLinOp.H Src/LinearSolvers/MLMG/AMReX_MLMG.cpp commit b3e38df8297bf32fd715f9f6491314a72c801d3b Author: Erik Date: Fri Sep 10 15:17:35 2021 -0400 Downgrade to docutils v0.16 (#2324) * Downgrade to docutils v0.16 to get bullet lists to render properly with Sphinx. .github/workflows/docs.yml commit f32f822e9ace1f8e4cd711b0bf7dbd174bd94e75 Author: Andrew Myers Date: Fri Sep 10 08:41:50 2021 -0700 Fix bug introduced in PR 2322 (#2325) Src/Base/AMReX_FabArray.H commit 36672da197ff9bda10a90d454212dc1c4ac3f9ff Author: Axel Huebl Date: Fri Sep 10 08:04:44 2021 -0700 CMake: Always CUDA Perf-Neutral Debug (#2321) Always default-ON for performance-neutral debug info to CUDA builds with CMake, even in `Release` mode. Tools/CMake/AMReXCUDAOptions.cmake commit 107ef9a1e1d20c149c2325ab5e8878cf4d7df052 Author: Andrew Myers Date: Thu Sep 9 16:36:41 2021 -0700 Add version of SumBoundary that takes src_nghost (#2322) Src/Base/AMReX_FabArray.H commit 3dc694a754d499c58d6f81f14c530e9e122f5858 Author: Erik Date: Thu Sep 9 12:12:49 2021 -0400 Add comment on "is_periodic" to doxygen (#2320) Co-authored-by: atmyers Src/Base/AMReX_Geometry.H commit d187e374f11c0fbe75c37544e95cb2185410b816 Author: Weiqun Zhang Date: Tue Sep 7 14:49:07 2021 -0700 Tests/EB_CNS on GPU (#2318) * Make some member functions public for CUDA. * Update calls to EBFluxRegister to run on device. * Fix race conditions in two kernels, one by avoiding write with if test and the other with atomics. * Use ParallelFor(MF) instead of ParallelFor(Box) for performance in a few places. * Use MFInterpolater instead of Interpolater for performance. * Remove Elixirs and rely on Gpu::streamSynchronize(). Maybe later, we could switch to using The_Async_Arena for these temporary Fabs for performance. * inline -> AMREX_FORCE_INLINE * Fix some 2D issues. Tests/EB_CNS/Exec/Pulse/cns_prob.H Tests/EB_CNS/Exec/ShockRef/cns_prob.H Tests/EB_CNS/Source/CNS.H Tests/EB_CNS/Source/CNS.cpp Tests/EB_CNS/Source/CNS_K.H Tests/EB_CNS/Source/CNS_advance.cpp Tests/EB_CNS/Source/CNS_advance_box.cpp Tests/EB_CNS/Source/CNS_advance_box_eb.cpp Tests/EB_CNS/Source/CNS_setup.cpp Tests/EB_CNS/Source/CNS_tagging.H Tests/EB_CNS/Source/diffusion/CNS_diffusion_K.H Tests/EB_CNS/Source/diffusion/CNS_diffusion_eb_K.H Tests/EB_CNS/Source/hydro/CNS_divop_K.H Tests/EB_CNS/Source/hydro/CNS_flux_redistribute.cpp Tests/EB_CNS/Source/hydro/CNS_hydro_K.H Tests/EB_CNS/Source/hydro/CNS_hydro_eb_K.H commit 0dab910e02864bb643cc91d78639df06f51fe17c Author: Weiqun Zhang Date: Tue Sep 7 10:17:14 2021 -0700 Remove the Old Fusing Approach (#2308) The old fusing macros and functions are removed in this commit. The old fusing approach is based on device function pointers. It only worked on Nvidia GPUs. Furthermore, the new ParallelFor(MF) and ParallelFor(Tag) approaches are much faster than the old approach. Src/Base/AMReX.cpp Src/Base/AMReX_FabArrayUtility.H Src/Base/AMReX_Gpu.H Src/Base/AMReX_GpuFuse.H Src/Base/AMReX_GpuFuse.cpp Src/Base/AMReX_GpuKernelInfo.H Src/Base/AMReX_GpuLaunch.H Src/Base/AMReX_GpuLaunchFunctsG.H Src/Base/AMReX_GpuLaunchMacrosG.H Src/Base/AMReX_MFIter.H Src/Base/AMReX_MFIter.cpp Src/Base/CMakeLists.txt Src/Base/Make.package Src/LinearSolvers/MLMG/AMReX_MLCellLinOp.cpp Tests/GPU/Fuse/GNUmakefile Tests/GPU/Fuse/Make.package Tests/GPU/Fuse/main.cpp commit 26a55b4a277226a00f8c9e507268b2fd3ef2f3ee Author: kngott Date: Tue Sep 7 10:16:54 2021 -0700 Amrvis build cleanup. (#2314) Src/Extern/amrdata/AMReX_DataServices.H Tools/GNUMake/Make.rules commit 30ad2cb920f59e8d1174fd3a8aa242b8b280df3e Author: Weiqun Zhang Date: Tue Sep 7 08:45:52 2021 -0700 Add const to Array's sum and product functions (#2316) Since C++14, `constexpr` does not imply `const`. This is a follow-up on #2217. Src/Base/AMReX_Array.H commit 4a79504336a34a2728e418e71f3d68dba3afed5d Author: Candace Gilet Date: Mon Sep 6 15:53:28 2021 -0400 Update hypre doc (#2315) Update documentation on using HYPRE with the nodal solver. Support for CoarseningStrategy::Sigma with hypre has been added. Docs/sphinx_documentation/source/LinearSolvers.rst commit a8fe43774d8ad7437ff71bb253175d64a168d933 Author: Andrew Myers Date: Fri Sep 3 16:33:34 2021 -0700 Fix signed/unsigned comparison in diagnostic neighbor list function. (#2312) Src/Particle/AMReX_NeighborList.H commit 168a690497396de4c6b89a36b6edb0430e51ef4c Author: Houjun Tang Date: Fri Sep 3 12:22:04 2021 -0700 HDF5 I/O and Compression (#2220) .github/workflows/linux.yml Src/Base/AMReX_PlotFileUtil.cpp Src/Particle/AMReX_ParticleHDF5.H Src/Particle/AMReX_Particles.H Src/Particle/AMReX_WriteBinaryParticleData.H Tests/HDF5Benchmark/GNUmakefile Tests/HDF5Benchmark/inputs Tests/HDF5Benchmark/main.cpp Tools/GNUMake/packages/Make.hdf5 commit 55c18a5e98841034df30f8b91300a64b1269567b Author: Weiqun Zhang Date: Fri Sep 3 10:56:40 2021 -0700 Use ParallelFor(Tag) in Linear Solver Preparation (#2297) Replace the old fusing approach with ParallelFor(Tag) in MLCellLinOp::prepareForSolve. Src/LinearSolvers/MLMG/AMReX_MLCellLinOp.cpp Src/LinearSolvers/MLMG/AMReX_MLLinOp_K.H commit 941607f89f68db1dc904d2321b80ac7f622740a9 Author: Weiqun Zhang Date: Fri Sep 3 08:39:44 2021 -0700 TagBoxArray: Remove omp parallel inside GPU launch region (#2309) Src/AmrCore/AMReX_TagBox.cpp commit 0a85e99beadea4cbe4b577a2854e094a834c5e75 Author: Andy Nonaka Date: Fri Sep 3 08:39:11 2021 -0700 make sure there are no NaNs in the plotfiles (#2306) The following routines would report no difference between two input plotfiles if one of them contained NaNs, which makes for a really bad comparison for regression usage. Now they abort with an error message if a NaN is detected. ComparePlotfiles.cpp DiffSameDomainRefined.cpp DiffSameDomainRefinedComposite.cpp DiffSameDomainRefinedStag.cpp DiffSameGrid.cpp DiffSameGrid2.cpp DiffSameGridRefined.cpp Tools/C_util/Convergence/ComparePlotfiles.cpp Tools/C_util/Convergence/DiffSameDomainRefined.cpp Tools/C_util/Convergence/DiffSameDomainRefinedComposite.cpp Tools/C_util/Convergence/DiffSameDomainRefinedStag.cpp Tools/C_util/Convergence/DiffSameGrid.cpp Tools/C_util/Convergence/DiffSameGrid2.cpp Tools/C_util/Convergence/DiffSameGridRefined.cpp commit 2bf85c3e7cae74149703c0e5b5020851a2433266 Author: Corey Wetterer-Nelson <78513275+c-wetterer-nelson@users.noreply.github.com> Date: Fri Sep 3 11:00:12 2021 -0400 add dropped CMake code for mesh+particle adaptor (#2310) Src/Extern/SENSEI/CMakeLists.txt commit 112ce5551c43c02ff5fb8ca4241ac2d22d89d3e6 Author: Weiqun Zhang Date: Fri Sep 3 07:59:19 2021 -0700 Update DiffMultiFab tool (#2307) Print the location of min and max diff. Make it work with MPI. An example of output is below, ``` diffmultifab3d.gnu.TEST.ex infile1=mf1 infile2=mf2 ngrow=0 Reading mf1 Reading mf2 Component 0 Min and max of the diff are -0.5738299806 and 0.3011247047 Min Index: (63,127,127) Max Index: (75,127,127) Min and max of 1st mf are 0.1249966244 and 1.000010767 Min and max of 2nd mf are 0.125 and 1 Component 1 Min and max of the diff are -1.273924814e-05 and 0.3954904262 Min Index: (31,127,127) Max Index: (64,127,127) Min and max of 1st mf are -1.273924814e-05 and 0.3954904262 Min and max of 2nd mf are 0 and 0 Component 2 Min and max of the diff are 0 and 0 Min and max of 1st mf are 0 and 0 Min and max of 2nd mf are 0 and 0 Writing mfdiff ``` Tools/C_util/DiffMultiFab/GNUmakefile Tools/C_util/DiffMultiFab/diffmultifab.cpp commit e31e0161658664256eec4c3ae1cf98da33273601 Author: Weiqun Zhang Date: Thu Sep 2 16:57:36 2021 -0700 Kernel fusing in linear solver's applyBC (#2295) Use ParallelFor(Tag) to fuse the kernels in the cell-centered solver's applyBC function. Src/LinearSolvers/MLMG/AMReX_MLCellLinOp.cpp Src/LinearSolvers/MLMG/AMReX_MLLinOp_K.H commit b15b1cf8d282cbb2c0d0bc0c7b049a79375ea63c Author: PaulMullowney <60452402+PaulMullowney@users.noreply.github.com> Date: Thu Sep 2 09:13:59 2021 -0600 Adding new parallel for into normalize method of node laplacian. (#2299) Co-authored-by: Paul Mullowney Src/LinearSolvers/MLMG/AMReX_MLNodeLap_1D_K.H Src/LinearSolvers/MLMG/AMReX_MLNodeLap_2D_K.H Src/LinearSolvers/MLMG/AMReX_MLNodeLap_3D_K.H Src/LinearSolvers/MLMG/AMReX_MLNodeLap_K.H Src/LinearSolvers/MLMG/AMReX_MLNodeLaplacian.cpp commit c428771ceb21e812b92e21bc6383c650192982f1 Author: Andrew Myers Date: Wed Sep 1 17:30:20 2021 -0700 Gpu::Atomic::If (#2301) Src/Base/AMReX_Functional.H Src/Base/AMReX_GpuAtomic.H Tests/GPU/AtomicIf/GNUmakefile Tests/GPU/AtomicIf/Make.package Tests/GPU/AtomicIf/main.cpp commit 17d5b9049942d1cf72f943ee24b9c34cff8f6dfa Author: Corey Wetterer-Nelson <78513275+c-wetterer-nelson@users.noreply.github.com> Date: Wed Sep 1 18:47:10 2021 -0400 add adaptor and bridge for amrMesh+Particles (#2285) Src/Extern/SENSEI/AMReX_AmrMeshParticleDataAdaptor.H Src/Extern/SENSEI/AMReX_AmrMeshParticleDataAdaptorI.H Src/Extern/SENSEI/AMReX_AmrMeshParticleInSituBridge.H Src/Extern/SENSEI/AMReX_ParticleDataAdaptorI.H commit 05539820d5878f2548b0b84328111efdde25ec7d Author: Weiqun Zhang Date: Wed Sep 1 10:04:07 2021 -0700 Update CHANGES for 21.09 (#2302) CHANGES commit cdb54ed66dac17d8867ee5dce01718852dd1788f Author: Erik Date: Wed Sep 1 12:29:03 2021 -0400 Change list of features to bullet list in Introduction.rst (#2300) Docs/sphinx_documentation/source/Introduction.rst commit 889686189996023c06e796c2cece3e5415a4b6b1 Author: Erik Date: Mon Aug 30 18:18:22 2021 -0400 Add Docs for `fboxinfo`, `fcompare`, `fextract`, etc. to Post-Processing Tools (#2290) Docs/sphinx_documentation/source/Post_Processing.rst Docs/sphinx_documentation/source/figs/ex_fsnapshot_resize.png commit e67bc03243c7df9acec8ae2ec66f9817bbda5fea Author: Wileam Y. Phan <50928756+wyphan@users.noreply.github.com> Date: Mon Aug 30 18:16:16 2021 -0400 Add gradient error tagging in Tests/Advection_AmrLevel (#2296) ## Summary Add proper `FillPatch()`ing for gradient checks in `AmrLevelAdv::errorEst()`. ## Additional background While working on #2268, I "accidentally" discovered that even though gradient error tagging is coded in `Advection_AmrLevel`, this check is never actually performed. This PR tries to fix that by adapting the error tagging code from `Tests/GPU/CNS`. Tested on local for CPU (error = exactly 0) and on Gigan for NVIDIA GPU (error = 1e-14 range). Tests/Amr/Advection_AmrLevel/Source/AmrLevelAdv.cpp commit d0b275b7064e20b87272eda07ea9adef0f63cf2a Author: asalmgren Date: Fri Aug 27 16:21:41 2021 -0700 Move slope routines out of amrex (#2294) * Fix logic for 3D interpolation of cell centroids to face centroids * Remove print statement that I left in by accident * Don't use std::min on the gpu! * When using hypre, limit number of possible multigrid levels by how much the EB can be coarsened * Move slope (EB and regular) routines out of amrex and into AMReX-Hydro * Move Tests/Slopes into AMReX-Hydro as well. Src/Base/AMReX_Slopes_K.H Src/Base/CMakeLists.txt Src/Base/Make.package Src/EB/AMReX_EB_slopes_K.H Src/EB/CMakeLists.txt Src/EB/Make.package Tests/Slopes/GNUmakefile Tests/Slopes/Make.package Tests/Slopes/MyEB.H Tests/Slopes/MyTest.H Tests/Slopes/MyTest.cpp Tests/Slopes/README.md Tests/Slopes/initData.cpp Tests/Slopes/initEB.cpp Tests/Slopes/initLinearData.cpp Tests/Slopes/initLinearDataFor2D.cpp Tests/Slopes/initLinearDataFor3D.cpp Tests/Slopes/inputs Tests/Slopes/inputs.2d.askew Tests/Slopes/inputs.2d.base Tests/Slopes/inputs.2d.fullyrotated Tests/Slopes/inputs.3d.linear.aligned.xy-x Tests/Slopes/inputs.3d.linear.aligned.xy-y Tests/Slopes/inputs.3d.linear.aligned.xz-x Tests/Slopes/inputs.3d.linear.aligned.xz-z Tests/Slopes/inputs.3d.linear.aligned.yz-y Tests/Slopes/inputs.3d.linear.aligned.yz-z Tests/Slopes/inputs.3d.linear.askew-all Tests/Slopes/inputs.3d.linear.askew-xy Tests/Slopes/inputs.3d.linear.askew-xz Tests/Slopes/inputs.3d.linear.askew-yz Tests/Slopes/main.cpp commit a94cd92ab7d73f72af9c46e2b1ac001689a4ae2f Author: Weiqun Zhang Date: Fri Aug 27 13:42:25 2021 -0700 Tweak GNU make on perlmutter (#2293) No need to throw error on `make clean`. Tools/GNUMake/sites/Make.nersc commit 806af6b0c49c6aa2e986e35f44287e952c1b637b Author: Weiqun Zhang Date: Fri Aug 27 11:58:04 2021 -0700 Use ParallelFor(Tag) in communication preparation. (#2292) Use ParallelFor(Tag) instead of the old kernel fusing approach in the initialization of masks in communication preparation. Src/Base/AMReX_FBI.H Src/Base/AMReX_PCI.H Src/Base/AMReX_TagParallelFor.H commit 6c804d92bc18e7ac53d479b0b06c674bfe02d9fc Author: Weiqun Zhang Date: Fri Aug 27 10:03:01 2021 -0700 Update inputs files of Advection_AmrLevel (#2291) There are no longer probin files. Tests/Amr/Advection_AmrLevel/Exec/SingleVortex/inputs.tracers Tests/Amr/Advection_AmrLevel/Exec/UniformVelocity/inputs Tests/Amr/Advection_AmrLevel/Exec/UniformVelocity/inputs-ci Tests/Amr/Advection_AmrLevel/Exec/UniformVelocity/inputs.regt Tools/RegressionTesting/AMReX-cuda-tests.ini Tools/RegressionTesting/AMReX-dpcpp-tests.ini Tools/RegressionTesting/AMReX-hip-tests.ini Tools/RegressionTesting/AMReX-tests.ini commit 1055b29661584f04f9270344033705b09119b625 Author: Weiqun Zhang Date: Fri Aug 27 09:03:12 2021 -0700 Optimization of thread safety check in building communication metadata (#2288) The previous approach is very slow for big boxes. Src/Base/AMReX_BoxArray.cpp Src/Base/AMReX_BoxList.cpp Src/Base/AMReX_FabArrayBase.cpp commit b20e88cff08fcbf6d92dc5b773dd25c096e355bc Author: kngott Date: Fri Aug 27 09:02:35 2021 -0700 Fix GNU builds on Perlmutter, + some bug fixes. (#2286) Tools/GNUMake/Make.defs Tools/GNUMake/Make.rules Tools/GNUMake/sites/Make.nersc commit ef4e008b231fe3c6707d61c75c1da5e4e6f68420 Author: Erik Date: Fri Aug 27 11:46:06 2021 -0400 Short range particles corr (#2271) Docs/sphinx_documentation/source/Particle.rst Docs/sphinx_documentation/source/Visualization.rst Docs/sphinx_documentation/source/index.rst commit 85b8d544e22984096113e85eef6f2d3b56e30d54 Author: Andrew Myers Date: Thu Aug 26 18:27:31 2021 -0700 Silence uninitialized value warning in valgrind in neighbor list construction. (#2287) I don't think this is an actual bug, but it's good to play nice with valgrind. Thanks to Bruce Palmer. Src/Particle/AMReX_NeighborList.H commit af631d7be0877acaf53c47ed01149a5cd557d1f2 Author: Corey Wetterer-Nelson <78513275+c-wetterer-nelson@users.noreply.github.com> Date: Thu Aug 26 21:20:44 2021 -0400 fix bug in AmrLevel exclusion logic (#2283) ## Summary Fixes a bug in the build logic regarding optional inclusion of the AMRLEVEL components, and return a required dependency to AMReX_AmrParticleDataAdaptor.H ## Additional background This bug was introduced in https://github.com/AMReX-Codes/amrex/pull/2258 where the SENSEI adaptors and bridges that processed data from the Amr class. A key dependency was removed, in the refactor. This Pr adds that dependency back in, and moves `AMReX_AmrParticleInSituBridge.H` behind the `AMReX_AMRLEVEL` compile option where it belongs. Src/Extern/SENSEI/AMReX_AmrParticleDataAdaptor.H Src/Extern/SENSEI/CMakeLists.txt commit 33d5e4d04c16386147ddad92b3203cd91b033e7a Author: Wileam Y. Phan <50928756+wyphan@users.noreply.github.com> Date: Thu Aug 26 17:09:03 2021 -0400 Port Tests/Amr/Advection_AmrLevel from Fortran to C++ GPU (#2268) ## Summary The Advection_AmrLevel test/tutorial has been fully ported from Fortran to C++ (except for `initdata()` subroutine that stays in Fortran). ## Additional background A good chunk of the code was copy-pasted and/or adapted from the AmrCore and GPU/CNS test/tutorials. All four tests {SingleVortex,UniformVelocity} x {2D,3D} have been tested to work on the CPU, NVIDIA GPU on Gigan, and AMD GPU on Spock (`fcompare` shows differences of the order of 1e-14). Co-authored-by: Weiqun Zhang Docs/sphinx_documentation/source/AmrLevel.rst Docs/sphinx_documentation/source/AmrLevel/figs/flowchart.odg Docs/sphinx_documentation/source/AmrLevel/figs/flowchart.png Tests/Amr/Advection_AmrCore/Exec/inputs-ci Tests/Amr/Advection_AmrLevel/CMakeLists.txt Tests/Amr/Advection_AmrLevel/Exec/Make.Adv Tests/Amr/Advection_AmrLevel/Exec/SingleVortex/Adv_prob.cpp Tests/Amr/Advection_AmrLevel/Exec/SingleVortex/Make.package Tests/Amr/Advection_AmrLevel/Exec/SingleVortex/Prob.f90 Tests/Amr/Advection_AmrLevel/Exec/SingleVortex/Prob_Parm.H Tests/Amr/Advection_AmrLevel/Exec/SingleVortex/face_velocity_2d.f90 Tests/Amr/Advection_AmrLevel/Exec/SingleVortex/face_velocity_2d_K.H Tests/Amr/Advection_AmrLevel/Exec/SingleVortex/face_velocity_3d.f90 Tests/Amr/Advection_AmrLevel/Exec/SingleVortex/face_velocity_3d_K.H Tests/Amr/Advection_AmrLevel/Exec/SingleVortex/inputs Tests/Amr/Advection_AmrLevel/Exec/SingleVortex/inputs-ci Tests/Amr/Advection_AmrLevel/Exec/SingleVortex/inputs.tracers Tests/Amr/Advection_AmrLevel/Exec/SingleVortex/probin Tests/Amr/Advection_AmrLevel/Exec/UniformVelocity/Adv_prob.cpp Tests/Amr/Advection_AmrLevel/Exec/UniformVelocity/Make.package Tests/Amr/Advection_AmrLevel/Exec/UniformVelocity/Prob.f90 Tests/Amr/Advection_AmrLevel/Exec/UniformVelocity/Prob_Parm.H Tests/Amr/Advection_AmrLevel/Exec/UniformVelocity/face_velocity_2d.f90 Tests/Amr/Advection_AmrLevel/Exec/UniformVelocity/face_velocity_2d_K.H Tests/Amr/Advection_AmrLevel/Exec/UniformVelocity/face_velocity_3d.f90 Tests/Amr/Advection_AmrLevel/Exec/UniformVelocity/face_velocity_3d_K.H Tests/Amr/Advection_AmrLevel/Exec/UniformVelocity/inputs Tests/Amr/Advection_AmrLevel/Exec/UniformVelocity/inputs-ci Tests/Amr/Advection_AmrLevel/Exec/UniformVelocity/inputs.regt Tests/Amr/Advection_AmrLevel/Exec/UniformVelocity/probdata.f90 Tests/Amr/Advection_AmrLevel/Exec/UniformVelocity/probin Tests/Amr/Advection_AmrLevel/README Tests/Amr/Advection_AmrLevel/Source/Adv.cpp Tests/Amr/Advection_AmrLevel/Source/Adv_F.H Tests/Amr/Advection_AmrLevel/Source/AmrLevelAdv.H Tests/Amr/Advection_AmrLevel/Source/AmrLevelAdv.cpp Tests/Amr/Advection_AmrLevel/Source/Kernels.H Tests/Amr/Advection_AmrLevel/Source/Make.package Tests/Amr/Advection_AmrLevel/Source/Src_2d/Adv_2d.f90 Tests/Amr/Advection_AmrLevel/Source/Src_2d/Make.package Tests/Amr/Advection_AmrLevel/Source/Src_2d/compute_flux_2d.f90 Tests/Amr/Advection_AmrLevel/Source/Src_2d/slope_2d.f90 Tests/Amr/Advection_AmrLevel/Source/Src_3d/Adv_3d.f90 Tests/Amr/Advection_AmrLevel/Source/Src_3d/Make.package Tests/Amr/Advection_AmrLevel/Source/Src_3d/compute_flux_3d.f90 Tests/Amr/Advection_AmrLevel/Source/Src_3d/slope_3d.f90 Tests/Amr/Advection_AmrLevel/Source/Src_K/Adv_K.H Tests/Amr/Advection_AmrLevel/Source/Src_K/Make.package Tests/Amr/Advection_AmrLevel/Source/Src_K/flux_2d_K.H Tests/Amr/Advection_AmrLevel/Source/Src_K/flux_3d_K.H Tests/Amr/Advection_AmrLevel/Source/Src_K/slope_K.H Tests/Amr/Advection_AmrLevel/Source/Src_K/tagging_K.H Tests/Amr/Advection_AmrLevel/Source/Src_nd/Adv_nd.f90 Tests/Amr/Advection_AmrLevel/Source/Src_nd/Make.package Tests/Amr/Advection_AmrLevel/Source/Src_nd/Tagging_nd.f90 Tests/Amr/Advection_AmrLevel/Source/Src_nd/tagging_params.f90 Tests/Amr/Advection_AmrLevel/Source/Tagging_params.cpp Tests/Amr/Advection_AmrLevel/Source/bc_nullfill.cpp Tests/GPU/CNS/Source/CNS.H commit cc16557fbcbd9e023272c9db8830d5b837265e10 Author: Weiqun Zhang Date: Thu Aug 26 11:47:44 2021 -0700 Use ROCm 4.3 in CI (#2280) * Use ROCm 4.3 in CI Since the VOP bug has been fixed in the latest ROCm release v4.3, CI can use the latest ROCm release now instead of 4.1. * Work-Around: OpenMPI 4.0.3 on Ubuntu 20.04 "mpic++ --showme" forgets to link open-pal ``` /opt/rocm/llvm/bin/clang++ -pthread CMakeFiles/cmTC_abf94.dir/test_mpi.cpp.o -o cmTC_abf94 -Wl,-rpath,/usr/lib/x86_64-linux-gnu/openmpi/lib /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi_cxx.so /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi.so ld.lld: error: /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi_cxx.so: undefined reference to opal_class_init_epoch [--no-allow-shlib-undefined] ld.lld: error: /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi_cxx.so: undefined reference to opal_list_item_t_class [--no-allow-shlib-undefined] ld.lld: error: /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi_cxx.so: undefined reference to opal_class_initialize [--no-allow-shlib-undefined] ld.lld: error: /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi_cxx.so: undefined reference to opal_uses_threads [--no-allow-shlib-undefined] ``` Results in `MPI_CXX` being not found. Co-authored-by: Axel Huebl .github/workflows/dependencies/dependencies_hip.sh .github/workflows/hip.yml commit 44d7b56878609c544b0f2906691786fc4d41633e Author: Weiqun Zhang Date: Thu Aug 26 08:04:52 2021 -0700 ParallelFor(Tag) (#2281) We have been using a ParallelFor function template that takes a Vector of Tags in communication functions. It launches a single GPU kernel for an irregular iteration space. This function is useful in many situations. Therefore, it is moved from a FillBoundary implementation header to its own header. And we have added two more versions of this for 1d and 3d loops. Src/Base/AMReX_FBI.H Src/Base/AMReX_FabArray.H Src/Base/AMReX_TagParallelFor.H Src/Base/CMakeLists.txt Src/Base/Make.package commit fc6dcb883c4460523b7ae8164a2e26922511b857 Author: Andrew Myers Date: Wed Aug 25 17:08:44 2021 -0700 remove faulty assertion from PC::Restart() (#2279) Src/Particle/AMReX_ParticleIO.H commit 298b4892d082bf281b7cb4bab9423991e059bf3a Author: Andrew Myers Date: Wed Aug 25 16:05:06 2021 -0700 Protect against nullptr when calling PODVector dtor. (#2272) Src/Base/AMReX_GpuAllocators.H Src/Base/AMReX_PODVector.H commit 44edcc104f551b6243984b567ccd6723ac336699 Author: Corey Wetterer-Nelson <78513275+c-wetterer-nelson@users.noreply.github.com> Date: Wed Aug 25 18:18:35 2021 -0400 remove amr adaptor/bridge when AMRLEVEL is off (#2258) Src/Extern/SENSEI/AMReX_AmrParticleDataAdaptor.H Src/Extern/SENSEI/AMReX_InSituBridge.cpp Src/Extern/SENSEI/CMakeLists.txt commit cbfe59d9d11140326e943681a8675e072cfedb91 Author: Weiqun Zhang Date: Wed Aug 25 15:17:06 2021 -0700 Parser: Hide local symbols (#2274) Previously `Parser::symbols()` returned a list of symbols including local symbols. This is problematic because WarpX makes sure that all symbols are either a registered variable or constant. But the local variables are unknown to WarpX. In this commit, the local symbols are excluded from the return value of `Parser::symbols()`. Src/Base/Parser/AMReX_IParser_Y.H Src/Base/Parser/AMReX_IParser_Y.cpp Src/Base/Parser/AMReX_Parser_Y.H Src/Base/Parser/AMReX_Parser_Y.cpp commit 1e5492c974dc866749e0557f329a32250e38e524 Author: Erik Date: Wed Aug 25 17:14:46 2021 -0400 Doc macros (#2275) ## Summary Convert some comments on macros to Doxygen style for inclusion in the docs. ## Additional background I did not see a straightforward way to include comments on the variables set at compile time. Docs/Doxygen/doxygen.conf Src/Base/AMReX_SPACE.H commit 742069a0445fd511a5c64271d4ca21f22bab9b59 Author: Wileam Y. Phan <50928756+wyphan@users.noreply.github.com> Date: Wed Aug 25 17:12:51 2021 -0400 Add EditorConfig section for rst files (#2273) ## Summary Add a section for `*.rst` files for EditorConfig. ## Additional background When editing the Sphinx `rst` source files for AmrLevel (PR #2268), I found out that the root `.editorconfig` file doesn't have a section for `rst` files. This PR adds that section, roughly based on the guidelines at https://docs.typo3.org/m/typo3/docs-how-to-document/master/en-us/GeneralConventions/CodingGuidelines.html . Note that I haven't gauged the "disruption" from whitespace changes that might be caused by this PR. .editorconfig commit 83de0324465f1f62d04c51a43e7212be348ad833 Author: Andrew Myers Date: Wed Aug 25 12:04:18 2021 -0700 Fix bug in the dual grid PC restart in the case where there are no particles on some levels. (#2276) Src/Particle/AMReX_ParticleIO.H commit bc3ad8347420a188838159b52d9bdcc5e9407060 Author: Weiqun Zhang Date: Tue Aug 24 10:59:59 2021 -0700 MF ParallelFor: LinearSolvers (#2267) Replace some `*_FUSIBLE` macros with MF ParallelFor. Note that the FUSIBLE macros are still used in some boundary functions. Src/Boundary/AMReX_FabSet.H Src/Boundary/AMReX_MultiMask.H Src/LinearSolvers/MLMG/AMReX_MLABecLap_1D_K.H Src/LinearSolvers/MLMG/AMReX_MLABecLap_2D_K.H Src/LinearSolvers/MLMG/AMReX_MLABecLap_3D_K.H Src/LinearSolvers/MLMG/AMReX_MLABecLaplacian.cpp Src/LinearSolvers/MLMG/AMReX_MLCellABecLap.cpp Src/LinearSolvers/MLMG/AMReX_MLCellABecLap_1D_K.H Src/LinearSolvers/MLMG/AMReX_MLCellABecLap_2D_K.H Src/LinearSolvers/MLMG/AMReX_MLCellABecLap_3D_K.H Src/LinearSolvers/MLMG/AMReX_MLCellLinOp.cpp Src/LinearSolvers/MLMG/AMReX_MLEBABecLap.cpp Src/LinearSolvers/MLMG/AMReX_MLEBABecLap_F.cpp Src/LinearSolvers/MLMG/AMReX_MLNodeLaplacian_sten.cpp Src/LinearSolvers/MLMG/AMReX_MLPoisson.cpp Src/LinearSolvers/MLMG/AMReX_MLPoisson_1D_K.H Src/LinearSolvers/MLMG/AMReX_MLPoisson_2D_K.H commit 57618b0d0a46f9321cd558a2707503ce128bbe82 Author: Weiqun Zhang Date: Mon Aug 23 19:32:32 2021 -0700 Fix FabArray move-ctor and move-= (#2270) A bug was introduced to the move ctor and move operator= in the MF ParallelFor PR #2073. The cached arrays were not properly moved. Src/Base/AMReX_FabArray.H commit 6df13f8bb81caf4fdf9cda141e20994cd83018b2 Author: Weiqun Zhang Date: Mon Aug 23 12:49:57 2021 -0700 Fine-tune FabArray::setBndry (#2269) Use either ParallelFor(MF) or ParallelFor(tag) depending on box sizes. Src/Base/AMReX_FabArray.H commit 60578291de339964af938b81f2d0f7db5c6bf78c Author: Weiqun Zhang Date: Fri Aug 20 19:02:33 2021 -0700 MF ParallelFor: PETSc (#2262) * Use the new MF ParallelFor in PETSc setup. * Fix some existing issues in PETSc setup, including compilation in 3D and with CUDA. Src/Base/AMReX_MFParallelForG.H Src/Extern/HYPRE/AMReX_Habec_3D_K.H Src/Extern/PETSc/AMReX_PETSc.H Src/Extern/PETSc/AMReX_PETSc.cpp Tools/GNUMake/packages/Make.petsc commit e488c939316a2d52f0c4d3394689c4c154c6deb4 Author: Jean M. Sexton Date: Fri Aug 20 16:31:20 2021 -0700 Add new machine to GNUMake based on Make.olcf (#2264) Tools/GNUMake/Make.machines Tools/GNUMake/sites/Make.frontier-coe commit 6c5909a99ca327ac68aef9f2233bf4ff41de5248 Author: Weiqun Zhang Date: Fri Aug 20 10:29:11 2021 -0700 MF ParallelFor: HYPRE (#2257) * Use the new MF ParallelFor in HYPRE setup and remove the use of fusible macros. * Fix typos Src/Base/AMReX_LayoutData.H Src/Extern/HYPRE/AMReX_HypreABecLap.cpp Src/Extern/HYPRE/AMReX_HypreABecLap2.cpp Src/Extern/HYPRE/AMReX_HypreABecLap3.cpp Src/Extern/PETSc/AMReX_PETSc.cpp commit 8f3d3511812d78ae5d5ca4f2620331cf8e3f3faa Author: Erik Date: Fri Aug 20 11:53:39 2021 -0400 Template wording (#2261) Changes the wording for the last checkbox on this pull request template. .github/pull_request_template.md commit 5e06ff8487368901c5fa79e5c7e8c944c62f7096 Author: Erik Date: Fri Aug 20 11:52:06 2021 -0400 Update contributing md (#2259) Adds style guide and short doxygen example to CONTRIBUTING.md CONTRIBUTING.md commit f840732f77100da5ebd40605a06e44ec44799516 Author: Weiqun Zhang Date: Thu Aug 19 12:27:44 2021 -0700 Fix #2256 (#2260) In #2256, a no-allocated MultiFab was passed to the MF ParalleFor. This resulted in segfault because of how the ParallelFor was implemented. In this commit, we fix the bug by reimplementing it without relying on Fabs being allocated. Src/Base/AMReX_FabArrayBase.H Src/Base/AMReX_FabArrayBase.cpp Src/Base/AMReX_MFParallelForG.H commit ebd1bac74492dc90def7fadea9e15ea77cb0e7a9 Author: Weiqun Zhang Date: Wed Aug 18 15:20:27 2021 -0700 MF ParallelFor: MultiFabUtil (#2256) Use the new MF ParallelFor in some MultiFabUtil functions. Src/Base/AMReX_MultiFabUtil.cpp Src/Base/AMReX_MultiFabUtil_1D_C.H Src/Base/AMReX_MultiFabUtil_2D_C.H Src/Base/AMReX_MultiFabUtil_3D_C.H Src/EB/AMReX_EBMultiFabUtil.cpp Src/F_Interfaces/Octree/AMReX_octree_fi.cpp commit 989d98b849ebaf31f000f60ff48ffd09304ff621 Author: Weiqun Zhang Date: Tue Aug 17 15:02:01 2021 -0700 Fix Particle CI Tests (#2252) A number of CI tests had issues with mixing single precision particle data with double precision amrex::Real. .github/workflows/linux.yml Src/AmrCore/AMReX_MFInterp_1D_C.H Src/Particle/AMReX_ParticleInit.H Tests/Particles/AssignDensity/main.cpp Tests/Particles/AssignMultiLevelDensity/main.cpp Tests/Particles/GhostsAndVirtuals/main.cpp Tests/Particles/ParallelContext/main.cpp Tests/Particles/ParticleMesh/main.cpp Tests/Particles/ParticleMeshMultiLevel/main.cpp Tests/Particles/ParticleReduce/main.cpp commit 4f5800cce8396eda1b4692faabe05fc26273e738 Author: Weiqun Zhang Date: Tue Aug 17 15:01:19 2021 -0700 Github Action Concurrency (#2254) Cancel previous jobs in progress if a PR is updated. .github/workflows/cuda.yml .github/workflows/docs.yml .github/workflows/hip.yml .github/workflows/intel.yml .github/workflows/linux.yml .github/workflows/macos.yml .github/workflows/style.yml .github/workflows/windows.yml commit ba6a7467e5235616be6730384302ed3b684eea05 Author: Erik Date: Tue Aug 17 16:44:25 2021 -0400 Add live links (#2255) * correct sp typo * spelling and comma * Add live links. Some other small formatting. Docs/sphinx_documentation/source/AmrLevel_Chapter.rst Docs/sphinx_documentation/source/EB.rst Docs/sphinx_documentation/source/Fortran.rst Docs/sphinx_documentation/source/GPU.rst Docs/sphinx_documentation/source/GPU_Chapter.rst Docs/sphinx_documentation/source/IO.rst Docs/sphinx_documentation/source/LinearSolvers.rst Docs/sphinx_documentation/source/LinearSolvers_Chapter.rst Docs/sphinx_documentation/source/Particle.rst Docs/sphinx_documentation/source/SWFFT.rst commit b97214dbeaf2a799a56c802a7e07d1265bcbaf5a Author: Weiqun Zhang Date: Tue Aug 17 13:12:28 2021 -0700 MF ParallelFor: MultiFab (#2249) Use the new ParallelFor in some MultiFab functions. Src/Base/AMReX_MultiFab.cpp commit 4f1ebf2213c396f077dd7c70fdf8c9e65e8caff2 Author: Andrew Myers Date: Tue Aug 17 12:10:18 2021 -0700 Reimplement AddParticlesAtLevel for GPU (#2236) * Reimplement AddParticlesAtLevel for GPU * Don't assume grid 0, tile 0 is empty. Src/Particle/AMReX_ParticleContainerI.H commit 48bad74b9ca0642fa794925156d053fd14e3a93e Author: Lucas Esclapez <13371051+esclapez@users.noreply.github.com> Date: Tue Aug 17 12:05:35 2021 -0700 Enable mixed (face) GpuBndryFuncFab (#2238) Add face-centered Fill function. Only foextrap, hoextrap, reflectodd/even supported right now. Src/Base/AMReX_FilCC_1D_C.H Src/Base/AMReX_FilCC_2D_C.H Src/Base/AMReX_FilCC_3D_C.H Src/Base/AMReX_FilFC_1D_C.H Src/Base/AMReX_FilFC_2D_C.H Src/Base/AMReX_FilFC_3D_C.H Src/Base/AMReX_FilFC_C.H Src/Base/AMReX_PhysBCFunct.H Src/Base/CMakeLists.txt Src/Base/Make.package commit 68327bc3ccfbb9bf31e467d479db6a854626e071 Author: Erik Date: Tue Aug 17 13:38:22 2021 -0400 Remove eb doc section (#2253) Docs/sphinx_documentation/source/EB.rst Docs/sphinx_documentation/source/EB/areas_and_volumes.fig Docs/sphinx_documentation/source/EB/areas_and_volumes.png Docs/sphinx_documentation/source/EB/eb_fluxes.fig Docs/sphinx_documentation/source/EB/eb_fluxes.png Docs/sphinx_documentation/source/EB/redist.fig Docs/sphinx_documentation/source/EB/redist.png commit 9a97484ad7940bfa9a1aa9064f07cc326992196a Author: Jean M. Sexton Date: Tue Aug 17 10:31:52 2021 -0700 Make sure all ba are defined in test, and only call Init once per PC (#2247) * Change boxes so all ba are defined, and use a new ParticleContainer for initializing tests * Remove duplicate workaround * GhostsAndVirtuals: Cleanup - Safety: use `.at()` over `operator[]` for performance-uncritical code - Some formatting * CI: Test Particles_GhostsAndVirtuals Co-authored-by: Axel Huebl .github/workflows/linux.yml .github/workflows/macos.yml Tests/Particles/GhostsAndVirtuals/main.cpp commit 2788e558156219f84840f0ab73d42dcfd344f21e Author: Weiqun Zhang Date: Mon Aug 16 18:23:23 2021 -0700 Shorten CI tests and Enable CNS in CI (#2251) * Split macos ci tests into two checks * Split hip ci tests * Change some build types from Debug to RelWithDebInfo * Enable Assertions and FPE in some tests * Add some inputs for ci * Enable CNS in CI .github/workflows/hip.yml .github/workflows/intel.yml .github/workflows/linux.yml .github/workflows/macos.yml Tests/Amr/Advection_AmrCore/CMakeLists.txt Tests/Amr/Advection_AmrCore/Exec/inputs-ci Tests/Amr/Advection_AmrLevel/CMakeLists.txt Tests/Amr/Advection_AmrLevel/Exec/SingleVortex/inputs-ci Tests/Amr/Advection_AmrLevel/Exec/UniformVelocity/inputs-ci Tests/EB/CNS/CMakeLists.txt Tests/EB/CNS/Exec/Sod/inputs-ci Tests/EB_CNS/CMakeLists.txt Tests/EB_CNS/Exec/Combustor/inputs-ci Tests/GPU/CNS/CMakeLists.txt Tests/GPU/CNS/Exec/Sod/inputs-ci Tests/LinearSolvers/NodalPoisson/CMakeLists.txt Tests/LinearSolvers/NodalPoisson/inputs-ci commit 79a57600d5eec3c9a0ffdbb717462728a1d56598 Author: Weiqun Zhang Date: Mon Aug 16 10:22:17 2021 -0700 Use the new ParallelFor in some FabArrayUtility functions (#2246) Src/Base/AMReX_FabArrayUtility.H commit a4f94e7524c0348af134968a9e17c7fd08f266f7 Author: Weiqun Zhang Date: Sun Aug 15 15:31:49 2021 -0700 Use new ParallelFor in FabArray (#2245) * Add FabArrayBase::isFusingCandidate() that is fine tuned for MI100, V100 and A100. * Convert some FabArray functions to using the new ParallelFor. Src/Base/AMReX_FabArray.H Src/Base/AMReX_FabArrayBase.H Src/Base/AMReX_FabArrayBase.cpp commit c94cbd9b78cc8fa559c00c45c49b9add0075cc0f Author: Axel Huebl Date: Sat Aug 14 18:50:47 2021 -0700 Fix: Slopes K Literals (#2250) * Fix: Slopes K Literals Address compile issues with GCC 9.3.0 (Manda) / 10.1.0 (Battra). Seen with GNUmake by Ann. * Slopes K: Cleanup - Add used headers - move into regular amrex:: namespace - simplify class usage in amrex:: namespace Src/Base/AMReX_Slopes_K.H commit 9f58fc5c936129424f2a64d5c2c4e4de243bc3b5 Author: Axel Huebl Date: Sat Aug 14 12:10:51 2021 -0700 CI: Build Embedded Boundaries (#2232) - [x] Add coverage for EB in CI - [x] fix DPC++ single precision build - [x] ~~fix `LinearSolvers` array bounds w/ GCC 7.5~~ -> false positive Co-authored-by: Weiqun Zhang .github/workflows/cuda.yml .github/workflows/hip.yml .github/workflows/intel.yml .github/workflows/linux.yml .github/workflows/macos.yml .github/workflows/windows.yml Src/Base/AMReX_Slopes_K.H Src/EB/AMReX_EBMultiFabUtil_2D_C.H Src/EB/AMReX_EBMultiFabUtil_3D_C.H Src/EB/AMReX_EB_utils.cpp Src/LinearSolvers/MLMG/AMReX_MLEBABecLap_2D_K.H Src/LinearSolvers/MLMG/AMReX_MLEBABecLap_3D_K.H Src/LinearSolvers/MLMG/AMReX_MLNodeLaplacian_eb.cpp Tests/EB/CNS/Source/CNS.H Tests/EB_CNS/Source/hydro/CNS_divop_K.H commit cc89a4bc60b54902f4b07f531a03cd2c1b834e58 Author: Andrew Myers Date: Fri Aug 13 19:34:28 2021 -0700 Also remove empty map entries when calling clearParticles (#2248) Src/Particle/AMReX_ParticleContainerI.H Src/Particle/AMReX_ParticleUtil.H Src/Particle/AMReX_Particles.H commit e8512831302f27820e0e6a0714f83f3905e89895 Author: Maxim Shatsky Date: Fri Aug 13 19:29:28 2021 +0300 Bug fix of rarely met issue from Tools/CMake/AMReXUtils.cmake file (#2244) Everything is already described here: https://github.com/AMReX-Codes/amrex/issues/2240 Tools/CMake/AMReXUtils.cmake commit 090596bd4241497616cfbb86f168556d539ceb42 Author: Erik Date: Thu Aug 12 16:30:34 2021 -0400 Add definition of MLMG (#2242) Docs/sphinx_documentation/source/LinearSolvers.rst Src/Base/AMReX_MultiFab.H commit fb39fa7ac33ed916f42cc26d134c377113e78fa6 Author: Weiqun Zhang Date: Thu Aug 12 12:40:37 2021 -0700 Move MF ParallelFor out of experimental:: (#2241) It's still available in experimental::, but it will be removed from it eventually. Src/AmrCore/AMReX_MFInterpolater.cpp Src/Base/AMReX_MFParallelFor.H Src/EB/AMReX_EBMFInterpolater.cpp Src/LinearSolvers/MLMG/AMReX_MLNodeLaplacian.cpp Src/LinearSolvers/MLMG/AMReX_MLNodeLaplacian_misc.cpp commit 9a7688371d000798c71a3ca246bd983bc9d300b6 Author: Andrew Myers Date: Thu Aug 12 08:58:11 2021 -0700 Fix some tests so that ctests works with CUDA enabled. (#2239) This also fixes and scoping bug in the particle iterator test. Tests/LinearSolvers/NodeTensorLap/CMakeLists.txt Tests/Particles/ParallelContext/CMakeLists.txt Tests/Particles/ParallelContext/inputs.rt.cuda Tests/Particles/ParticleIterator/main.cpp Tests/Particles/Redistribute/CMakeLists.txt commit d39e0b86813852d6c2a38f57f6243cdb750e08df Author: Weiqun Zhang Date: Thu Aug 12 08:53:50 2021 -0700 Option for dynamic tiling in CPU build of MF ParallelFor (#2235) Examples: experimental::ParallelFor(mf, nghost, TileSize{my_tile_size}, DynamicTiling{true}, [=] ...); experimental::ParallelFor(mf, nghost, ncomp, TileSize{my_tile_size}, DynamicTiling{true}, [=] ...); Co-authored-by: Andrew Myers Src/Base/AMReX_MFParallelFor.H Src/Base/AMReX_MFParallelForC.H Src/Base/AMReX_MFParallelForG.H commit 694279b0fb23b90a40ac8ba93fe5ed90b8510123 Author: Wileam Y. Phan <50928756+wyphan@users.noreply.github.com> Date: Wed Aug 11 21:55:14 2021 -0400 Add sum(), product(), size(), {x,y,z}{lo,hi,len}() member functions to GpuArray and Array?D (#2217) Now `GpuArray` and `Array1`/`2`/`3D` has member functions `sum()` and `product()`. These functions will take the sum or product over the entire array. For Array2D, both can optionally take in `int axis` and `int loc`. Same goes for `Array3D`, but it takes two positions. The following static member functions to return array bounds and sizes are added: * `Array1D`: `lo()`, `hi()`, `len()`, `size()` * `Array2D`: `xlo()`, `xhi()`, `xlen()`, `ylo()`, `yhi()`, `ylen()`, `size()` * `Array3D`:`xlo()`, `xhi()`, `xlen()`, `ylo()`, `yhi()`, `ylen()`, `zlo()`, `zhi()`, `zlen()`, `size()` The `*lo()` and `*hi()` functions return an `int`, and the `*len()` (per direction) and `size()` (total in all directions) functions return a `uint`. In addition, the `begin()` and `end()` member functions have also been added for `Array?D`, which return the start pointer address and the pointer address right after the last element, for iterator purposes, if needed. For `Array2`/`3D`, it's implemented as if the array is one-dimensional. Co-authored-by: etpalmer63 Co-authored-by: Weiqun Zhang Src/Base/AMReX_Array.H commit e82d45264b7840ff9b6247c4a7eb534f55ed5233 Author: Weiqun Zhang Date: Wed Aug 11 10:55:13 2021 -0700 Fix derive data in AmrLevel::writePlotFile (#2234) Previously, it only worked with single-component DeriveList. Src/Amr/AMReX_AmrLevel.cpp Src/Amr/AMReX_Derive.H Src/Amr/AMReX_Derive.cpp Tests/GPU/CNS/Exec/RT/inputs Tests/GPU/CNS/Exec/RT/inputs-rt Tests/GPU/CNS/Exec/Sod/inputs Tests/GPU/CNS/Exec/Sod/inputs-rt Tests/GPU/CNS/Source/CNS_derive.cpp Tests/GPU/CNS/Source/CNS_setup.cpp commit f4c2629340bf5fdac41d869321e0514f68654789 Author: Weiqun Zhang Date: Tue Aug 10 19:41:27 2021 -0700 Add MFNodeBilinear for nodal linear interpolation (#2202) Src/Amr/AMReX_AmrLevel.cpp Src/AmrCore/AMReX_Interp_3D_C.H Src/AmrCore/AMReX_Interpolater.cpp Src/AmrCore/AMReX_MFInterp_1D_C.H Src/AmrCore/AMReX_MFInterp_2D_C.H Src/AmrCore/AMReX_MFInterp_3D_C.H Src/AmrCore/AMReX_MFInterpolater.H Src/AmrCore/AMReX_MFInterpolater.cpp commit 0ca401daa57fd80e7ae7a12eb5fd28b9d10ce11b Author: PaulMullowney <60452402+PaulMullowney@users.noreply.github.com> Date: Tue Aug 10 11:33:23 2021 -0600 Adding experimental parfor into the FSmooth method (#2230) Co-authored-by: Paul Mullowney Src/LinearSolvers/MLMG/AMReX_MLNodeLaplacian_misc.cpp commit 075f30f8ad323c2a8726e35d52ff70f53c681a49 Author: Weiqun Zhang Date: Tue Aug 10 10:15:07 2021 -0700 Parser: Exception handling (#2233) For unknown characters, syntax errors, and unknown variables, throw exceptions with hopefully more helpful messages. Src/Base/Parser/AMReX_IParser.H Src/Base/Parser/AMReX_IParser.cpp Src/Base/Parser/AMReX_IParser_Exe.cpp Src/Base/Parser/AMReX_IParser_Y.cpp Src/Base/Parser/AMReX_Parser.H Src/Base/Parser/AMReX_Parser.cpp Src/Base/Parser/AMReX_Parser_Exe.cpp Src/Base/Parser/AMReX_Parser_Y.cpp Src/Base/Parser/amrex_iparser.l Src/Base/Parser/amrex_iparser.lex.cpp Src/Base/Parser/amrex_parser.l Src/Base/Parser/amrex_parser.lex.cpp commit 0528d849c6c3d17193263f1c4aa15a6b07e7ed1a Author: Weiqun Zhang Date: Mon Aug 9 18:12:57 2021 -0700 Parser: Noinline transcendental functions (#2229) It's too expensive to have them inlined. Src/Base/Parser/AMReX_Parser_Y.H commit a2ad6250aabed7042f33b7c5356274104165ef4b Author: asalmgren Date: Sun Aug 8 13:50:59 2021 -0700 Fix 2d eb slope limiting (#2226) There was a bug in the 2D EB slope limiting -- this fixes that Src/EB/AMReX_EB_slopes_K.H commit 867d1deb3b62ca99ce50a95eb190a548eb119112 Author: asalmgren Date: Sat Aug 7 13:22:58 2021 -0700 New test eb cns (#2208) This directory Tests/EB_CNS is designed to replace Tests/EB/CNS -- it is a pure-C++ version of EB/CNS and matches the current regression tests to sufficient precision. It is not GPU-ready. Tests/EB_CNS/CMakeLists.txt Tests/EB_CNS/Exec/Combustor/CNS_bcfill.cpp Tests/EB_CNS/Exec/Combustor/GNUmakefile Tests/EB_CNS/Exec/Combustor/Make.package Tests/EB_CNS/Exec/Combustor/cns_prob.H Tests/EB_CNS/Exec/Combustor/cns_prob.cpp Tests/EB_CNS/Exec/Combustor/cns_prob_parm.H Tests/EB_CNS/Exec/Combustor/cns_prob_parm.cpp Tests/EB_CNS/Exec/Combustor/inputs Tests/EB_CNS/Exec/Combustor/inputs.regt Tests/EB_CNS/Exec/Make.CNS Tests/EB_CNS/Exec/Pulse/GNUmakefile Tests/EB_CNS/Exec/Pulse/Make.package Tests/EB_CNS/Exec/Pulse/cns_prob.H Tests/EB_CNS/Exec/Pulse/cns_prob.cpp Tests/EB_CNS/Exec/Pulse/cns_prob_parm.H Tests/EB_CNS/Exec/Pulse/inputs Tests/EB_CNS/Exec/Pulse/inputs.regt Tests/EB_CNS/Exec/ShockRef/GNUmakefile Tests/EB_CNS/Exec/ShockRef/Make.package Tests/EB_CNS/Exec/ShockRef/cns_prob.H Tests/EB_CNS/Exec/ShockRef/cns_prob.cpp Tests/EB_CNS/Exec/ShockRef/cns_prob_parm.H Tests/EB_CNS/Exec/ShockRef/inputs Tests/EB_CNS/Exec/ShockRef/inputs.amr Tests/EB_CNS/Exec/ShockRef/inputs.regt Tests/EB_CNS/Exec/Sod/GNUmakefile Tests/EB_CNS/Exec/Sod/Make.package Tests/EB_CNS/Exec/Sod/cns_prob.H Tests/EB_CNS/Exec/Sod/cns_prob.cpp Tests/EB_CNS/Exec/Sod/cns_prob_parm.H Tests/EB_CNS/Exec/Sod/inputs Tests/EB_CNS/Source/CNS.H Tests/EB_CNS/Source/CNS.cpp Tests/EB_CNS/Source/CNSBld.cpp Tests/EB_CNS/Source/CNS_K.H Tests/EB_CNS/Source/CNS_advance.cpp Tests/EB_CNS/Source/CNS_advance_box.cpp Tests/EB_CNS/Source/CNS_advance_box_eb.cpp Tests/EB_CNS/Source/CNS_bcfill.cpp Tests/EB_CNS/Source/CNS_derive.H Tests/EB_CNS/Source/CNS_derive.cpp Tests/EB_CNS/Source/CNS_index_macros.H Tests/EB_CNS/Source/CNS_init_eb2.cpp Tests/EB_CNS/Source/CNS_io.cpp Tests/EB_CNS/Source/CNS_parm.H Tests/EB_CNS/Source/CNS_parm.cpp Tests/EB_CNS/Source/CNS_setup.cpp Tests/EB_CNS/Source/CNS_tagging.H Tests/EB_CNS/Source/Make.package Tests/EB_CNS/Source/diffusion/CNS_diffusion_K.H Tests/EB_CNS/Source/diffusion/CNS_diffusion_eb_K.H Tests/EB_CNS/Source/diffusion/Make.package Tests/EB_CNS/Source/hydro/CNS_divop_K.H Tests/EB_CNS/Source/hydro/CNS_flux_redistribute.cpp Tests/EB_CNS/Source/hydro/CNS_hydro_K.H Tests/EB_CNS/Source/hydro/CNS_hydro_eb_K.H Tests/EB_CNS/Source/hydro/Make.package Tests/EB_CNS/Source/main.cpp commit eed6b0c572301d793c098c3dae7286010d046182 Author: PaulMullowney <60452402+PaulMullowney@users.noreply.github.com> Date: Sat Aug 7 12:08:36 2021 -0600 Adding experimental parfor into the restriction operator (#2204) Co-authored-by: Paul Mullowney Src/LinearSolvers/MLMG/AMReX_MLNodeLaplacian.cpp commit 660a17309966e07fdda41b66d7016e23795a846d Author: Weiqun Zhang Date: Sat Aug 7 10:52:26 2021 -0700 Fix PArena (#2221) In some PArena functions, we forgot to test if the memory pool is supported. ## Additional background https://github.com/AMReX-Astro/Castro/issues/1955 Src/Base/AMReX_PArena.cpp commit 468feb057586e06e069b22f62468c572f7172856 Author: Axel Huebl Date: Sat Aug 7 10:51:53 2021 -0700 Test Build Variant (#2223) The AMReX "test" build variant that is used by [regression tests](https://github.com/AMReX-Codes/regression_testing) by default was not documented, which leads to some confusion when debugging failing apps. This PR: - documents GNUmake's `TEST` & `USE_ASSERTION` options - adds a CMake option `AMReX_TESTING` that are in sync with the define `-DAMREX_TESTING` that changes FAB default values and other debug-like options. Docs/sphinx_documentation/source/BuildingAMReX.rst Tools/CMake/AMReXSetDefines.cmake commit 2c98061cb00d552ec6262cdffc475e4c0cc92445 Author: Weiqun Zhang Date: Sat Aug 7 10:44:29 2021 -0700 EB: Fix small cells and multicuts (#2225) The previous approach of fix small cells and multicuts is not consistent. If a small cell is turned into a covered cell, we need to fix its neighbors too. In this new approach, we build the EB data iteratively. This commit also introduces a new ParmParse parameter, eb2.maxiter, that controls the maximal number of iterations. Src/Base/AMReX_GpuBuffer.H Src/EB/AMReX_EB2_2D_C.cpp Src/EB/AMReX_EB2_3D_C.H Src/EB/AMReX_EB2_3D_C.cpp Src/EB/AMReX_EB2_C.H Src/EB/AMReX_EB2_GeometryShop.H Src/EB/AMReX_EB2_Level.H Src/EB/AMReX_EB2_MultiGFab.H Src/EB/AMReX_EB2_MultiGFab.cpp Src/EB/AMReX_EB2_ND_C.cpp Tests/LinearSolvers/CellEB/MyEB.H commit bca7ff1fdd2d670eda38c0304ebf8741f15b0873 Author: Jean M. Sexton Date: Thu Aug 5 13:55:59 2021 -0700 Make CreateVirtualParticles use GPU functions for aggregation_type=Cell (#2219) Src/Particle/AMReX_ParticleContainerI.H Tests/Particles/GhostsAndVirtuals/fixed_grids.init Tests/Particles/GhostsAndVirtuals/main.cpp commit cbd153a6d0b483b82b7ec5c540488cd5b97c5189 Author: asalmgren Date: Thu Aug 5 12:11:52 2021 -0700 Change getgradphi in nodalproj (#2222) * Add options to return const or non-const vector of pointers to phi and gradphi from nodal projector * oops -- left in extra const Src/LinearSolvers/Projections/AMReX_NodalProjector.H commit 6690a90f407c536c0acf43e6c8fc3862ab6d7097 Author: Jean M. Sexton Date: Tue Aug 3 16:48:21 2021 -0700 Add static_cast for ParticleReal, and check assertion (#2215) Src/Amr/AMReX_Derive.cpp Src/Particle/AMReX_NeighborParticlesCPUImpl.H Tests/Particles/AsyncIO/main.cpp Tests/Particles/NeighborParticles/MDParticleContainer.H Tests/Particles/NeighborParticles/MDParticleContainer.cpp Tests/Particles/NeighborParticles/main.cpp Tests/Particles/ParallelContext/main.cpp Tests/Particles/ParticleMesh/main.cpp Tests/Particles/ParticleReduce/main.cpp Tests/Particles/ParticleTransformations/main.cpp Tests/Particles/Redistribute/main.cpp commit 3b2c8de1ea1d8b67cb9910a142249c5cc6fe7e75 Author: Weiqun Zhang Date: Tue Aug 3 15:04:09 2021 -0700 Fix EBInterpolater (#2214) Fix bug in #2203. The fine factory may not contain information on boxes outside the fine target region. Src/EB/AMReX_EBInterpolater.cpp Src/EB/AMReX_EBMFInterpolater.cpp commit ac818cd22aded63eff8358ad54e0567232163341 Author: Jean M. Sexton Date: Tue Aug 3 14:49:40 2021 -0700 Add check for OpenMP::get_max_threads()=0 case for CI warning (#2212) Src/Base/AMReX_MultiFabUtil.cpp commit 5da293757b8bc67225702d3fe1ddc6fb0c010fa4 Author: Weiqun Zhang Date: Mon Aug 2 15:15:44 2021 -0700 Add EBMFCellConsLinInterp for EB cell data interpolation (#2203) Src/Amr/AMReX_StateDescriptor.H Src/AmrCore/AMReX_FillPatchUtil.H Src/AmrCore/AMReX_MFInterpolater.H Src/EB/AMReX_EBInterpolater.H Src/EB/AMReX_EBInterpolater.cpp Src/EB/AMReX_EBMFInterpolater.H Src/EB/AMReX_EBMFInterpolater.cpp Src/EB/CMakeLists.txt Src/EB/Make.package commit af968f828c51d7d6d07646c69fe8e53b85a7d6a1 Author: Weiqun Zhang Date: Mon Aug 2 14:05:11 2021 -0700 Bilinear interpolation for cell data (#2205) Add MFCellBilinear for cell data. Port CellBilinear to C++. Src/AmrCore/AMReX_INTERP_1D.F90 Src/AmrCore/AMReX_INTERP_2D.F90 Src/AmrCore/AMReX_INTERP_3D.F90 Src/AmrCore/AMReX_Interpolater.H Src/AmrCore/AMReX_Interpolater.cpp Src/AmrCore/AMReX_MFInterp_1D_C.H Src/AmrCore/AMReX_MFInterp_2D_C.H Src/AmrCore/AMReX_MFInterp_3D_C.H Src/AmrCore/AMReX_MFInterpolater.H Src/AmrCore/AMReX_MFInterpolater.cpp commit ee8f6828dd1162ef6bcf123c417b31abf63f4eef Author: Axel Huebl Date: Mon Aug 2 10:08:24 2021 -0700 CMake: Prebuild & SetupCUDA (#2209) ## Summary With CMake 3.20+, we don't need to include the `SetupCUDA.cmake` scripts anymore. ## Additional background Follow-up to #2012 Tools/CMake/AMReXConfig.cmake.in commit 2695be7cad80f2b4ec25abe57925cc166e3aa824 Author: Weiqun Zhang Date: Mon Aug 2 08:59:50 2021 -0700 Reduction of MulitFab to 1D data (#2182) Add `sumToLine` function that reduces MultiFab's data to a 1D line. Src/Base/AMReX_MultiFabUtil.H Src/Base/AMReX_MultiFabUtil.cpp commit b076fa56f696acc3591a2b6ee03cc6f981731cf4 Author: Weiqun Zhang Date: Mon Aug 2 08:59:15 2021 -0700 YAFluxRegister with blocking factor of 1 (#2155) Fix YAFluxRegister for the case that the blocking factor is 1. In that case, a coarse cell might have fine neighbors at both x-lo and x-hi faces. Src/Boundary/AMReX_YAFluxRegister_1D_K.H Src/Boundary/AMReX_YAFluxRegister_2D_K.H Src/Boundary/AMReX_YAFluxRegister_3D_K.H commit 3fdfac26cd238cfc0c8c4fd8a754dfd4b8cd3eeb Author: Weiqun Zhang Date: Mon Aug 2 08:32:44 2021 -0700 Fix lincc_interp (#2201) This fixes a bug in lincc_interp that was introduced more than 2 years ago when the Fortran code was converted to C++. The slope was mistakenly limited twice. For each component, we compute a limiting factor and we use the minimum of all components as the final limiting factor. That final limiting factor should be applied to the unlimited central difference slope, not the monotonized central slope. Src/AmrCore/AMReX_MFInterp_1D_C.H Src/AmrCore/AMReX_MFInterp_2D_C.H Src/AmrCore/AMReX_MFInterp_3D_C.H commit 4ee1fd09cfdda8ad975e8e62aa70772bfe2a7455 Author: Weiqun Zhang Date: Mon Aug 2 08:31:51 2021 -0700 Add MFPCInterp for piece-wise constant interpolation (#2200) Src/AmrCore/AMReX_MFInterpolater.H Src/AmrCore/AMReX_MFInterpolater.cpp commit fa89d68a8909c9630d0a9034ca27fda50137119c Author: Weiqun Zhang Date: Sun Aug 1 15:22:12 2021 -0700 Fix Coarse/Fine Boundary BoxArray generated in FPInfo for FillPatch (#2207) The issue was degenerate boxes could appear when the coarse/fine boxarray was split into smaller chunks. This is usually not an issue except for a small extra cost. But in IAMR this could result in external Dirichlet pressure boundary function that has not been implemented being called. Src/Base/AMReX_FabArrayBase.cpp commit 7034f81724850188d104e34c787c302d3d372c44 Author: Weiqun Zhang Date: Sun Aug 1 14:42:40 2021 -0700 Update CHANGES for 21.08 (#2206) CHANGES commit b87ec0a8fc084dbe886973e1e3c8a4eedf6b7dfa Author: kngott Date: Thu Jul 29 09:29:22 2021 -0700 Fix bulleted list in IO Docs. (#2198) Docs/sphinx_documentation/source/IO.rst commit fbdf1d043ed572d8388eb445a7e091d0a11acde9 Author: kngott Date: Thu Jul 29 08:54:02 2021 -0700 Add AsyncVector. (#2197) Src/Base/AMReX_GpuAllocators.H Src/Base/AMReX_GpuContainers.H Tests/GPU/Vector/main.cpp commit 0124f3b425adbe00e8ec021397d654eb411df9b0 Author: Jean M. Sexton Date: Thu Jul 29 08:08:47 2021 -0700 Make CreateVirtualParticles use GPU functions for aggregation_type=None (#2195) Src/Particle/AMReX_ParticleContainerI.H Src/Particle/AMReX_ParticleTransformation.H Src/Particle/AMReX_TracerParticle_mod_K.H commit 4387cf22f3a5417b17b2445b22fd3659fa0f1462 Author: Weiqun Zhang Date: Wed Jul 28 18:09:40 2021 -0700 Fix divide by zero due to roundoff errors (#2199) Src/AmrCore/AMReX_MFInterp_1D_C.H Src/AmrCore/AMReX_MFInterp_2D_C.H Src/AmrCore/AMReX_MFInterp_3D_C.H commit 5619ef246175454d1777cd5dc4cd0013ad2966dd Author: asalmgren Date: Wed Jul 28 13:42:55 2021 -0700 Change mask for ctr to ctroid interp (#2196) Src/EB/AMReX_EBMultiFabUtil.cpp commit 282878dfbf1c823a90f3265c318e15185676710a Author: Andrew Myers Date: Wed Jul 28 12:14:45 2021 -0700 Add OpenMP support to DenseBins::build (#2193) Src/Particle/AMReX_DenseBins.H Tests/Particles/DenseBins/CMakeLists.txt Tests/Particles/DenseBins/GNUmakefile Tests/Particles/DenseBins/Make.package Tests/Particles/DenseBins/inputs Tests/Particles/DenseBins/main.cpp commit 7ac29e889e4c0949eba6dffcc1e03ce27dc2751c Author: PaulMullowney <60452402+PaulMullowney@users.noreply.github.com> Date: Tue Jul 27 17:15:31 2021 -0600 Integration of experimental parfor into NodeLaplacian Fapply. (#2190) Co-authored-by: Paul Mullowney Src/LinearSolvers/MLMG/AMReX_MLNodeLaplacian_misc.cpp commit 5bdae60f411b47b2360c755ad21b7cc550a547d8 Author: kngott Date: Tue Jul 27 16:08:41 2021 -0700 Add some missing FabArray function timers. (#2194) Src/Base/AMReX_FabArray.H commit 35edeb12e000a6dd75765c65dcb6dd93b3cd340a Author: Andrew Myers Date: Tue Jul 27 14:12:08 2021 -0700 Fix filter functions for callables that return bool. (#2192) Src/Particle/AMReX_ParticleTransformation.H Tests/Particles/ParticleTransformations/main.cpp commit 16cdf1504d9e00b906053431e8037c9f6d5b3f9c Author: Andrew Myers Date: Tue Jul 27 14:11:56 2021 -0700 Reduce code duplication in DenseBins::build (#2191) Src/Particle/AMReX_DenseBins.H commit bdebb1055d3cc84d95e379833821572a579bb838 Author: Weiqun Zhang Date: Mon Jul 26 17:58:04 2021 -0700 MFCellConsLinInterp (#2189) Add MFCellConsLinInterp that is similar to CellConsLinInterp. The difference is that MFCellConsLinInterp works on MultiFab, whereas CellConsLinInterp works on FArrayBox. The new interpolater launches fewer kernels, therefore has better performance on GPU when there are many boxes. Src/Amr/AMReX_AmrLevel.H Src/Amr/AMReX_AmrLevel.cpp Src/Amr/AMReX_StateDescriptor.H Src/Amr/AMReX_StateDescriptor.cpp Src/AmrCore/AMReX_FillPatchUtil.H Src/AmrCore/AMReX_FillPatchUtil.cpp Src/AmrCore/AMReX_FillPatchUtil_I.H Src/AmrCore/AMReX_InterpBase.H Src/AmrCore/AMReX_InterpBase.cpp Src/AmrCore/AMReX_Interp_1D_C.H Src/AmrCore/AMReX_Interp_2D_C.H Src/AmrCore/AMReX_Interp_3D_C.H Src/AmrCore/AMReX_Interpolater.H Src/AmrCore/AMReX_Interpolater.cpp Src/AmrCore/AMReX_MFInterp_1D_C.H Src/AmrCore/AMReX_MFInterp_2D_C.H Src/AmrCore/AMReX_MFInterp_3D_C.H Src/AmrCore/AMReX_MFInterp_C.H Src/AmrCore/AMReX_MFInterpolater.H Src/AmrCore/AMReX_MFInterpolater.cpp Src/AmrCore/CMakeLists.txt Src/AmrCore/Make.package Src/Base/AMReX_FACopyDescriptor.H Src/Base/AMReX_MFParallelForG.H Src/LinearSolvers/MLMG/AMReX_MLNodeLaplacian_sync.cpp commit 3181f76910e6ac28ec5e099cba3e192bfdbf9076 Author: PaulMullowney <60452402+PaulMullowney@users.noreply.github.com> Date: Sun Jul 25 19:53:42 2021 -0400 Introduction of the experimental parfor into NodeLaplacian interpolation (#2183) All algorithms fully implemented including interpadd_aa, semi_interpadd_aa, interp_c, interp_rap, and interpadd_ha. Src/LinearSolvers/MLMG/AMReX_MLNodeLaplacian.cpp commit f35a59b079878a43e64418f36be356227e7fc1b9 Author: Andrew Myers Date: Sun Jul 25 15:55:32 2021 -0700 Enable default construction for AmrParticleContainer and add isDefined() method. (#2188) A Particle container is considered "defined" if it has been given a set of grids, otherwise not. Src/AmrCore/AMReX_AmrParticles.H Src/Particle/AMReX_ParticleContainerBase.H commit 45711e08a54d42ced219ffe00f6e8b435ebc7bf3 Author: jmsexton03 Date: Fri Jul 23 17:04:41 2021 -0700 Make CreateGhostParticles use GPU functions (#2185) Src/Particle/AMReX_ParticleContainerBase.H Src/Particle/AMReX_ParticleContainerI.H Src/Particle/AMReX_ParticleTransformation.H Tests/Particles/GhostsAndVirtuals/CMakeLists.txt Tests/Particles/GhostsAndVirtuals/fixed_grids.init Tests/Particles/GhostsAndVirtuals/inputs Tests/Particles/GhostsAndVirtuals/main.cpp Tests/Particles/GhostsAndVirtuals/particle_file.init commit 27069ef1d885a35401f963ead2e3bad19cc9dc8f Author: Axel Huebl Date: Thu Jul 22 08:47:39 2021 -0700 CI: HIP Cleanup (#2186) Move the PATH setup for the AMD `clang`/`clang++` executables to a common location for the setup scripts. As a reminder, this is needed so the system clang is not taken by accident. .github/workflows/dependencies/dependencies_hip.sh .github/workflows/hip.yml commit 91fa2b7e6ff33117bd537635c8c55192359863a2 Author: Weiqun Zhang Date: Wed Jul 21 12:12:52 2021 -0700 Add ROCMClang as an LLVM compiler to cmake (#2184) * Add ROCMClang as an LLVM compiler to cmake Since CMake >= 3.21, ROCm is identified as ROCMClang. * HIP: Fix legacy CI / hipcc Until we get ROCM 4.4, we need to work around this new detection: https://gitlab.kitware.com/cmake/cmake/-/blob/v3.21.0/Modules/CMakeDetermineCompilerId.cmake#L153-159 Co-authored-by: Axel Huebl .github/workflows/hip.yml Tools/CMake/AMReXGenerateConfigHeader.cmake commit 948bd72d3e2ee4ee7ee6e1861a08f559ac4d1ffc Author: asalmgren Date: Tue Jul 20 14:53:41 2021 -0700 Fix typo (#2181) * Fix typo : The_Async_Async --> The_Async_Arena Docs/sphinx_documentation/source/GPU.rst commit fe9ae3e2a978de929686ad070c713baaf634ec1c Author: jmsexton03 Date: Tue Jul 20 12:07:50 2021 -0700 Add bounds-checking in InitFromAscii for CI null-dereference error (#2180) Src/Particle/AMReX_ParticleInit.H Tests/Particles/InitFromAscii/main.cpp commit a449f384df5ddece0d4e063fe9001e280ed64024 Author: jmsexton03 Date: Tue Jul 20 09:57:32 2021 -0700 Scale uniform random interval to account for gridbox length (#2179) Src/Particle/AMReX_ParticleInit.H commit 48d84eb342b04aae7e18217980a4514e5a9de3ea Author: Weiqun Zhang Date: Tue Jul 20 08:44:32 2021 -0700 Fix and update multifab ParallelFor (#2178) - Fix a bug in caching multifab ParallelFor meta-data. - Add new versions of multifab ParallelFor that are templated on max threads per block. Src/Base/AMReX_FabArrayBase.H Src/Base/AMReX_FabArrayBase.cpp Src/Base/AMReX_MFParallelFor.H Src/Base/AMReX_MFParallelForG.H commit 779835b7d122f1c94be7c85486e951e8766730ec Author: Weiqun Zhang Date: Mon Jul 19 10:41:51 2021 -0700 Optimization of ParallelFor for single box FabArray (#2176) When there is only one Fab, we can simplify the MF ParallelFor funciton. Src/Base/AMReX_MFParallelForG.H commit ee8facf7e09d0b40af1fe680665660c24a19d32a Author: asalmgren Date: Mon Jul 19 09:11:47 2021 -0700 Add ns to gpu cns (#2177) Add the ability to compute and use diffusive fluxes to Tests/GPU/CNS. Default is now to include diffusive terms so the existing inputs files now have cns.do_visc=false. Tests/GPU/CNS/CMakeLists.txt Tests/GPU/CNS/Exec/RT/inputs Tests/GPU/CNS/Exec/RT/inputs-rt Tests/GPU/CNS/Exec/Sod/inputs Tests/GPU/CNS/Exec/Sod/inputs-rt Tests/GPU/CNS/Source/CNS.H Tests/GPU/CNS/Source/CNS.cpp Tests/GPU/CNS/Source/CNS_advance.cpp Tests/GPU/CNS/Source/CNS_index_macros.H Tests/GPU/CNS/Source/CNS_parm.H Tests/GPU/CNS/Source/diffusion/CNS_diffusion_K.H Tests/GPU/CNS/Source/diffusion/Make.package Tests/GPU/CNS/Source/hydro/CNS_hydro_K.H commit 388ce7e96d74eef0f082f19d95e16647a2d34ccb Author: Axel Huebl Date: Mon Jul 19 08:44:23 2021 -0700 CI: IntelLLVM known to CMake (#2136) ## Summary `IntelLLVM` is now a recognized CMake compiler, so we don't need to add a Clang-ish identification anymore. Also fix a CMake -Wdev warning (DPC++/SYCL) - beta08 work-around: `-mlong-double-64` (https://github.com/intel/llvm/issues/2187) -> still needed! - beta09 work-around: `-fno-sycl-early-optimizations` (link to upstream issue?) -> keep for now, RT tests pending to verify fix ## Additional background setting the CXX compiler ID was a work-around for the 2021.1 DPC++ release / CMake 3.19.0-3.19.1 https://gitlab.kitware.com/cmake/cmake/-/issues/21551#note_869580 .github/workflows/intel.yml Tools/CMake/AMReXSYCL.cmake commit 98c9e7197c5c94135fd9bb070aeea12b8fb58629 Author: Andrew Myers Date: Fri Jul 16 12:21:41 2021 -0700 Add default move constructors / assignment operators for other pc types. (#2174) Src/AmrCore/AMReX_AmrParticles.H Src/Particle/AMReX_NeighborParticles.H Src/Particle/AMReX_TracerParticles.H commit 6e27136f76280e565d30ff654031d1b1ac683171 Author: Weiqun Zhang Date: Thu Jul 15 16:58:58 2021 -0700 Fix warning with Intel icpc (#2173) * Fix warning with Intel icpc The classical EDG based Intel compiler gives warning about `fallthrough` when `-std=c++14` is used, even though it passes the feature test for `fallthrough`. * CI: Add Intel ICC/ICPC/IFORT Co-authored-by: Axel Huebl .github/workflows/intel.yml Src/Base/AMReX_Extension.H commit 6a49159efff4f7d742b604ffbe61f39ab4fa1a0a Author: Weiqun Zhang Date: Thu Jul 15 14:15:47 2021 -0700 Gradient in WarpX EB nodal solver (#2171) Implement function that computes the gradient of potential in the EB nodal solver for WarpX. Src/LinearSolvers/MLMG/AMReX_MLEBNodeFDLap_K.H Src/LinearSolvers/MLMG/AMReX_MLEBNodeFDLaplacian.H Src/LinearSolvers/MLMG/AMReX_MLEBNodeFDLaplacian.cpp commit 1be4862218d034349ae6f24d948ad39abeac1a2c Author: Candace Gilet Date: Wed Jul 14 22:12:18 2021 -0400 Update MacProjector::project to check if m_umac is nullptr (#2166) * Remove unnecessary temporary and copy. * Update MacProjector::project to check if m_umac is nullptr. If nullptr, don't compute div(umac) or attempt to average down or update umac. * Remove tabs Src/LinearSolvers/Projections/AMReX_MacProjector.cpp commit 8a2ddee47fecb368349db09a4505fe753ddeb65d Author: Weiqun Zhang Date: Wed Jul 14 16:56:47 2021 -0700 EB nodal solver (#2145) New EB nodal solver with Dirichlet EB. This uses a finite-difference stencil. It will be used by WarpX's static solver. Currently only Dirichlet and periodic domain BC are supported. Support for Neumann BC at the domain boundary will be added later. Currently, it supports single level only. Src/EB/AMReX_EBDataCollection.H Src/EB/AMReX_EBDataCollection.cpp Src/EB/AMReX_EBFabFactory.H Src/LinearSolvers/CMakeLists.txt Src/LinearSolvers/MLMG/AMReX_MLEBNodeFDLap_2D_K.H Src/LinearSolvers/MLMG/AMReX_MLEBNodeFDLap_3D_K.H Src/LinearSolvers/MLMG/AMReX_MLEBNodeFDLap_K.H Src/LinearSolvers/MLMG/AMReX_MLEBNodeFDLaplacian.H Src/LinearSolvers/MLMG/AMReX_MLEBNodeFDLaplacian.cpp Src/LinearSolvers/MLMG/AMReX_MLNodeLinOp.H Src/LinearSolvers/MLMG/AMReX_MLNodeLinOp.cpp Src/LinearSolvers/MLMG/Make.package commit cc1ceb1188537b0d126be323537f8aaa4294983c Author: kngott <30483578+kngott@users.noreply.github.com> Date: Wed Jul 14 15:57:39 2021 -0700 Fix missing MPI flags in Perlmutter. (#2170) Tools/GNUMake/sites/Make.nersc commit 24a586d89da996195ef81aff2a47c7d2bf0825c7 Author: Andrew Myers Date: Wed Jul 14 14:26:25 2021 -0700 Add ssh instructions to CONTRIBUTING.md (#2168) CONTRIBUTING.md commit 6ab65e7fc0ace3cd116f2b7e2c51a3506e7d4ff1 Author: Weiqun Zhang Date: Tue Jul 13 20:46:31 2021 -0700 More reliable workaround for DPCPP reduction issue (#2167) It seems that the DPC++ reduction issue is due to writing to pinned memory on device. The workaround is to write to device memory and then memcpy. Src/Base/AMReX_Reduce.H commit d9eeb62638eb3352fb4a7eaf6ff93b9f0c668e04 Author: Weiqun Zhang Date: Mon Jul 12 17:32:14 2021 -0700 Workaround for a DPC++ reduction issue (#2165) Src/Base/AMReX_Reduce.H commit cd1f5430be29aed0a85348669394d7fbb8355dba Author: Weiqun Zhang Date: Mon Jul 12 08:45:53 2021 -0700 add ParmParse::remove (#2160) Src/Base/AMReX_ParmParse.H Src/Base/AMReX_ParmParse.cpp commit 0ba4a8f628d240642431b537fd62e809ddf6aa41 Author: Weiqun Zhang Date: Mon Jul 12 08:45:18 2021 -0700 Minor optimization of reducing 4 bytes (#2161) Box length is used in many kernels. Because the length in z-direction is not needed, we only need 2 integers, not Dim3 that has 3 integers. Src/AmrCore/AMReX_TagBox.cpp Src/Base/AMReX_BaseFabUtility.H Src/Base/AMReX_GpuLaunchFunctsG.H Src/Base/AMReX_Reduce.H commit 135010c8baab688b9a190f008e5098a50a91edb2 Author: Weiqun Zhang Date: Mon Jul 12 08:44:52 2021 -0700 Remove MFGhostIter (#2162) It was added several years in an experiment of overlapping communication and computation. It has never been used. Src/Base/AMReX_FabArrayBase.H Src/Base/AMReX_FabArrayBase.cpp Src/Base/AMReX_MFIter.H Src/Base/AMReX_MFIter.cpp commit 30401db6fc699259b388c09ef5a5cbcc40f04639 Author: Weiqun Zhang Date: Mon Jul 12 08:44:24 2021 -0700 Fix typos with the help of codespell (#2163) .github/workflows/docker/sensei/Dockerfile CHANGES Docs/Notes/Readme.typecheck Docs/sphinx_documentation/source/AMReX_Profiling_Tools.rst Docs/sphinx_documentation/source/Basics.rst Docs/sphinx_documentation/source/BuildingAMReX.rst Docs/sphinx_documentation/source/GPU.rst Docs/sphinx_documentation/source/GPU_Chapter.rst Docs/sphinx_documentation/source/GridCreation.rst Docs/sphinx_documentation/source/SWFFT.rst Docs/sphinx_documentation/source/Testing.rst Docs/sphinx_documentation/source/Visualization.rst Src/Amr/AMReX_Amr.H Src/Amr/AMReX_AmrLevel.H Src/Amr/AMReX_AmrLevel.cpp Src/Amr/AMReX_StateDescriptor.H Src/AmrCore/AMReX_AmrCore.H Src/AmrCore/AMReX_AmrMesh.H Src/AmrCore/AMReX_AmrMesh.cpp Src/AmrCore/AMReX_Cluster.H Src/AmrCore/AMReX_FluxRegister.cpp Src/AmrCore/AMReX_Interpolater.cpp Src/AmrCore/AMReX_TagBox.H Src/AmrCore/CMakeLists.txt Src/Base/AMReX_Arena.H Src/Base/AMReX_AsyncOut.H Src/Base/AMReX_BLProfiler.cpp Src/Base/AMReX_BaseFab.H Src/Base/AMReX_BaseUmap.H Src/Base/AMReX_BaseUmap_nd.f90 Src/Base/AMReX_Box.H Src/Base/AMReX_BoxIterator.H Src/Base/AMReX_CoordSys.cpp Src/Base/AMReX_DArena.cpp Src/Base/AMReX_FArrayBox.H Src/Base/AMReX_FArrayBox.cpp Src/Base/AMReX_FILCC_3D.F90 Src/Base/AMReX_FPC.cpp Src/Base/AMReX_FabArray.H Src/Base/AMReX_FabArrayBase.H Src/Base/AMReX_FabArrayBase.cpp Src/Base/AMReX_FabConv.cpp Src/Base/AMReX_GpuDevice.cpp Src/Base/AMReX_IntVect.H Src/Base/AMReX_MFIter.H Src/Base/AMReX_MFParallelFor.H Src/Base/AMReX_MultiFab.H Src/Base/AMReX_NonLocalBC.H Src/Base/AMReX_NonLocalBCImpl.H Src/Base/AMReX_ParmParse.H Src/Base/AMReX_ParmParse.cpp Src/Base/AMReX_Partition.H Src/Base/AMReX_Utility.cpp Src/Base/AMReX_Vector.H Src/Base/AMReX_iMultiFab.H Src/Base/AMReX_parstream.cpp Src/Base/Parser/AMReX_IParser_Y.H Src/Base/Parser/AMReX_IParser_Y.cpp Src/Base/Parser/AMReX_Parser_Y.H Src/Base/Parser/AMReX_Parser_Y.cpp Src/Boundary/AMReX_BndryData.H Src/EB/AMReX_EB2_3D_C.cpp Src/EB/AMReX_EBToPVD.cpp Src/EB/AMReX_EB_slopes_K.H Src/EB/AMReX_EB_utils.cpp Src/Extern/Conduit/AMReX_Conduit_Blueprint.cpp Src/Extern/Conduit/AMReX_Conduit_Blueprint_ParticlesI.H Src/Extern/HYPRE/AMReX_HypreIJIface.cpp Src/Extern/ProfParser/AMReX_BLWritePlotFile.cpp Src/Extern/ProfParser/AMReX_CommProfStats.cpp Src/Extern/SENSEI/AMReX_AmrDataAdaptor.cpp Src/Extern/SENSEI/AMReX_AmrMeshDataAdaptor.cpp Src/Extern/SENSEI/AMReX_AmrParticleInSituBridge.H Src/Extern/SENSEI/AMReX_InSituBridge.H Src/Extern/SENSEI/AMReX_InSituUtils.H Src/Extern/SENSEI/AMReX_ParticleInSituBridge.H Src/Extern/SWFFT/CheckDecomposition.c Src/Extern/SWFFT/Dfft.H Src/Extern/SWFFT/README Src/Extern/SWFFT/distribution.c Src/Extern/amrdata/AMReX_DataServices.cpp Src/F_Interfaces/Base/AMReX_mpi_reduce_int.F90 Src/F_Interfaces/Base/AMReX_mpi_reduce_real.F90 Src/LinearSolvers/MLMG/AMReX_MLALaplacian.H Src/LinearSolvers/MLMG/AMReX_MLEBABecLap.H Src/LinearSolvers/MLMG/AMReX_MLMG.H Src/LinearSolvers/MLMG/AMReX_MLMG.cpp Src/LinearSolvers/Projections/AMReX_NodalProjector.H Src/Particle/AMReX_NeighborParticles.H Src/Particle/AMReX_ParticleArray.H Src/Particle/AMReX_Particles.H Src/SDC/AMReX_SDCstruct.cpp Tests/Amr/Advection_AmrLevel/Source/AmrLevelAdv.cpp Tests/Amr/Advection_AmrLevel/Source/Src_2d/Adv_2d.f90 Tests/Amr/Advection_AmrLevel/Source/Src_3d/Adv_3d.f90 Tests/DivFreePatch/main.cpp Tests/EB/CNS/Exec/Sod/inputs Tests/EB/CNS/Source/diffusion/cns_eb_diff_wall.F90 Tests/FortranInterface/Advection_F/Source/Src_2d/advect_2d_mod.F90 Tests/FortranInterface/Advection_F/Source/Src_3d/advect_3d_mod.F90 Tests/FortranInterface/Advection_octree_F/Source/Src_2d/advect_2d_mod.F90 Tests/GPU/CNS/Source/CNS_bcfill.cpp Tests/GPU/Fuse/GNUmakefile Tests/LinearSolvers/ABecLaplacian_C/scalingtest/results.org Tests/LinearSolvers/ABecLaplacian_F/mytest.F90 Tests/LinearSolvers/LeastSquares/README.md Tests/LinearSolvers/MAC_Projection_EB/main.cpp Tests/Particles/NeighborParticles/MDParticleContainer.cpp Tests/Slopes/README.md Tools/CMake/AMReXCUDAOptions.cmake Tools/CMake/AMReXOptions.cmake Tools/CMake/AMReXParallelBackends.cmake Tools/CMake/AMReXSetDefines.cmake Tools/CMake/AMReXTargetHelpers.cmake Tools/CMake/AMReXTypecheck.cmake Tools/C_scripts/makebuildinfo_C.py Tools/C_scripts/mkdep Tools/C_scripts/strip72 Tools/C_util/AppendToPlotFile.cpp Tools/C_util/AugmentPlotfile/AugmentPlotfile.cpp Tools/C_util/Convergence/ComparePlotfiles.cpp Tools/C_util/README Tools/F_scripts/extract.parallel Tools/F_scripts/f90doc/typing.pl Tools/F_scripts/write_probin.py Tools/GNUMake/Make.rules Tools/GNUMake/comps/cray.mak Tools/GNUMake/comps/nvcc.mak Tools/GNUMake/comps/nvhpc.mak Tools/GNUMake/comps/pgi.mak Tools/GNUMake/sites/Make.nersc Tools/GNUMake/sites/Make.nrel Tools/Migration/step-7-bindc/bindc.sh Tools/Postprocessing/python/dumpparthistory.py Tools/Release/ppCleanup.py Tools/Release/ppCleanupDir.py Tools/Release/release.py Tools/libamrex/mkpkgconfig.py commit bef02a4c6394d24578ceed82ab9d73ede5a637e2 Author: Weiqun Zhang Date: Sat Jul 10 15:09:36 2021 -0700 Parser Precision (#2154) Use double internally. If the arguments are all floats, the final result will be converted to float. This is necessary to avoid overflow in some WarpX single precision tests. Src/Base/Parser/AMReX_Parser.H Src/Base/Parser/AMReX_Parser.cpp Src/Base/Parser/AMReX_Parser_Exe.H Src/Base/Parser/AMReX_Parser_Exe.cpp Src/Base/Parser/AMReX_Parser_Y.H Src/Base/Parser/AMReX_Parser_Y.cpp Src/Base/Parser/amrex_parser.tab.h Src/Base/Parser/amrex_parser.y commit f8c2fe41d3187e69f747816f1f6ab0d38335c11e Author: asalmgren Date: Sat Jul 10 14:11:49 2021 -0700 Add to particlemeshtest (#2159) Tests/Particles/ParticleMeshMultiLevel/CMakeLists.txt Tests/Particles/ParticleMeshMultiLevel/Make.package Tests/Particles/ParticleMeshMultiLevel/main.cpp Tests/Particles/ParticleMeshMultiLevel/mypc.H Tests/Particles/ParticleMeshMultiLevel/trilinear_deposition_K.H commit d325cf8eab0d78bb356d5bcf6c90b9bef7ef63ae Author: Weiqun Zhang Date: Sat Jul 10 14:09:06 2021 -0700 IParser: Integer Parser (#2158) Add a parser for integers. Because there is rounding in integer division, it's not safe to use the real number parser for integer. The two parsers have a lot of similarity, but floating point number specific functions (e.g., `sqrt`, `sin`, etc.) are not supported in `IParser`. In addition to `/` whose result truncates towards zero, the integer parser also supports `//` whose result truncates towards negative infinity. Docs/sphinx_documentation/source/Basics.rst Src/Base/CMakeLists.txt Src/Base/Make.package Src/Base/Parser/AMReX_IParser.H Src/Base/Parser/AMReX_IParser.cpp Src/Base/Parser/AMReX_IParser_Exe.H Src/Base/Parser/AMReX_IParser_Exe.cpp Src/Base/Parser/AMReX_IParser_Y.H Src/Base/Parser/AMReX_IParser_Y.cpp Src/Base/Parser/AMReX_Parser.cpp Src/Base/Parser/AMReX_Parser_Y.cpp Src/Base/Parser/GNUmakefile Src/Base/Parser/amrex_iparser.l Src/Base/Parser/amrex_iparser.lex.cpp Src/Base/Parser/amrex_iparser.lex.h Src/Base/Parser/amrex_iparser.tab.cpp Src/Base/Parser/amrex_iparser.tab.h Src/Base/Parser/amrex_iparser.y Src/Base/Parser/amrex_parser.l Src/Base/Parser/amrex_parser.lex.cpp Src/Base/Parser/amrex_parser.y Tests/Parser/main.cpp commit fb41ccbb852a89026cda4b1d26eee6f1fab56f9e Author: asalmgren Date: Fri Jul 9 14:26:29 2021 -0700 Remove single level assertions for NeighborParticleContainers on GPUs (#2157) * Remove single-level assertions for NeighborParticleContainer on GPUs Src/Particle/AMReX_NeighborParticlesGPUImpl.H commit cebf3df6c3e068a0c4f69b04198c0ba9af94d50a Author: Candace Gilet Date: Fri Jul 9 15:08:59 2021 -0400 Docs: fix typo in author name (#2156) Docs/sphinx_documentation/source/GridCreation.rst commit b0ffd259a5bf7eba82fc47a3871ae6b61c8860a5 Author: Erik Date: Thu Jul 8 15:57:11 2021 -0400 correct sp typo (#2153) Docs/sphinx_documentation/source/GettingStarted.rst commit 91ea9a0243b48a5d66c0c09e18df16e2db60e27f Author: Erik Schnetter Date: Thu Jul 8 13:44:41 2021 -0400 Disable `jn` on Windows (#2152) Disable `jn` on Windows – not just when `dpcpp` is used, but also when building with other compilers in MINGW. Src/Base/Parser/AMReX_Parser_Y.H commit 6b2ef0b507a24d2d90a04c81895c33aaf315454a Author: Mu-Hua Chien Date: Thu Jul 8 11:24:46 2021 -0400 add an option for hypre solver for Tests/Linearsolvers/NodalPoisson (#2139) Add input-rt.hypre for NodalPoisson example Tests/LinearSolvers/NodalPoisson/GNUmakefile Tests/LinearSolvers/NodalPoisson/MyTest.H Tests/LinearSolvers/NodalPoisson/MyTest.cpp Tests/LinearSolvers/NodalPoisson/inputs-rt.hypre commit 49b6ad4a85f09f86e7601a07744ed6e63f29a164 Author: Erik Date: Thu Jul 8 11:20:46 2021 -0400 Doxygen Side-Bar Navigation, Doxygen Logo and Mainpage Content (#2150) Enables side-bar navigation for Doxygen, and adds logo and content to main splash page. Docs/Doxygen/customdoxygen.css Docs/Doxygen/doxygen.conf Docs/Doxygen/footer.html Docs/Doxygen/header.html Docs/Doxygen/main.dox Docs/Doxygen/resize_AMReX_logo.png commit e7591aabfe4578ddf9339688632700a70ac17af0 Author: Weiqun Zhang Date: Wed Jul 7 20:38:29 2021 -0700 Allow empty string Parser (#2151) Src/Base/Parser/AMReX_Parser.H Src/Base/Parser/AMReX_Parser.cpp commit 1d6b01af423bf0f39d9efba8ac59ebff75179de0 Author: Erik Date: Wed Jul 7 12:13:38 2021 -0400 Doxygen comment formatting (#2148) Src/Particle/AMReX_Particles.H commit fffd3d976c7e8a3fa1e9a59e35e7cbfbd37a2493 Author: Andrew Myers Date: Tue Jul 6 21:53:51 2021 -0700 Fix the size of the BC Vector in the multi-level version of ParticleToMesh. (#2149) Src/AmrCore/AMReX_AmrParticles.H commit 7c185502a57fb263f0819635c14f7ee84e96f0b7 Author: Andrew Myers Date: Tue Jul 6 10:56:01 2021 -0700 Make ParticleToMesh and MeshToParticle support iMultifab. (#2137) Src/Base/AMReX_FabArray.H Src/Base/AMReX_FabArrayUtility.H Src/Base/AMReX_MultiFab.H Src/Base/AMReX_MultiFab.cpp Src/Base/AMReX_iMultiFab.H Src/Base/AMReX_iMultiFab.cpp Src/Particle/AMReX_ParGDB.H Src/Particle/AMReX_ParticleContainerBase.H Src/Particle/AMReX_ParticleMesh.H Tests/Particles/ParticleMesh/main.cpp commit 901bac18bdc3635b992121ba1023b0b9d14f8729 Author: Nicholas Brunhart-Lupo Date: Fri Jul 2 15:42:16 2021 -0600 Fix crashes with Conduit Blueprint Code (#2144) ## Summary This PR contributes two fixes to the Conduit Blueprint code. ### Nestsets fix (Issue AMReX-Codes/amrex#2135) The first fix here is within `Nestsets`, which resolves a one-past the end access on `ref_ratio`. The problematic lines are 116 to 120, of the form `window["ratio/i"] = ref_ratio[level+1][0];` For some inputs, this is one-past the end of the `ref_ratio` array. It was advised in https://github.com/AMReX-Codes/amrex/issues/2135#issuecomment-872692909 to resolve this by removing the `+1`. ### MultiLevelToBlueprint fix (Issue AMReX-Codes/amrex#2133) The second fix resolves a bug inside `MultiLevelToBlueprint`, where the `box_offsets` array is not built correctly. The problematic code is here: https://github.com/AMReX-Codes/amrex/blob/14dde9aaf741f84cf85adb67a7cdc5bef4c0764a/Src/Extern/Conduit/AMReX_Conduit_Blueprint.cpp#L363-L376 During the construction of the `box_offsets` array the first round of the loop at line 364, with `i == 0` will size the `box_offsets` array to 1. Subsequent rounds through the loop will attempt to write to non-existent slots of the vector here: https://github.com/AMReX-Codes/amrex/blob/14dde9aaf741f84cf85adb67a7cdc5bef4c0764a/Src/Extern/Conduit/AMReX_Conduit_Blueprint.cpp#L374 This fix sizes `box_offsets` before the loop starts, and changes all accesses to the vector by index, not by appending. This appears to be the right fix as advised here: https://github.com/AMReX-Codes/amrex/issues/2133#issuecomment-871501258 Src/Extern/Conduit/AMReX_Conduit_Blueprint.cpp commit 85021c840b13c0f89c0db7039cda66585e9b239a Author: kngott <30483578+kngott@users.noreply.github.com> Date: Fri Jul 2 08:30:43 2021 -0700 Perlmutter Build: Adjust to allow CPU build + cleanup. (#2143) Tested on PrgEnv-nvidia, PrgEnv-gnu for USE_CUDA={TRUE,FALSE} and BL_NO_FORT={TRUE,FALSE}. Tools/GNUMake/sites/Make.nersc commit ab69825ddee58a2ca834a2437376a05c0b8a76af Author: Weiqun Zhang Date: Thu Jul 1 17:06:29 2021 -0700 Fix mismatched tag warning (#2141) AmrInfo is defined as a struct, and it was declared as class in forward declaration. This is valid C++. But the Intel oneAPI compiler gives a warning that this may result in linker errors under the Microsoft C++ ABI. Src/AmrCore/AMReX_AmrCoreFwd.H commit 83087f64af364dd7e80a191fc905a890cf1f635e Author: Weiqun Zhang Date: Thu Jul 1 17:05:57 2021 -0700 Default values for members of ParserExecutor (#2142) Set the default values of pointers inside ParserExecutor to nullptr and add a conversion operator so that it can be tested for null. Src/Base/Parser/AMReX_Parser.H commit d7360e84bb68505316645c346abd1bbe3662d7ce Author: Candace Gilet Date: Thu Jul 1 13:37:23 2021 -0400 Docs: Add labels to chapters that were missing one. (#2134) Docs/sphinx_documentation/source/Basics.rst Docs/sphinx_documentation/source/GettingStarted_Chapter.rst Docs/sphinx_documentation/source/ManagingGridHierarchy_Chapter.rst commit bee2568d831aa1c84e6a9fe24c3e69f3ec5b49a0 Author: Axel Huebl Date: Thu Jul 1 10:36:08 2021 -0700 CMake: HIP Modernizing & RDC (#2031) ## Summary Add `-fgpu-rdc` flags to HIP if requested via `AMReX_GPU_RDC`. Modernize HIP logic with recommended targets that avoid the flaky `hipcc` compiler scripts: https://rocmdocs.amd.com/en/latest/Installation_Guide/Using-CMake-with-AMD-ROCm.html#using-hip-in-cmake Add support for AMDs `clang++`/`clang` compiler for HIP instead of using the legacy `hipcc` perl wrapper as C++ compiler. This also increases support towards Cray compiler wrappers, which also refer to `clang++`/`clang` underneath (Spock/OLCF). Close #1688 ## Additional background Relocatable-device-code (RDC) flags are needed for `extern` device variable support (for codes that use global variables on device). Also needed when linking with Ascent. Follow-up to #2029 With HIP GPU RDC, static libs emitting & linking does get more fancy: https://github.com/ROCmSoftwarePlatform/rccl/pull/260 .github/workflows/hip.yml Docs/sphinx_documentation/source/GPU.rst Tools/CMake/AMReXParallelBackends.cmake commit 5c83ba4d6f5f9c4db9812bbbf0b032f94aac7205 Author: Weiqun Zhang Date: Thu Jul 1 08:10:58 2021 -0700 Update CHANGES for 21.07 (#2138) CHANGES commit 14dde9aaf741f84cf85adb67a7cdc5bef4c0764a Author: Weiqun Zhang Date: Mon Jun 28 20:29:50 2021 -0700 Parser::compileHost (#2132) Add a function that compiles the parser expression into host code only. Src/Base/Parser/AMReX_Parser.H commit 97d907762f776ff4400868029fcc53709703291c Author: Andrew Myers Date: Mon Jun 28 14:03:43 2021 -0700 Fix MR bug in NL construction - the number of cells needs to be increased by a factor of ref_fac. (#2131) Src/Particle/AMReX_NeighborParticlesI.H commit fd1fd11302601a085b2e7ff01f07d36ffa0add66 Author: Weiqun Zhang Date: Mon Jun 28 10:11:55 2021 -0700 EB Implicit Function using Parser (#2129) Add ParserIF to EB. One can use a parser function to generate geometry. For example, the following ParmParse parameters can be used in 3D to place two cylinders with a radius of 0.1 at (-0.5,0,) and (0.5,0,). ``` eb2.geom_type = parser eb2.parser_function = "max(0.01-(x+0.5)^2-y^2, 0.01-(x-0.5)^2-y^2)" ``` Put the internal data of Parser in a shared_ptr for memory safety. Docs/sphinx_documentation/source/Basics.rst Src/Base/Parser/AMReX_Parser.H Src/Base/Parser/AMReX_Parser.cpp Src/EB/AMReX_EB2.cpp Src/EB/AMReX_EB2_GeometryShop.H Src/EB/AMReX_EB2_IF.H Src/EB/AMReX_EB2_IF_Parser.H Src/EB/CMakeLists.txt Src/EB/Make.package commit 55cb40092910183729ca81e6eef3d0ce386af317 Author: Weiqun Zhang Date: Sun Jun 27 10:52:09 2021 -0700 AMReX Version (#2128) In git fails to obtain AMReX version in the make system, grep for the version number in the CHANGES file. CHANGES Tools/GNUMake/Make.defs commit 6ca4306167ed432399a41aaa7a4a80a3f15f65e9 Author: Candace Gilet Date: Fri Jun 25 21:55:09 2021 -0400 Fix typo in 3d NodeLaplacian. (#2126) This fixes an error for cases with for dz != dy Src/LinearSolvers/MLMG/AMReX_MLNodeLap_3D_K.H commit 81535291686e88a4d2cc3bf36a77bca6a6bf48c3 Author: Weiqun Zhang Date: Fri Jun 25 14:48:53 2021 -0700 Parser (#2121) Add amrex::Parser that can be used at runtime to evaluate math expressions given by strings. It works on both host and device. This is moved from WarpX with additional performance optimization of eliminating recursion. Support for local automatic variables are added. .github/workflows/linux.yml .github/workflows/style/check_tabs.sh .github/workflows/style/check_trailing_whitespaces.sh Docs/sphinx_documentation/source/Basics.rst Src/Base/AMReX_BaseFwd.H Src/Base/CMakeLists.txt Src/Base/Make.package Src/Base/Parser/AMReX_Parser.H Src/Base/Parser/AMReX_Parser.cpp Src/Base/Parser/AMReX_Parser_Exe.H Src/Base/Parser/AMReX_Parser_Exe.cpp Src/Base/Parser/AMReX_Parser_Y.H Src/Base/Parser/AMReX_Parser_Y.cpp Src/Base/Parser/GNUmakefile Src/Base/Parser/README Src/Base/Parser/amrex_parser.l Src/Base/Parser/amrex_parser.lex.cpp Src/Base/Parser/amrex_parser.lex.h Src/Base/Parser/amrex_parser.tab.cpp Src/Base/Parser/amrex_parser.tab.h Src/Base/Parser/amrex_parser.y Tests/CMakeLists.txt Tests/Parser/CMakeLists.txt Tests/Parser/GNUmakefile Tests/Parser/Make.package Tests/Parser/main.cpp commit 67b0794d02c2f13e2c0156b752c486da6863deea Author: Andrew Myers Date: Thu Jun 24 17:52:45 2021 -0700 Remove incorrect call to coarsen. (#2125) Src/Particle/AMReX_NeighborParticlesI.H commit 8575439bdbc52bda3eb38f12660a5d3c1e0ae584 Author: Weiqun Zhang Date: Wed Jun 23 22:59:37 2021 -0700 Avoid std inclusive and exclusive scan for _GLIBCXX_RELEASE < 10 (#2123) Src/Base/AMReX_Scan.H commit 5db3b4fff09400a7c040b6792e06b2d9051fd01a Author: Andrew Myers Date: Tue Jun 22 12:37:48 2021 -0700 Fix Arena docs. (#2118) * Fix Arena docs. * remove device arena from printstats output in docs Docs/sphinx_documentation/source/GPU.rst commit 91a4ff2ce1f0a950ee7092697ea48f9e814391d3 Author: c-wetterer-nelson <78513275+c-wetterer-nelson@users.noreply.github.com> Date: Tue Jun 22 14:28:39 2021 -0400 Add SENSEI Adaptor for Particles Containers (#2016) Docs/sphinx_documentation/source/BuildingAMReX.rst Src/Amr/AMReX_Amr.H Src/Amr/AMReX_Amr.cpp Src/Extern/SENSEI/AMReX_AmrDataAdaptor.cpp Src/Extern/SENSEI/AMReX_AmrInSituBridge.H Src/Extern/SENSEI/AMReX_AmrInSituBridge.cpp Src/Extern/SENSEI/AMReX_AmrMeshDataAdaptor.cpp Src/Extern/SENSEI/AMReX_AmrMeshInSituBridge.H Src/Extern/SENSEI/AMReX_AmrMeshInSituBridge.cpp Src/Extern/SENSEI/AMReX_AmrParticleDataAdaptor.H Src/Extern/SENSEI/AMReX_AmrParticleDataAdaptorI.H Src/Extern/SENSEI/AMReX_AmrParticleInSituBridge.H Src/Extern/SENSEI/AMReX_InSituBridge.H Src/Extern/SENSEI/AMReX_InSituBridge.cpp Src/Extern/SENSEI/AMReX_InSituUtils.H Src/Extern/SENSEI/AMReX_InSituUtils.cpp Src/Extern/SENSEI/AMReX_ParticleDataAdaptor.H Src/Extern/SENSEI/AMReX_ParticleDataAdaptorI.H Src/Extern/SENSEI/AMReX_ParticleInSituBridge.H Src/Extern/SENSEI/CMakeLists.txt Src/Extern/SENSEI/Make.package Tools/CMake/AMReXConfig.cmake.in Tools/CMake/AMReXOptions.cmake Tools/CMake/AMReXSetDefines.cmake Tools/CMake/AMReXTargetHelpers.cmake Tools/CMake/AMReX_Config.H.in Tools/GNUMake/Make.defs Tools/GNUMake/tools/Make.sensei Tools/libamrex/configure.py commit c797f6b6979aea63cc6bda574765c29ba89e7b9c Author: kwryankrattiger <80296582+kwryankrattiger@users.noreply.github.com> Date: Thu Jun 17 15:45:09 2021 -0500 CI: Add job to test SENSEI build (#2115) .github/workflows/docker/sensei/Dockerfile .github/workflows/docker/sensei/install_deps.sh .github/workflows/docker/sensei/install_sensei.sh .github/workflows/docker/sensei/install_vtk_minimal.sh .github/workflows/docker/sensei/tools.sh .github/workflows/linux.yml commit 4eb054cf1d37f3e2fa79f01a17116072ccfbbcb8 Author: Weiqun Zhang Date: Thu Jun 17 10:22:09 2021 -0700 Fix warning in algoim (#2117) Split the constructor of ImplicitIntegral into two templates to avoid an unreachable loop warning. Src/EB/AMReX_algoim_K.H Src/LinearSolvers/MLMG/AMReX_MLPoisson.cpp Tools/GNUMake/Make.defs commit 04464a4f6d1201dfd01fd248c3ad64c5a6809a74 Author: Andy Nonaka Date: Thu Jun 17 09:46:31 2021 -0700 Enhance DiffSameGrid2.cpp (#2114) This utility, when using L0 norm, now outputs an additional column with the absolute error divided by the maximum value on the coarse grid. This is useful for comparing/verifying data with local regions of very small (close to zero in a machine precision sense) values that may fluctuate based on roundoff errors. Tools/C_util/Convergence/DiffSameGrid2.cpp commit 501626901f500b45fce5102c01d15cd8f74c7187 Author: Andrew Myers Date: Wed Jun 16 15:49:57 2021 -0700 Don't rely on managed memory in ParticleCopyPlan (#2116) Src/Particle/AMReX_ParticleCommunication.H Src/Particle/AMReX_ParticleCommunication.cpp commit baf0d4cd96fa81ace1c964a2e5601685e97d41f6 Author: Axel Huebl Date: Tue Jun 15 16:10:11 2021 -0700 HIP: Increase to C++17 (#2113) Bump HIP requirements to C++17 or newer (like DPC++). At the moment, we only test against AMD's clang++/hipcc, which supports C++17 anyway. .github/workflows/hip.yml Docs/sphinx_documentation/source/BuildingAMReX.rst Tools/CMake/AMReX_Config.cmake Tools/GNUMake/comps/hip.mak commit f7ae247580204b1f607e22b09c046120d2cc6e38 Author: Weiqun Zhang Date: Tue Jun 15 14:26:44 2021 -0700 Port remaining Fortran and CPU kernels in FluxRegister to GPU (#2112) Src/AmrCore/AMReX_FLUXREG_F.H Src/AmrCore/AMReX_FLUXREG_nd.F90 Src/AmrCore/AMReX_FluxRegister.H Src/AmrCore/AMReX_FluxRegister.cpp Src/AmrCore/CMakeLists.txt Src/AmrCore/Make.package Src/Base/AMReX_FBI.H Src/Particle/AMReX_NeighborParticlesGPUImpl.H commit 659ab43420a2c4a918cda2702910771fbbc395d3 Author: Weiqun Zhang Date: Mon Jun 14 18:01:42 2021 -0700 TinyProfiler: Add spaces to print (#2110) Add more spaces so that the output looks nicer when there are multiple layers of profilers. Src/Base/AMReX_TinyProfiler.H Src/Base/AMReX_TinyProfiler.cpp commit 8182a3a630e580ed9158bd1da33276380f170d41 Author: Andrew Myers Date: Mon Jun 14 14:33:08 2021 -0700 Reimplement partitionParticlesByDest (#2109) Src/Particle/AMReX_ParticleUtil.H commit d7ff203305e8c8605ca6f92e9212acf717da56de Author: Weiqun Zhang Date: Mon Jun 14 12:10:05 2021 -0700 GpuArray template parameter (#2108) Use unsigned int as the type for size instead of std::size_t so that the forward declaration does not need to include another header. Src/Base/AMReX_Array.H Src/Base/AMReX_BaseFwd.H commit c221a567dbd8cadd474146d9cc36ba5152178fb1 Author: Weiqun Zhang Date: Mon Jun 14 10:38:00 2021 -0700 Fix a bug in in-place ExclusiveSum (#2105) For ExclusiveSum versions using memcpy to get the total sum, we need to copy the last element of the input data to the host before the scan so that it's safe for in-place scan. Also use std::exclusive_scan and inclusive_scan on CPU if available. Src/Base/AMReX_Scan.H commit 8357d530311235e3f76899051ca8d44a51b971c0 Author: Luca Fedeli Date: Mon Jun 14 17:18:20 2021 +0200 add missing class (#2106) Src/Base/AMReX_BaseFwd.H commit fc9bc1e3efdaead0253d0a55dcf1abb7abc06bd7 Author: Weiqun Zhang Date: Sun Jun 13 12:42:38 2021 -0700 Add more warnings to CI and fix warnings (#2104) Also fixed a bug in device version of `Random_int`. .github/workflows/cuda.yml .github/workflows/hip.yml .github/workflows/intel.yml .github/workflows/linux.yml .github/workflows/macos.yml Src/Base/AMReX_BLBackTrace.cpp Src/Base/AMReX_Random.H Src/Base/AMReX_Reduce.H Src/Base/AMReX_Scan.H Src/Particle/AMReX_NeighborParticlesCPUImpl.H Src/Particle/AMReX_NeighborParticlesI.H Src/Particle/AMReX_ParticleTransformation.H Src/Particle/AMReX_WriteBinaryParticleData.H Tests/Amr/Advection_AmrLevel/Source/AmrLevelAdv.cpp Tests/MultiBlock/Advection/main.cpp Tests/MultiBlock/IndexType/main.cpp Tests/Particles/AssignMultiLevelDensity/main.cpp Tests/Particles/GhostsAndVirtuals/main.cpp Tests/Particles/NeighborParticles/MDParticleContainer.cpp Tests/Particles/ParallelContext/main.cpp Tests/Particles/ParticleArray/main.cpp Tests/Particles/ParticleIterator/main.cpp Tests/Particles/ParticleMesh/main.cpp Tests/Particles/SparseBins/main.cpp Tests/Particles/TypeDescriptor/main.cpp Tools/CMake/AMReXParallelBackends.cmake Tools/GNUMake/comps/dpcpp.mak Tools/GNUMake/comps/gnu.mak Tools/GNUMake/comps/hip.mak Tools/Plotfile/fextract.cpp commit 2ff4519b6e16c1f0906e637559f62d183158734c Author: Weiqun Zhang Date: Sat Jun 12 08:07:04 2021 -0700 Missing defined in preprocessing (#2102) Src/Base/AMReX_FPC.cpp commit 2a676f8a25e6cd75ed2e3936bd0cd2e30141363f Author: Michael Zingale Date: Fri Jun 11 20:42:24 2021 -0400 add AUTO_BUILD_SOURCES to the dependencies for GNU make (#2101) AUTO_BUILD_SOURCES can be filled by an application code with the source files that are created automatically at build time to ensure they are made before compilation / dependency resolution. Tools/GNUMake/Make.rules commit 64e41a8d314fc28f6d6473095ac5bdcd256c445f Author: Weiqun Zhang Date: Fri Jun 11 17:20:08 2021 -0700 Add a number of forward declaration headers (#2100) Src/Amr/AMReX_AmrFwd.H Src/Amr/CMakeLists.txt Src/Amr/Make.package Src/AmrCore/AMReX_AmrCoreFwd.H Src/AmrCore/CMakeLists.txt Src/AmrCore/Make.package Src/Base/AMReX_Array4.H Src/Base/AMReX_BaseFwd.H Src/Base/CMakeLists.txt Src/Base/Make.package Src/Boundary/AMReX_BoundaryFwd.H Src/Boundary/CMakeLists.txt Src/Boundary/Make.package commit 4cf09ce1a3eb2c06460866af04b0ec420300f1d0 Author: Axel Huebl Date: Wed Jun 9 20:01:14 2021 -0700 Particle Device Members: Inline (#2097) Member functions that are not `operator()` are not inlined by default. I think we should inline those getters, since they are too tiny / too deep in hot loops to make sense to indirect. (The compiler likely figured this out on its own already, at least I saw no immediate difference in a massive WarpX kernel with NVCC and looking at ptxas info.) Src/Particle/AMReX_Particle.H commit 7eaf1ee13726cfb689d4f0bffe3b5d056b09eceb Author: Axel Huebl Date: Wed Jun 9 15:19:21 2021 -0700 CI: Split Linux Workflows (#2096) Split workflows into - linux (CPU) - cuda - intel (DPC++/ICC/ICX) - hip workflows. Individual workflows can be individually restarted and this increases order a bit as well. .github/workflows/cuda.yml .github/workflows/hip.yml .github/workflows/intel.yml .github/workflows/linux.yml commit e6c52ec95486d8d26ec679d9aea561fa6266d844 Author: Axel Huebl Date: Wed Jun 9 09:17:03 2021 -0700 CMake 3.20+ CUDA: Fast Math (#2094) Honor CUDA fast math option `AMReX_CUDA_FASTMATH` (default: ON) in modernized CUDA CMake logic. Tools/CMake/AMReXParallelBackends.cmake commit 56de38905af1fe6b1eff8fe390de50bcc65449c2 Author: Axel Huebl Date: Mon Jun 7 19:20:17 2021 -0700 CMake 3.20+ CUDA: Shell-Escape --Werror (#2093) ## Summary When adding a list of flags to CMake targets, duplicated flags get by default deduplicated. This caused with our new CMake 3.20+ CUDA logic #2012 that this gets passed to the linker: `--Werror ext-lambda-captures-this cross-execution-space-call`. The CMake `SHELL:` prefix is exactly made for this instance, keeping flags that belong together also over transformations like this together: https://cmake.org/cmake/help/latest/command/target_compile_options.html#arguments Since we injected directly into the global `CMAKE_CUDA_FLAGS` flags for earlier CMake versions, this deduplication logic does not apply and thus we have not experienced it earlier. ## Additional background Seen first with CI in AMReX Tutorials (e.g. https://github.com/AMReX-Codes/amrex-tutorials/pull/12). Tools/CMake/AMReXParallelBackends.cmake commit 88b50cf2e427c5358bed05d792f557f5da303028 Author: Wileam Y. Phan <50928756+wyphan@users.noreply.github.com> Date: Mon Jun 7 22:04:23 2021 -0400 Add link to "Building with CMake" page in INSTALL file (#2092) Add a link to "Building with CMake" page in the AMReX documentation web page into the INSTALL file. INSTALL commit 53614b4a5b8bd769918523af10341b22c57a5596 Author: Weiqun Zhang Date: Mon Jun 7 12:18:04 2021 -0700 Remove VecReduce and FabReduce (#2091) They were used for the old atomics based reduction approach. Only PeleC uses VecReduce. The use of VecReduce there can be replaced with Reduce::Sum. Src/Base/AMReX_GpuLaunchFunctsC.H Src/Base/AMReX_GpuLaunchFunctsG.H commit 79678dd3b8c1ecf92c0aaf896c4da1ab7e297465 Author: Weiqun Zhang Date: Mon Jun 7 12:17:23 2021 -0700 Add more versions of ParallelFor(MF) (#2089) Src/Base/AMReX_Algorithm.H Src/Base/AMReX_FBI.H Src/Base/AMReX_FabArray.H Src/Base/AMReX_FabArrayBase.cpp Src/Base/AMReX_MFParallelFor.H Src/Base/AMReX_MFParallelForC.H Src/Base/AMReX_MFParallelForG.H Src/Base/CMakeLists.txt Src/Base/Make.package commit 87e01fe44b483adab034072dc000aed67aa1b5a7 Author: Roberto Porcu <53792251+rporcu@users.noreply.github.com> Date: Fri Jun 4 17:53:15 2021 -0400 Fix indexes bug for particles output in GPU version of packIOData (#2087) Src/Particle/AMReX_WriteBinaryParticleData.H commit 9e3fb2f574453dc13e7e1d03a53f0ce41a9ad0e1 Author: Weiqun Zhang Date: Fri Jun 4 11:32:00 2021 -0700 Remove deprecated Reduce functions (#2086) It includes Min, Max, and MinMax with the signature like `T Sum (N n, U const* v, T init_val, BOP bop)`. The other versions are still available. Src/Base/AMReX_Reduce.H commit eb2fb8eb11d4df88d5b6c04a68f2971a6c3b97f0 Author: Weiqun Zhang Date: Thu Jun 3 21:31:05 2021 -0700 Fix some macro issues in CMake Fortran build (#2084) Include AMReX_Config.H in Fortran files that need preprocessing. Do not pass those definitions already defined in AMReX_Config.H to the Fortran compiler in the cmake build. This fixes the warnings about redefined macros. Src/AmrCore/AMReX_INTERP_2D.F90 Src/AmrCore/AMReX_INTERP_3D.F90 Src/Base/AMReX_FILCC_1D.F90 Src/Base/AMReX_FILCC_2D.F90 Src/Base/AMReX_FILCC_3D.F90 Src/Base/AMReX_acc_mod.F90 Src/Base/AMReX_filcc_mod.F90 Src/Base/AMReX_fort_mod.F90 Src/Base/AMReX_omp_mod.F90 Src/Boundary/AMReX_LO_UTIL.F90 Src/EB/AMReX_ebcellflag_mod.F90 Src/F_Interfaces/AmrCore/AMReX_fillpatch_mod.F90 Src/F_Interfaces/Base/AMReX_fab_mod.F90 Src/F_Interfaces/Base/AMReX_parallel_mod.F90 Src/F_Interfaces/Particle/AMReX_particlecontainer_mod.F90 Src/LinearSolvers/MLMG/AMReX_MLLinOp_nd.F90 Tests/FortranInterface/Advection_F/Source/compute_dt_mod.F90 Tests/FortranInterface/Advection_F/Source/evolve_mod.F90 Tests/FortranInterface/Advection_octree_F/Source/compute_dt_mod.F90 Tests/FortranInterface/Advection_octree_F/Source/evolve_mod.F90 Tools/CMake/AMReXSetDefines.cmake commit 5fd8671f9a9e7b319b03faae03c1fdfa371626f5 Author: Weiqun Zhang Date: Thu Jun 3 10:06:26 2021 -0700 Disable kernel fusing if gpu rdc is off. (#2083) Because the kernel fusing depends on rdc. Src/Base/AMReX_GpuFuse.H Src/Base/AMReX_GpuFuse.cpp Src/Base/AMReX_GpuLaunch.H Src/Base/AMReX_GpuLaunchFunctsG.H Src/Base/AMReX_GpuLaunchMacrosG.H Tools/CMake/AMReXSetDefines.cmake Tools/CMake/AMReX_Config.H.in Tools/GNUMake/Make.defs commit 5fca8d0268c461720d9665f0aef6a4d25bd11381 Author: Weiqun Zhang Date: Wed Jun 2 21:23:35 2021 -0700 ParallelFor for MultiFab/FabArray (#2073) Add an experimental feature that can be used to launch a single kernel to work on MultiFabs or FabArrays. An example is shown below, where `mf` is a MultiFab. ``` auto ma = mf.arrays(); amrex::ParallelFor(mf, [=] AMREX_GPU_DEVICE (int box_no, int i, int j, int k) { ma[box_no](i,j,k) = 1.0; }); Gpu::synchronize(); ``` It's still unclear how we handle tiling and OpenMP and how it performs on CPU. However, there is strong evidence that this improves performance for GPU runs when there are small boxes. The purpose of this PR is to get the feature into the AMReX repo first and do further tuning later. Since the new function is not used at all, it should not affect any codes. Src/Base/AMReX_CArena.cpp Src/Base/AMReX_FabArray.H Src/Base/AMReX_FabArrayBase.H Src/Base/AMReX_FabArrayBase.cpp Src/Base/AMReX_MFIter.H Src/Base/AMReX_MFParallelForC.H Src/Base/AMReX_MFParallelForG.H Src/Base/AMReX_NonLocalBC.H Src/Base/AMReX_TypeTraits.H Src/Base/CMakeLists.txt Src/Base/Make.package Src/EB/AMReX_MultiCutFab.H commit 7404b8a26a60e30cb6be4caf991522676c4bc36c Author: Andrew Myers Date: Wed Jun 2 20:34:20 2021 -0700 ParticleContainer-level reduction functions that work on tuples of data at once. (#2082) Src/Particle/AMReX_ParticleReduce.H Tests/Particles/ParticleReduce/main.cpp commit ea9345194adaf8a202a1b3a48d438697d008bdad Author: Axel Huebl Date: Wed Jun 2 20:15:29 2021 -0700 AMReX_PROBINT: Depends on AMReX_AMRLEVEL (#2081) CMake: do not enable if `AMReX_AMRLEVEL` is off anyway. Tools/CMake/AMReXOptions.cmake commit 3f70dd18d624d90161d323e218ccb398f86fecea Author: Weiqun Zhang Date: Wed Jun 2 20:14:41 2021 -0700 Clean up device random engine states (#2080) Src/Base/AMReX_Random.cpp Src/Base/AMReX_RandomEngine.H commit 241e4ef89f36363c2b00bf99141527db420cea4a Author: Weiqun Zhang Date: Wed Jun 2 20:14:12 2021 -0700 Arena aliases (#2078) Make The_Device_Arena and The_Managed_Arena an alias to The_Arena unless the memory allocation type in incompatible. This should help avoid unnecessary memory fragmentation. Src/Base/AMReX_Arena.cpp commit d20237636ccec63dcfb8540cfa681e0ebd893a40 Author: Axel Huebl Date: Wed Jun 2 16:34:29 2021 -0700 Docs: RDC Option (#2079) ## Summary Add a bit more detailed text to the AMReX_GPU_RDC option. This should help people searching for the option and explain the acronym. ## Additional background Did not yet find a way to slip in "separable compilation" as a term as well... Docs/sphinx_documentation/source/BuildingAMReX.rst Tools/CMake/AMReXOptions.cmake commit f7fd082bc8ab099d42784f69d6fe49dea3d388a7 Author: Axel Huebl Date: Wed Jun 2 13:39:30 2021 -0700 CMake CUDA: Filter out SM Before Pascal (#2077) Filter out unsupported CUDA architectures in CMake, for both legacy and CMake 3.20+ logic. Tools/CMake/AMReXUtils.cmake commit 2acc1caacf687c92e3cb07a54f7e7ba1f4251888 Author: Andrew Myers Date: Wed Jun 2 13:24:22 2021 -0700 fix bug introduced in PR 2063 (#2076) Src/Particle/AMReX_WriteBinaryParticleData.H commit 439663145ebf0d9f5e66caad738b2bd968ed9331 Author: Andrew Myers Date: Wed Jun 2 13:02:54 2021 -0700 fix signed / unsigned comparison warning (#2075) Src/Particle/AMReX_ParticleCommunication.cpp commit c099e342334c01c8fa2055b2e1e806c62bdf3035 Author: Axel Huebl Date: Wed Jun 2 10:36:22 2021 -0700 CMake 3.20+ CUDA: Policy & Include (#2074) ## Summary Do not include the deprecated CUDA setup with CMake 3.20+ and support new policy for CUDA_ARCHITECTURES. Also fix errors of the kind: ``` ERROR: set_target_properties called with incorrect number of arguments. Value in AMREX_CUDA_ARCHS is 53;60;61;70;75;80;86;86 ``` by quoting the semicolon-separated list. ## Additional background Those are regressions to #2012 that appear with our modernized logic in CMake 3.20+ with CUDA. I saw those regressions with [HiPACE++](https://github.com/Hi-PACE/hipace). CMakeLists.txt Tools/CMake/AMReXParallelBackends.cmake Tools/CMake/AMReXTargetHelpers.cmake commit e11fa66c9755e2909280930c387020af2849c1cf Author: Weiqun Zhang Date: Tue Jun 1 15:47:46 2021 -0700 GPU RDC Support (#2061) Disable GPU RDC by default. The user can enable it by `USE_GPU_RDC=TRUE` in gnu make and `-DAMReX_GPU_RDC=ON` in cmake. Remove CUDA specific random number functions that rely on rdc. Co-authored-by: Axel Huebl Docs/sphinx_documentation/source/BuildingAMReX.rst Src/Base/AMReX.H Src/Base/AMReX.cpp Src/Base/AMReX_Arena.H Src/Base/AMReX_Random.H Src/Base/AMReX_Random.cpp Tools/CMake/AMReXOptions.cmake Tools/CMake/AMReXTargetHelpers.cmake Tools/GNUMake/Make.defs Tools/GNUMake/comps/hip.mak Tools/GNUMake/comps/nvcc.mak Tools/Plotfile/AMReX_PPMUtil.cpp commit 70922cb7cb79bc06a8843f932367c72cf7071ef7 Author: Marc Day Date: Tue Jun 1 16:45:03 2021 -0600 Generalize fextract allowing to specify coords perpendicular to slicedir (#2035) Generalize fextract. Allow spec as coordinate + dir, so that 1D slice can be positioned as desired, within constraints of backward compatibility. Tools/Plotfile/fextract.cpp commit 702af37a91c60bf3916306884ef78319beb5b527 Author: Cody Balos Date: Tue Jun 1 13:11:23 2021 -0700 add interface between sundials SUNMemory and the Arena (#1800) ## Summary As @asalmgren @jmsexton03 @mic84 @ajnonaka @dwillcox @gardner48 @cswoodward and I discussed... this PR adds an interface between the AMReX Arena and the SUNDIALS SUNMemory API. This is currently used in Pele and will be used in Nyx also. In general, any AMReX based app that also uses SUNDIALS (and its provided data structures) will likely find this useful when targeting GPUs. CC @drummerdoc ## Additional background In SUNDIALS 5.4.0, the SUNMemory API was introduced to allow users to substitute their own memory management underneath SUNDIALS data structures. In Pele, using the AMReX arena underneath the SUNDIALS data structures results in as much as a 15% speedup. I have tried to structure the code to follow AMReX conventions. Since we also will be bringing in an interface between the MultiFab and the SUNDIALS N_Vector, I have also tried to structure the code so that it will be easy to add that interface in as well. Co-authored-by: Weiqun Zhang Co-authored-by: David Gardner Co-authored-by: Michele Rosso Co-authored-by: Jean M. Sexton Docs/sphinx_documentation/source/BuildingAMReX.rst Docs/sphinx_documentation/source/GPU.rst GNUmakefile.in Src/Base/AMReX.cpp Src/CMakeLists.txt Src/Extern/SUNDIALS/AMReX_SUNMemory.H Src/Extern/SUNDIALS/AMReX_SUNMemory.cpp Src/Extern/SUNDIALS/AMReX_Sundials.H Src/Extern/SUNDIALS/CMakeLists.txt Src/Extern/SUNDIALS/Make.package Tools/CMake/AMReXConfig.cmake.in Tools/CMake/AMReXOptions.cmake Tools/CMake/AMReXSetDefines.cmake Tools/CMake/AMReXThirdPartyLibraries.cmake Tools/CMake/AMReX_Config.H.in Tools/GNUMake/Make.defs Tools/GNUMake/packages/Make.sundials commit ef1c484bc467486be3cdff6c90884b0352c39ab5 Author: Andrew Myers Date: Tue Jun 1 12:27:43 2021 -0700 Do not rely on managed memory in ParticleContainer::WriteAsciiFile (#2070) This function is mainly used for small runs / debugging and does not provide filters, hence it should be fine just to copy all the particles to pinned host memory before writing them to disk. Src/Particle/AMReX_ParticleIO.H commit ecffb6a400180f2b2497e66f7c655fafd6bd6eb4 Author: Axel Huebl Date: Tue Jun 1 12:03:13 2021 -0700 HIP ROCTX: USE_RPATH Control (#2064) Just rely on USE_RPATH instead for GNUmake. Follow-up to #2057 Tools/GNUMake/comps/hip.mak commit e7a725644a3eb72dba5c9cd74d35eff4d96af8a8 Author: Andrew Myers Date: Tue Jun 1 12:02:08 2021 -0700 Particle plotfile/checkpoint without managed memory. (#2063) Src/Particle/AMReX_NeighborList.H Src/Particle/AMReX_ParticleIO.H Src/Particle/AMReX_ParticleTile.H Src/Particle/AMReX_Particles.H Src/Particle/AMReX_WriteBinaryParticleData.H commit b5826c021316179174228e56ad74736c8823caba Author: Weiqun Zhang Date: Tue Jun 1 09:50:40 2021 -0700 Support preprocessing of AMREX_SPACEDIM in input files. (#2049) Src/Base/AMReX_ParmParse.H Src/Base/AMReX_ParmParse.cpp commit 1aa8d382af3195e96eea4625dc452c832dfedc53 Author: Weiqun Zhang Date: Tue Jun 1 08:06:13 2021 -0700 Update CHANGES for 21.06 (#2072) CHANGES commit 55ad11dae86fca6737e6e7b8c1656100e5a3bead Author: Axel Huebl Date: Fri May 28 13:03:39 2021 -0700 CI: Cover Fortran (#2067) * CI: Cover Fortran It looks like at some point we changed the `AMReX_Fortran` default from ON to OFF and forgot to adjust CI. This re-enables Fortran compile tests. * CMake: Fix Install Smoke Test (Fortran) Avoid erroring out if Fortran is enabled. .github/workflows/linux.yml Tests/CMakeTestInstall/CMakeLists.txt Tools/CMake/AMReXInstallHelpers.cmake commit 1492a91642e47515b688e58047232daa148196e0 Author: Mark Meredith Date: Fri May 28 00:59:42 2021 -0400 Set HIP_LIBRARIES before using (#2065) Tools/CMake/AMReXParallelBackends.cmake commit 52847883765f041e908350c4d2c03799f6924b1c Author: Axel Huebl Date: Thu May 27 18:09:02 2021 -0700 CMake: HIP Compiler - don't fail (#2062) Potentially, users might use `CC` (Cray Prg Envs) and later `clang++` (underlying compiler) as compilers. Only warn if not met. ROCm/Cray environments are currently being developed and we want to keep things flexible for codes trying to use the compiler wrapper instead of hpcc. Tools/CMake/AMReXParallelBackends.cmake commit 2c2c6992b4169613ba0c956a2a2bd120552f446a Author: jmsexton03 Date: Thu May 27 15:38:31 2021 -0700 Add roctx ranges to TinyProfiler (#2057) ## Summary Adds roctx markup similarly to existing nvtx markup. ## Additional background New features are controlled by added compile flag USE_ROCTX. This assumes the location of roctracer/libroctracer64 and roctracer/libroctx64 are similar to other ROCm library installations such as rocrand. Example run to generate results.json ``` export HIP_PATH=/path/to/rocm/root cd amrex/Tests/GPU/CNS/Exec/Sod make USE_HIP=TRUE TINY_PROFILE=TRUE USE_ROCTX=TRUE USE_MPI=FALSE NO_CONFIG_CHECKING=TRUE srun -n 1 ${HIP_PATH}/bin/rocprof --hsa-trace --stats --timestamp on --roctx-trace ./CNS3d.*.TPROF.HIP.ex inputs ``` This results.json file can be viewed with a browser as described in the documentation using chrome://tracing/ https://github.com/ROCm-Developer-Tools/rocprofiler/blob/amd-master/doc/rocprof.md#43rd-party-visualization-tools Co-authored-by: Axel Huebl Co-authored-by: Weiqun Zhang .github/workflows/dependencies/dependencies_hip.sh Docs/sphinx_documentation/source/BuildingAMReX.rst Src/Base/AMReX_GpuDevice.cpp Src/Base/AMReX_TinyProfiler.H Src/Base/AMReX_TinyProfiler.cpp Tools/CMake/AMReXOptions.cmake Tools/CMake/AMReXParallelBackends.cmake Tools/CMake/AMReXSetDefines.cmake Tools/CMake/AMReX_Config.H.in Tools/GNUMake/comps/hip.mak commit 01361ca148d01c6416e6e87ba672c83b694f3385 Author: kngott <30483578+kngott@users.noreply.github.com> Date: Thu May 27 14:46:35 2021 -0700 Clean up some unused variables. (#2060) Src/LinearSolvers/MLMG/AMReX_MLMG.cpp Tests/MultiBlock/IndexType/main.cpp Tests/Particles/GhostsAndVirtuals/main.cpp commit 7dcbf3946a95a96b05f6c9ca6c2ee4d7fb5e8dca Author: Mark Meredith Date: Thu May 27 17:22:44 2021 -0400 Link with HIP_LIBRARIES (#2055) * Link with HIP_LIBRARIES * Try LINK_LANGUAGE GenExpr * Only link with HIP_LIBRARIES for C++ only config * fix * Add HIP_LIBRARIES for all non-Fortran * CI: HIP no-Fortran Build * ROCm spelling Co-authored-by: Axel Huebl .github/workflows/linux.yml Tools/CMake/AMReXParallelBackends.cmake commit 0afebba6ae751a2b58fe3e1f4e8c05e2b9cf2025 Author: Lucas Esclapez <13371051+esclapez@users.noreply.github.com> Date: Thu May 27 12:46:15 2021 -0700 Extrapolator with nGrow > 1. (#2054) Extend the AMReX_Extrapolator in order to handle more than 1 layer of ghost cells. This is done by repeating the original algorithm several times while increasing the box size by 1 at every iteration. Src/Amr/AMReX_Extrapolater.H Src/Amr/AMReX_Extrapolater.cpp Src/Amr/AMReX_extrapolater_2D_K.H Src/Amr/AMReX_extrapolater_3D_K.H commit db8e47927065fcf283f6aae631f91a07b222876b Author: kngott <30483578+kngott@users.noreply.github.com> Date: Wed May 26 12:40:32 2021 -0700 Perlmutter Make: CUDA_PATH (#2059) * Perlmutter: find CUDA_ROOT or CUDA_PATH. * Cover all bases. Tools/GNUMake/sites/Make.nersc commit affbd9fe2ca8fecab786c465911067ca05a2c225 Author: drangara <69211175+drangara@users.noreply.github.com> Date: Wed May 26 15:29:53 2021 -0400 Fix to correct CUDA flag and add missing libraries in hypre cmake (#2056) * Fix to correct CUDA flag and add missing libraries in hypre cmake * use CUDAToolkit instead of find_library Tools/CMake/AMReXThirdPartyLibraries.cmake commit 4b29e5d85c2d0eb09b21f50a05a7ef590919d18b Author: kngott <30483578+kngott@users.noreply.github.com> Date: Wed May 26 10:12:30 2021 -0700 Perlmutter make (#2058) * Perlmutter make first draft. Allow Pele to start testing. * Keep mpi_nvidia. Tools/GNUMake/sites/Make.nersc commit 33d8bbb1b15009c68bc71ee72c9b067b730148d5 Author: Weiqun Zhang Date: Tue May 25 15:16:38 2021 -0700 Fix compilation of the TensorOverset test (#2053) Tests/LinearSolvers/TensorOverset/MyTest.cpp commit dedbdce8b963167b5ae8a64ca82cf9c8ce6f5d86 Author: Weiqun Zhang Date: Tue May 25 15:16:16 2021 -0700 Remove IndexSequence (#2052) Since we use C++14 now, our own implementation of index sequence can be removed. Src/Base/AMReX_IndexSequence.H Src/Base/AMReX_Tuple.H Src/Base/CMakeLists.txt Src/Base/Make.package Src/EB/AMReX_EB2_IF_Intersection.H Src/EB/AMReX_EB2_IF_Union.H Src/Particle/AMReX_ParticleArray.H commit 12b8518603dd480e01119aa445410d8684858472 Author: Weiqun Zhang Date: Tue May 25 11:49:43 2021 -0700 Use Cray wrappers to get mpi flags on spock (#2051) Tools/GNUMake/sites/Make.olcf commit 0589d3a76098d90e0f7d364234233643cb212699 Author: Mark Meredith Date: Tue May 25 14:30:38 2021 -0400 CMake link with roc::rocprim for HIP (#2050) * CMake link with roc::rocprim for HIP * link with HIP_LIBRARIES * Update Tools/CMake/AMReXParallelBackends.cmake Co-authored-by: Axel Huebl Co-authored-by: Axel Huebl Tools/CMake/AMReXParallelBackends.cmake commit fd5163bb48fdc6b802cf3e9a8d2c861954475e8c Author: Andrew Myers Date: Tue May 25 09:24:32 2021 -0700 Fix PODVector to work without managed memory. (#2048) In a number of places in `amrex::PODVector`, we were always running operations on the host, assuming the presence of managed memory. This fixes these to run either on the host or on the device, depending on the execution policy of the `PODVector`'s `Allocator` (meaning, for managed and device vectors we run on the device, for host and pinned we run on the host). Additionally, this improves the existing test of `PODVector` by passing it `amrex::the_arena_is_managed=0`, which is more sensitive to errors. This fixes an issue reported by @jmsexton03 regarding Nyx and the InitFromBinaryFile capabilities. Src/Base/AMReX_PODVector.H Tests/GPU/Vector/inputs Tests/GPU/Vector/main.cpp commit 11bdf9e45e39070d5072562db633a1da45d36bd6 Author: Andrew Myers Date: Tue May 25 09:13:36 2021 -0700 When building neighbor masks, perform setVal on device so it will work without managed memory. (#2046) This fixes an issue reported by Chris Lishka from Intel re: mfix. Src/Particle/AMReX_NeighborParticlesGPUImpl.H commit e6654533565fad924793bb3515dd1ee086d1bccb Author: kngott <30483578+kngott@users.noreply.github.com> Date: Tue May 25 09:11:52 2021 -0700 Async Output Docs (#2041) Docs/sphinx_documentation/source/IO.rst commit af5a869484df0ee842afcbf231473d1c472499e5 Author: Weiqun Zhang Date: Mon May 24 17:22:51 2021 -0700 Option to set amd gpu targets (#2038) In GNU make, one can set amd gpu targets with make argument AMD_ARCH or environment variable AMREX_AMD_ARCH. Also set MPI wrapper's underlying compiler to hipcc. This is needed if the machine is unknown. For machines known to the gnu make build system, we could either use mpicxx or hipcc -Ipath/to/mpi/include. Docs/sphinx_documentation/source/BuildingAMReX.rst Tools/GNUMake/Make.defs Tools/GNUMake/comps/hip.mak Tools/GNUMake/sites/Make.frontier-coe Tools/GNUMake/sites/Make.olcf commit e28bbc2d08e79ccb9103d38a9df30aac9f335b16 Author: Weiqun Zhang Date: Mon May 24 17:19:55 2021 -0700 Non-GPU build on spock (#2047) Do not link to gpu aware mpi if it's not a GPU build. Tools/CMake/AMReXUtils.cmake Tools/GNUMake/sites/Make.olcf commit 727829666aa490873a747f3d00c2b3c7ddc91508 Author: Axel Huebl Date: Mon May 24 12:34:53 2021 -0700 Work-Around: oneAPI 2021.3.0 SYCL with oneDPL (#2025) Add recommended work-around flags to oneAPI 2021.3.0 SYCL compilers. https://software.intel.com/content/www/us/en/develop/articles/intel-oneapi-dpcpp-library-release-notes.html#inpage-nav-2-3 ``` The use of oneDPL together with the GNU C++ standard library (libstdc++) version 9 or 10 may lead to compilation errors (caused by oneTBB API changes). To overcome these issues, include oneDPL header files before the standard C++ header files, or disable parallel algorithms support in the standard library. For more information, please see Intel® oneAPI Threading Building Blocks (oneTBB) Release Notes. ``` ``` An application using Parallel STL algorithms in libstdc++ versions 9 and 10 may fail to compile due to incompatible interface changes between earlier versions of Threading Building Blocks (TBB) and oneAPI Threading Building Blocks (oneTBB). Disable support for Parallel STL algorithms by defining PSTL_USE_PARALLEL_POLICIES (in libstdc++ 9) or _GLIBCXX_USE_TBB_PAR_BACKEND (in libstdc++ 10) macro to zero before inclusion of the first standard header file in each translation unit. ``` Docs/sphinx_documentation/source/GPU.rst Tools/CMake/AMReXOptions.cmake Tools/CMake/AMReXSYCL.cmake Tools/CMake/AMReXSetDefines.cmake Tools/CMake/AMReX_Config.H.in commit 646d5f63445ff46e4df23cbdb1fbe662ca1708fb Author: Axel Huebl Date: Mon May 24 11:20:34 2021 -0700 ParamParse: Find Entries under Prefix (#2043) Extend the "unused params" checks to find generally parameters under a given inputs prefix. Demonstrator & test in: https://github.com/AMReX-Codes/amrex-tutorials/pull/7 Src/Base/AMReX_ParmParse.H Src/Base/AMReX_ParmParse.cpp commit 8935b4ca1e8b8ab2e69dc8f1f85c84a276e8efe0 Author: Max Katz Date: Mon May 24 09:14:23 2021 -0700 Add relative gradient to AMRErrorTag (#2044) Src/AmrCore/AMReX_ErrorList.H Src/AmrCore/AMReX_ErrorList.cpp commit 25b08dc3a46b9b2b96f8e004cb15e1bf67e94bef Author: Weiqun Zhang Date: Fri May 21 10:23:39 2021 -0700 Workaround argument mismatch in MPI reduce for gfortran >= 10 (#2042) In #2040, we added -fallow-argument-mismatch as a workaround. This commit reverts the change and implements an alternative approach that does not require that compiler compiler. We just need to make sure that there are no argument mismatches in the same file. Src/F_Interfaces/Base/AMReX_fi_mpi_mod.F90 Src/F_Interfaces/Base/AMReX_mpi_reduce_int.F90 Src/F_Interfaces/Base/AMReX_mpi_reduce_real.F90 Src/F_Interfaces/Base/AMReX_parallel_mod.F90 Src/F_Interfaces/Base/Make.package Src/F_Interfaces/CMakeLists.txt Tools/GNUMake/comps/gnu.mak Tools/GNUMake/comps/hip.mak Tools/GNUMake/comps/llvm.mak commit 2255a3c7196ca7ac9225f58edb82c150a2cc0ec9 Author: Andrew Myers Date: Thu May 20 16:17:37 2021 -0700 Wrappers for builtin clz extensions and tests thereof. (#2036) Src/Base/AMReX_Algorithm.H Src/Base/AMReX_Extension.H Tests/CLZ/CMakeLists.txt Tests/CLZ/GNUmakefile Tests/CLZ/Make.package Tests/CLZ/main.cpp Tests/CMakeLists.txt commit 90977cbfe14d79fa391fe4607c5b59513b9f1413 Author: Weiqun Zhang Date: Thu May 20 12:54:13 2021 -0700 Workaround for gfortran 10 (#2040) Starting from gfortran 10, mismatches between actual and dummy argument lists in a single file have been rejected with an error. This causes issues for MPI calls. This commit adds -fallow-argument-mismatch as a workaround. Tools/GNUMake/comps/gnu.mak Tools/GNUMake/comps/hip.mak Tools/GNUMake/comps/llvm.mak commit 2bbfef42fcd4b5d4386a6145f48eddab7692f907 Author: Axel Huebl Date: Thu May 20 10:27:06 2021 -0700 HIP: AMREX_AMD_ARCH Env Variable (#2034) Adding an AMREX_AMD_ARCH env variable to control the AMD GPU arch from user environments & profiles. Docs/sphinx_documentation/source/GPU.rst Tools/CMake/AMReXOptions.cmake commit 16714342b3c08a69438313a9007f93d8c5b7211b Author: Weiqun Zhang Date: Tue May 18 09:50:30 2021 -0700 Add spock support to GNU make (#2028) Tools/GNUMake/Make.machines Tools/GNUMake/sites/Make.olcf commit a1faf316ff6556aabf76fd6d605bf4a59d6c0f59 Author: Weiqun Zhang Date: Tue May 18 08:32:04 2021 -0700 nvcc --display-error-number --diag-error errNum (#2033) Since 11.2, nvcc can display a diagnostic number for any warning message and it can turn specified warnings into errors These two new flags are added to the build system, and it's now an error to write a __device__ variable in a host function. Co-authored-by: Michele Rosso Tools/CMake/AMReXParallelBackends.cmake Tools/CMake/AMReX_SetupCUDA.cmake Tools/GNUMake/comps/nvcc.mak commit aff93b8a26348bb5b42d6af9c3a27ad44e9389f9 Author: Andrew Myers Date: Tue May 18 08:21:08 2021 -0700 Build neighbor masks without managed memory (#2030) Construct these vectors on the host then copy to the device for platforms without managed memory. Src/Particle/AMReX_NeighborParticlesGPUImpl.H commit 79f79aae9f5cc21c09b0353bc0612f41daed4bad Author: Weiqun Zhang Date: Mon May 17 19:13:16 2021 -0700 Option to use oneDPL (#2027) If both USE_ONEDPL and USE_DPCPP are TRUE in GNU make, use oneDPL for the pointer based inclusive and exclusive sum. Also updated the memory fence calls and fixed the compilation for the scan of long for dpcpp. Src/Base/AMReX_FBI.H Src/Base/AMReX_Scan.H Tools/GNUMake/Make.defs Tools/GNUMake/comps/dpcpp.mak commit c6c493bb458515e3ba4b856af097562dcb49a19b Author: mic84 Date: Mon May 17 18:28:54 2021 -0700 CMake: add support for CMake >= 3.20 (#2012) * CMake: group CUDA options in a standalone file * CMake: continue enabling new CUDA support * CMake: make set_mininum_cxx_compiler_version work for a generic language * CMake: rename AMReX_Utils.cmake -> AMReXUtils.cmake * CMake: fix bug * CMake: get rid of NVCC_VERSION_MAJOR and NVCC_VERSION_MINOR * CMake: some upgrades * CMake: upgrade CUDA support * CMake: fix bug * CMake: forgot to guard a set_target_property * CMake: handle CMAKE_CUDA_ARCHITECTURES * CMake: add CUDA flags * CMake: bump up minimum GCC version to 5.1. * CMake: add print for debugging * CMake: add support for PLATFORM_ID genex * Revert "CMake: add print for debugging" This reverts commit 146f645ebae1ac5525e1f5f11cbd167e479dd00f. * CMake: evaluate SHELL genex * CMake: fix printing out of CUDA flags * CMake: fix bug * Update Doc * Update Docs/sphinx_documentation/source/GPU.rst Co-authored-by: Axel Huebl Co-authored-by: Axel Huebl CMakeLists.txt Docs/sphinx_documentation/source/GPU.rst Src/CMakeLists.txt Tools/CMake/AMReXCUDAOptions.cmake Tools/CMake/AMReXConfig.cmake.in Tools/CMake/AMReXFlagsTargets.cmake Tools/CMake/AMReXGenexHelpers.cmake Tools/CMake/AMReXParallelBackends.cmake Tools/CMake/AMReXTargetHelpers.cmake Tools/CMake/AMReXUtils.cmake Tools/CMake/AMReX_Config.cmake Tools/CMake/AMReX_SetupCUDA.cmake Tools/CMake/AMReX_Utils.cmake commit 9cf1ca565447d5776bd7e7fadd01fdf2e114d53e Author: Weiqun Zhang Date: Mon May 17 17:46:06 2021 -0700 Turn on RDC support for HIP (#2029) This will allow `extern` device variables. Tools/GNUMake/comps/hip.mak commit 6f1085a23de1f7ee1a5f92dd173e984c59839511 Author: Max Katz Date: Sun May 16 11:05:05 2021 -0700 Device profilerStart and profilerStop APIs (#2026) Src/Base/AMReX_GpuDevice.H Src/Base/AMReX_GpuDevice.cpp commit c6a8fa243e8449938b8b4b5b16a9e95df9487d48 Author: Emmanuel Motheau Date: Sun May 16 11:04:20 2021 -0700 Workaround to fix the AugmentPlotfile tool (#2023) Tools/C_util/AugmentPlotfile/AugmentPlotfile.cpp commit 442c6302b76ea0ce7d0018f69ae957686d4c4add Author: Weiqun Zhang Date: Fri May 14 18:46:38 2021 -0700 Use rocm 4.1.1 for hip ci (#2024) .github/workflows/dependencies/dependencies_hip.sh .github/workflows/linux.yml commit 6795c1f45833f1784b24d2277664d670f8c7171c Author: Max Katz Date: Fri May 14 17:41:34 2021 -0700 Make nvcc play nicely with CC wrapper (#2019) Tools/GNUMake/Make.defs Tools/GNUMake/comps/cray.mak commit ab84978412f2a00aabf966cbc54b87da39f9ab60 Author: Max Katz Date: Fri May 14 17:34:08 2021 -0700 Add Perlmutter to list of NERSC hosts (#2021) Tools/GNUMake/Make.machines Tools/GNUMake/sites/Make.nersc commit 65226552590664d61d32b2d3d8e0de7e691d7e62 Author: Max Katz Date: Fri May 14 17:32:59 2021 -0700 Cray ftn does not accept isystem (#2020) Tools/GNUMake/Make.defs commit d0210641e10dd257a5f2942e992e3c9ab5a4fcfb Author: Max Katz Date: Fri May 14 17:31:38 2021 -0700 Clang Cray can also show up with explicit Clang identifier (#2018) Tools/GNUMake/comps/cray.mak commit 2756788d7b06c3cc7ad5c471ee271edcfa484845 Author: kngott <30483578+kngott@users.noreply.github.com> Date: Fri May 14 12:20:10 2021 -0700 Label function HOST_DEVICE. (#2022) Src/Base/AMReX_RealBox.H commit f7d87b8be6b9bc9373aa17e8b2805256afcd0c55 Author: Max Katz Date: Fri May 14 08:37:11 2021 -0700 Remove restrictions on nvcc host compiler (#2017) These restrictions were originally enforced back in the days of CUDA Fortran, there is no need to require this now. Tools/GNUMake/Make.defs commit 6c9a3f73f3034b5e2b299b6d0a6a7a4333dc1875 Author: Weiqun Zhang Date: Wed May 12 21:58:45 2021 -0700 Remove minBlocksPerMultiprocessor from launch bounds (#2015) After reading CUDA Programming Guide, I think we should remove minBlocksPerMultiprocessor from launch bounds, so that it's closer to what we had before. https://docs.nvidia.com/cuda/cuda-c-programming-guide/index.html#launch-bounds Src/Base/AMReX_GpuLaunchGlobal.H commit 19a2152a097dda52df64e2e42ba60e267fda74be Author: Weiqun Zhang Date: Wed May 12 20:53:02 2021 -0700 Loosen a HIP regresstion test's tolerance (#2014) Tools/RegressionTesting/AMReX-hip-tests.ini commit ad3eea065f3387fd374d9480a31419ba3bbb72bc Author: Weiqun Zhang Date: Wed May 12 20:49:06 2021 -0700 Launch bounds (#2013) Set launch bounds to AMREX_GPU_MAX_THREADS (256 by default) for all kernel launches. This helps resolve some HIP compiler issues we have had recently. Src/Base/AMReX_GpuLaunch.H Src/Base/AMReX_GpuReduce.H commit 50847f55331c15f10e38f4e710c0fef292737fe5 Author: Weiqun Zhang Date: Wed May 12 07:45:20 2021 -0700 Disable HIP CI (#2011) ROCm 4.2 has a bug that causes the ci to fail. .github/workflows/linux.yml commit 73dc2422bb07c7346c29c01d1bbfa5a83352ceae Author: asalmgren Date: Tue May 11 20:45:04 2021 -0700 Test on vfrac rather than isRegular when doing the slope limiting -- we were doing this in one (#2008) place but not another place which made the behavior inconsistent. Src/EB/AMReX_EB_slopes_K.H commit 20be66e90b6685865f534ab83b1b8850ad848bcb Author: Weiqun Zhang Date: Tue May 11 16:44:04 2021 -0700 Arena initial size (#2007) Add ParmParse parameters to allow the users to change the initial size of various Arenas. In particular this allows the users to not allocate any managed memory if they set `amrex.the_arena_is_managed=0 amrex.the_managed_arena_init_size=0`. Src/Base/AMReX_Arena.cpp commit 8a650f69446681320504fe5156591dc547bcb0c9 Author: Weiqun Zhang Date: Tue May 11 16:34:48 2021 -0700 Use CPU memory in memory pools for OpenMP threads (#2006) The memory pools are primarily used by Fortran functions as scratch space inside OpenMP parallel regions. There are no reasons to allocate them in managed device memory. Src/Base/AMReX_MemPool.cpp commit bf524ee95ff4a068ddf0a27599d4ce652e25b3cb Author: Weiqun Zhang Date: Tue May 11 13:02:03 2021 -0700 No casting to unsigned long long needed in iMultiFab::sum (#2005) Because we no longer use atomics in reduction, we do not need to cast to unsigned long long in iMultiFab::sum anymore. Have to install tgtermes.sty to build the docs. .github/workflows/dependencies/documentation.sh Src/Base/AMReX_iMultiFab.cpp commit fb0c16e34b9340e68ee63013e64570240e436815 Author: Max Katz Date: Sat May 8 16:12:26 2021 -0700 Avoid call to amrex::max with only one argument in 1D (#2004) Src/LinearSolvers/MLMG/AMReX_MLCellLinOp.cpp commit be62957c227508cda828bd4d4b7ea2ab9d3a2b87 Author: Weiqun Zhang Date: Fri May 7 14:03:21 2021 -0700 Cray complier type detection (#2001) There are two types of Cray compilers, the classical and clang based. The version number is not a reliable way to determine the type. For example, cce/9 on Cori has both versions. Cray compilers for aarch64 on Ookami are classical. Thus we switch to grep "LLVM" in `CC --version`. This works on both Cori and Ookami. Tools/GNUMake/comps/cray.mak commit 3c3bb354d7fb8c9d4b1072d5aabab74b7cc34117 Author: Weiqun Zhang Date: Fri May 7 13:25:22 2021 -0700 Arm Clang (#2002) Add support for Arm Clang in GNU make. Tools/GNUMake/Make.defs Tools/GNUMake/comps/armclang.mak Tools/libamrex/configure.py Tools/libamrex/mkconfig.py commit 9496af4b7bc2e768608c026b06e3afe6f55ce01c Author: Andrew Myers Date: Fri May 7 10:56:30 2021 -0700 Move Morton code utility functions somewhere they can be reused. (#1999) Also ports them to GPU. Src/Base/AMReX_DistributionMapping.cpp Src/Base/AMReX_Morton.H Src/Base/CMakeLists.txt Src/Base/Make.package commit ce04df6c1cb870860672c218f3e3b641f1ec24ec Author: Andrew Myers Date: Wed May 5 17:05:05 2021 -0700 Templated implementation of ostream operator for both Dim3 and XDim3 (#1997) Src/Base/AMReX_Dim3.H Src/Base/AMReX_Dim3.cpp Src/Base/CMakeLists.txt Src/Base/Make.package commit 8d7d25ff6ae426f31cd609a97d66bc8608fc03d1 Author: Weiqun Zhang Date: Wed May 5 16:57:15 2021 -0700 Disable HIP device assertion (#1998) It makes the compliation very slow and it does not currently work. Src/Base/AMReX.H Src/Base/AMReX_GpuQualifiers.H commit a2a23d75fafb05f7f2a974d0fe7bf5ff584a2823 Author: Andrew Myers Date: Wed May 5 14:59:01 2021 -0700 Generalize elemwiseMin and elemwiseMax. (#1996) This allows these functions to be called on `XDim3` objects, as well as any other object that has `x`, `y`, and `z` data members, instead of just `Dim3`. Additionally, it makes them variadic so they work on any number of objects, instead of just two. Src/Base/AMReX_Algorithm.H commit c1375ccb916f35caca50cc52c72535025e179124 Author: Weiqun Zhang Date: Wed May 5 14:29:04 2021 -0700 Add new parameter to BoxArray::contains(BoxArray) (#1995) In regrid, we need to figure out how many cells we need to grow the current coarse level grids in order for the fine grids to be properly nested. We used BoxArray::contains(BoxArray) in a loop and grew the coarse BoxArray in the loop. In this commit, we add a new parameter to BoxArray::contains(BoxArray) to specify the number of grow cells without having to actually grow the BoxArray. This is better than the previous version because growing BoxArray would invalidate the hash data in BoxArray. We also modify the implementation of BoxArray::contains(Box) in this commit. Src/Base/AMReX_BoxArray.H Src/Base/AMReX_BoxArray.cpp commit b8dad4457ae795d39bfc4d9e7c3c1927102873df Author: kngott <30483578+kngott@users.noreply.github.com> Date: Tue May 4 15:50:50 2021 -0700 Remove Warnings in DEBUG in AssignParticle Test (#1993) Tests/Particles/AssignDensity/main.cpp commit c6702b5772d40bd7d34aeb65fa445510e8bce01e Author: Weiqun Zhang Date: Tue May 4 15:49:22 2021 -0700 Optimizing away temporary BoxArrays in AmrMesh::MakeNewGrids (#1994) Previously, we stored the information needed for proper nesting in BoxList. Then when they were used, they were repeatedly converted to BoxArray for hash based fast intersection. In this commit, we store them directly in BoxArray, thus removing those hidden temporaries. Src/Amr/AMReX_Amr.H Src/Amr/AMReX_Amr.cpp Src/AmrCore/AMReX_AmrMesh.cpp Src/AmrCore/AMReX_Cluster.H Src/AmrCore/AMReX_Cluster.cpp Src/AmrCore/AMReX_TagBox.H Src/AmrCore/AMReX_TagBox.cpp Src/Base/AMReX_BoxDomain.H Src/Base/AMReX_BoxDomain.cpp commit 57e8af38fa513674ad9097f6bc6858d8eb669015 Author: kngott <30483578+kngott@users.noreply.github.com> Date: Tue May 4 14:43:40 2021 -0700 Non-blocking SumBoundary (#1991) Non-blocking SumBoundary, using the stored MF pointer in PCData. Src/Base/AMReX_MultiFab.H Src/Base/AMReX_MultiFab.cpp commit bc335117a000dd5df511d2dc298ce041e09bafee Author: Weiqun Zhang Date: Tue May 4 12:40:10 2021 -0700 Remove an expensive assertion in BoxArray::contains (#1992) The assertion gets extremely expensive for large runs with long skinny patches when it's used by ClusterList::intersect. Src/Base/AMReX_BoxArray.cpp commit 50280bc83decd9c04df80697db3d95ca2d70b947 Author: kngott <30483578+kngott@users.noreply.github.com> Date: Tue May 4 12:29:26 2021 -0700 Docs: Add non-blocking comms. (#1988) Very brief discussion of non-blocking comms for the documentation (currently, I can't find any reference). Docs/sphinx_documentation/source/Basics.rst commit 741fdd27e355d2c5992facd79d41fc283c4ea777 Author: Zach Jibben Date: Mon May 3 22:12:21 2021 -0600 Define BL_USE_MPI compilation flag when MPI and Fortran are enabled (#1990) Resolves #1989 Tools/CMake/AMReXSetDefines.cmake commit b451e90934eb0a8f0d10878e4e087b3aa933d78e Author: kngott <30483578+kngott@users.noreply.github.com> Date: Mon May 3 16:10:47 2021 -0700 FBData object (#1983) Src/Base/AMReX_FabArray.H Src/Base/AMReX_FabArrayCommI.H commit b27fc508b608eb6f78a9d9bcfe11f1ef0a226461 Author: kngott <30483578+kngott@users.noreply.github.com> Date: Mon May 3 15:41:18 2021 -0700 Deprecate FabArray::copy() (#1985) This deprecates the legacy `FabArray::copy()` in favor of the better named `FabArray::ParallelCopy()` functions. This should help alleviate the problem of confusion between `FabArray::copy()` and `MultiFab::Copy()`, which is a local copy. Also removes "copy" calls from the rest of the repo. Src/Amr/AMReX_AuxBoundaryData.cpp Src/AmrCore/AMReX_FillPatchUtil.cpp Src/AmrCore/AMReX_FillPatchUtil_I.H Src/Base/AMReX_FACopyDescriptor.H Src/Base/AMReX_FabArray.H Src/Base/AMReX_MultiFab.cpp Src/Base/AMReX_MultiFabUtil.cpp Src/Base/AMReX_MultiFabUtilI.H Src/Base/AMReX_VisMF.cpp Src/Boundary/AMReX_FabSet.cpp Src/EB/AMReX_EBMultiFabUtil.cpp Src/Extern/ProfParser/AMReX_BLProfUtilities.cpp Src/Extern/amrdata/AMReX_DataServices.cpp Src/Particle/AMReX_ParticleMesh.H Src/Particle/AMReX_TracerParticles.cpp commit ff82946b64fa3dcd4c01df52e564c83ecd752089 Author: kngott <30483578+kngott@users.noreply.github.com> Date: Mon May 3 15:29:33 2021 -0700 PCData object for non-blocking ParallelCopy (#1973) Src/Base/AMReX_FabArray.H Src/Base/AMReX_FabArrayCommI.H commit 783cbf183adac05004bf1b048942888243e59032 Author: Weiqun Zhang Date: Mon May 3 10:39:06 2021 -0700 Fix proper nesting issue for large error buffer (#1987) Reimplement the code that ensures proper nesting by projecting the fine grids onto the coarse. The previous implementation has a bug that causes it to fail when the error buffer is very large. In the very beginning of BoxLib, the enforcement of proper nesting was actually done in a similar way as the approach in this commit (i.e., calling TagBoxArray::setVal(BoxArray,TagBox::CLEAR) with projected down BoxArray). However, it had the issue of buffering the buffer cells, creating unnecessarily big refinement patches. That issue was fixed in a block of very complicated code that was hard to reason, also in the very beginning of BoxLib days. Since it worked, the code had not been touched at all. However, it was reported recently it failed when a very large error buffet was used. In this commit, we switched back to the original setVal approach and it is performed after the tagged cells are buffered already, thus avoiding the issue of buffering the buffer cells. Furthermore, it is performed after the tags are coarsened, thus performing less work. This also adds a check to make sure that blocking factors do not vary too much between levels. Otherwise regrid will break. Src/AmrCore/AMReX_AmrMesh.cpp Src/AmrCore/AMReX_TagBox.cpp commit 2b397a9ba0c99376177ec33138e8ed3de707cd8a Author: Houjun Tang Date: Mon May 3 10:36:17 2021 -0700 Update HDF5 I/O to use new HDF5 Async VOL APIs (#1948) Src/Base/AMReX_PlotFileUtil.cpp Src/Particle/AMReX_ParticleHDF5.H Src/Particle/AMReX_ParticleUtil.H Src/Particle/AMReX_ParticleUtil.cpp Src/Particle/AMReX_Particles.H Tests/HDF5Benchmark/GNUmakefile Tests/HDF5Benchmark/main.cpp commit c670d955e90912515389f4715321abe67c4cf206 Author: Mark Meredith Date: Mon May 3 11:59:13 2021 -0400 Remove unused variables (fix warnings) (#1976) Found when building MFiX Src/EB/AMReX_EB_slopes_K.H commit cff96a9f23638819cff688071ff2588d8cfadd63 Author: Weiqun Zhang Date: Sun May 2 10:26:43 2021 -0700 Update CHANGES for 21.05 (#1986) CHANGES commit 8e8b3d75376a081c02c1735fe8d1520a12bfb9f3 Author: PaulMullowney <60452402+PaulMullowney@users.noreply.github.com> Date: Fri Apr 30 10:19:42 2021 -0600 Hypre Default Params (#1984) This PR reverts the default values for bamg_relax_type back to 6 (Symmetric Gauss Seidel). This appears to be more robust than 11 (2 stage Gauss Seidel). Src/Extern/HYPRE/AMReX_HypreIJIface.cpp commit fe2830d4dd426c96cc0454dfb337e08bd489c7e9 Author: PaulMullowney <60452402+PaulMullowney@users.noreply.github.com> Date: Thu Apr 29 16:48:56 2021 -0600 Fix to this parameter (#1982) Src/Extern/HYPRE/AMReX_HypreIJIface.cpp commit bfca28a1f40a9774790d09c3247b39675986aaae Author: PaulMullowney <60452402+PaulMullowney@users.noreply.github.com> Date: Thu Apr 29 10:12:41 2021 -0600 modifying some hypre params and changing defaults. (#1981) Src/Extern/HYPRE/AMReX_HypreIJIface.cpp commit 47d580e2943dcc90998b8852ead53baaf4dd9a76 Author: cdaley Date: Wed Apr 28 19:57:11 2021 -0700 Support OpenMP offload with the NVIDIA compiler (#1980) These changes enable the ElectromagneticPIC OpenMP offload application to succeed (Particles/ElectromagneticPIC/Exec/OpenMP). It requires NVIDIA HPC SDK 21.3 or higher. Earlier versions of NVIDIA HPC SDK fail for one of two reasons: no interoperability between CUDA and OpenMP offload, or failure to support Fortran automatic arrays in OpenMP offload regions. I tested on Cori-GPU with nvhpc/21.3 and cuda/11.1.1. Co-authored-by: Christopher Daley Tools/GNUMake/Make.defs Tools/GNUMake/comps/nvhpc.mak Tools/libamrex/mkconfig.py commit a61b40464f3f16eff420d19d99cff0fa010eb2b6 Author: Weiqun Zhang Date: Tue Apr 27 11:32:55 2021 -0700 Use full path to AMReX_Config.H in gnu make target (#1979) Tools/GNUMake/Make.rules commit 30879b025d222b383aa22540598dc1891ce73478 Author: Weiqun Zhang Date: Mon Apr 26 16:43:20 2021 -0700 Fix GNU make for nvcc 11.0 (#1977) This fixes a bug introduced in #1967. It turns out nvcc 11.0 does not support -MP. Tools/GNUMake/comps/nvcc.mak commit 0340e766b71005c9675d6e9b759c66bb5c705297 Author: Weiqun Zhang Date: Mon Apr 26 13:20:16 2021 -0700 Add CI for 2D Single Precision EB HIP build (#1974) Also fix some warnings. .github/workflows/linux.yml Src/Base/AMReX_GpuLaunchGlobal.H Src/Base/AMReX_Reduce.H Src/EB/AMReX_EB2_2D_C.cpp commit 521ff492834377cc5ee91d6eb72db2471ccde3b7 Author: Weiqun Zhang Date: Mon Apr 26 13:00:09 2021 -0700 Fix dependency of f90.depends (#1975) Tools/GNUMake/Make.rules commit b1709947c0a354dc5dbefeb3421497478ef2333c Author: Weiqun Zhang Date: Mon Apr 26 09:07:35 2021 -0700 GNU Make: -MMD -MP (#1967) Use -MMD -MP instead of -M to generate dependency unless it is not supported by the compiler. In the new approach, the dependencies are generated during compilation without a separate step. Tools/GNUMake/Make.defs Tools/GNUMake/Make.rules Tools/GNUMake/comps/cray.mak Tools/GNUMake/comps/hip.mak Tools/GNUMake/comps/llvm.mak Tools/GNUMake/comps/nvcc.mak Tools/GNUMake/comps/pgi.mak commit e6833a86ebf35861017cfd5f16ca66ac9dc8b7ae Author: Luca Fedeli Date: Mon Apr 26 17:13:19 2021 +0200 Fix missing Real() in AMReX_EB2_2D_C.cpp (#1972) This PR fixes few missing `Real()` in `AMReX_EB2_2D_C.cpp`, which currently prevent the compilation of a function in single precision. Src/EB/AMReX_EB2_2D_C.cpp commit 32597f78f68fc8b948ceb10a22c0d01ab2b513ec Author: Weiqun Zhang Date: Mon Apr 26 08:11:58 2021 -0700 Undeprecate ReduceData::value() (#1970) ReduceData::value() with no argument was recently marked as a depracated function because it involves calling a std::function. However, the call is never made inside a loop like MFIter. So there is no practical difference in performance between ReduceData::vaule and ReduceData::value(ReduceOps&). It is hereby undeprecated to avoid warning messages for a lot of applications. Src/Base/AMReX_Reduce.H commit 5eedbae2d805d86014788427bee39e6cccb2155c Author: Shreyas Ananthan Date: Sun Apr 25 13:12:48 2021 -0600 Workaround compile error on Apple M1 (ARM64) (#1969) Fixes #1968 Src/Base/AMReX.cpp commit 41d1ea1873494d5e59c6f7da16d73b634e2a9362 Author: asalmgren Date: Fri Apr 23 10:54:14 2021 -0700 Changes to EB slopes (#1965) * 1) Change test on flag.isRegular() to (vfrac == 1) -- this only matters when we have cells with vfrac == 1 but with an EB face -- this change will allow us to use the non-EB stencil when parallel to an EB face if all the vfrac's we need are = 1, and is also consistent with how we treat slopes tangential to domain boundaries 2) pass max_order in so we can default to 4th order if max_order == 4 and there are enough cells with vfrac == 1 3) Modify the Barth-Jespersen limiting so that, for example, it doesn't use cells at i-1 when computing a slope on the i+1/2 face. NOTE: this contains a change to the interfaces to the slope routines so all code that calls these will need to be modified to pass in vfrac and max_order * Remove trailing white spaces * remove more trailing white spaces * the last of the trailing white spaces ... * Compile in 2d Src/EB/AMReX_EB_slopes_K.H commit c0570ec8d60f1a6227b5835f9105801a40a81eab Author: Axel Huebl Date: Fri Apr 23 10:24:16 2021 -0700 Docs: CMake Multiple Archs (#1964) Answer a user question with a new doc entry. Docs/sphinx_documentation/source/GPU.rst commit e85b665bcb032f4064fb9845a925c6d3d8c8aa03 Author: Weiqun Zhang Date: Fri Apr 23 09:06:37 2021 -0700 Split Large Files (#1966) Split some large files in MLMG into smaller files to improve compile time. Src/LinearSolvers/CMakeLists.txt Src/LinearSolvers/MLMG/AMReX_MLEBABecLap.cpp Src/LinearSolvers/MLMG/AMReX_MLEBABecLap_F.cpp Src/LinearSolvers/MLMG/AMReX_MLEBTensorOp.cpp Src/LinearSolvers/MLMG/AMReX_MLEBTensorOp_bc.cpp Src/LinearSolvers/MLMG/AMReX_MLNodeLaplacian.cpp Src/LinearSolvers/MLMG/AMReX_MLNodeLaplacian_eb.cpp Src/LinearSolvers/MLMG/AMReX_MLNodeLaplacian_hypre.cpp Src/LinearSolvers/MLMG/AMReX_MLNodeLaplacian_misc.cpp Src/LinearSolvers/MLMG/AMReX_MLNodeLaplacian_sten.cpp Src/LinearSolvers/MLMG/AMReX_MLNodeLaplacian_sync.cpp Src/LinearSolvers/MLMG/AMReX_MLTensorOp.cpp Src/LinearSolvers/MLMG/AMReX_MLTensorOp_grad.cpp Src/LinearSolvers/MLMG/Make.package commit 84cc7114097b548dce3b704b6eb9681b425f1e94 Author: Weiqun Zhang Date: Thu Apr 22 17:56:10 2021 -0700 Build Time Log (#1963) In GNU Make, if one uses `LOG_BUILD_TIME=TRUE`, a log of build time will be available in the end. Tools/C_scripts/gatherbuildtime.py Tools/GNUMake/Make.defs Tools/GNUMake/Make.rules commit 6e5a03f8f8346e0f4840d1b5670f9afa60a0fc4a Author: Weiqun Zhang Date: Thu Apr 22 10:19:42 2021 -0700 Using rocPRIM in HIP Version of Scan (#1962) Reimplement HIP version of scan using rocprim. This uses a few things in rocprim::detail. Hopefully it will not break in a future release of HIP. Also some minor tweaks to the CUDA version of Scan. Src/Base/AMReX_GpuLaunchGlobal.H Src/Base/AMReX_Scan.H commit a0790de872d6b18b563712b9f1b18323b8d00072 Author: Weiqun Zhang Date: Tue Apr 20 16:21:37 2021 -0700 Reimplementation of ReduceOps (#1955) The new approach does reduction in two passes. Unlike the old approach, this does not require atomics, and has much better performance for types that do not have hardware support for atomics. Src/AmrCore/AMReX_TagBox.cpp Src/Base/AMReX_BaseFab.H Src/Base/AMReX_FArrayBox.H Src/Base/AMReX_FabArrayUtility.H Src/Base/AMReX_GpuDevice.H Src/Base/AMReX_GpuDevice.cpp Src/Base/AMReX_GpuReduce.H Src/Base/AMReX_Reduce.H Src/Base/AMReX_Tuple.H Src/Base/AMReX_iMultiFab.cpp Src/EB/AMReX_EB2_GeometryShop.H Src/EB/AMReX_EB2_Level.cpp Src/EB/AMReX_EBCellFlag.cpp Src/LinearSolvers/MLMG/AMReX_MLCellABecLap.cpp Src/Particle/AMReX_ParticleContainerI.H Src/Particle/AMReX_ParticleInit.H Src/Particle/AMReX_ParticleLocator.H Src/Particle/AMReX_ParticleReduce.H Src/Particle/AMReX_ParticleUtil.H Src/Particle/AMReX_WriteBinaryParticleData.H Tests/Particles/Intersection/main.cpp commit 1796dd911bae1ae0b58ae92e4e023bdb00a0f44a Author: mic84 Date: Mon Apr 19 17:01:36 2021 -0700 CMake: implement logic to support compiler ID IntelLLVM (new in CMake 3.20) (#1959) CMake 3.20+ has its own compiler identification for the new oneAPI LLVM-ICX/DPCXX implementation. Tools/CMake/AMReXFlagsTargets.cmake Tools/CMake/AMReXGenerateConfigHeader.cmake Tools/CMake/AMReXOptions.cmake Tools/CMake/AMReXSYCL.cmake commit 0a50fded0df7539b5e8e32b2a88966d8d671e264 Author: Weiqun Zhang Date: Mon Apr 19 14:19:53 2021 -0700 DPCPP: fix atomicAdd for float (#1958) We recently removed the template specialization for atomicAdd of float in the DPC++ backend. That was a mistake because SYCL does not support atomicAdd for float, and we have to use atomicCAS to implement it. Src/Base/AMReX_GpuAtomic.H Tests/Amr/Advection_AmrCore/Source/Src_K/slope_K.H commit c409f9e1ffc870fd4926190ffceffc7d0d5ad409 Author: Weiqun Zhang Date: Mon Apr 19 13:30:05 2021 -0700 CPU version of TagBox::buffer (#1957) Add a CPU version of TagBox::buffer. @MSABuschman reported in #1951 that TagBox::buffer has been very slow since commit #1258 if the error buffer size is large. The function was rewritten in #1258 to do the work on GPU. In this PR, the old version is reintroduced for CPU. Note that the current implementation is expected to have poor performance on GPU if it has a very large error buffer. It's still not clear how we should implement this function for GPU if a large error buffer is used. .github/workflows/linux.yml Src/AmrCore/AMReX_TagBox.H Src/AmrCore/AMReX_TagBox.cpp commit 4320729fe46992851b3f222d28833109cff0234e Author: jmsexton03 Date: Sun Apr 18 11:21:27 2021 -0700 Fix out of bound error in AsyncWrite (#1753) Fix out of bound error when the local number of fabs is zero. Co-authored-by: Weiqun Zhang Src/Base/AMReX_VisMF.cpp commit b30cae951ba2c7c004f06e3d7d382c7e79793887 Author: Weiqun Zhang Date: Sun Apr 18 08:15:16 2021 -0700 Fab data I/O (#1956) When Fab data are in GPU device memory (including managed memory), use pinned memory as buffer to avoid accessing the data on the host. Previously, MultiFab, plotfile and checkpoint writers except for the async versions relied on that the memory was managed and therefore could be used directly in I/O. Src/Base/AMReX_FArrayBox.cpp Src/Base/AMReX_VisMF.cpp commit f21f15d8f7a3098d9fa13ac6b316ca518f464ac5 Author: Max Katz Date: Fri Apr 16 17:54:37 2021 -0700 Allow user to opt out of renaming MTMPI suffix (#1954) If MPI_THREAD_MULTIPLE is going to be on by default in an application, might as well not need to rename the suffix, so this gives the application the ability to just keep the suffix MPI. Tools/GNUMake/Make.defs commit 3812eadb83a667a615630a2ff8d92def964c3e58 Author: drangara <69211175+drangara@users.noreply.github.com> Date: Fri Apr 16 19:57:47 2021 -0400 Add triangle wave function to least squares test (#1947) * Add a triangle wave test for Least Squares centroids, fix some bugs in test setup * update doc * fix formatting * fix formatting Tests/LinearSolvers/LeastSquares/Make.package Tests/LinearSolvers/LeastSquares/MyTest.H Tests/LinearSolvers/LeastSquares/MyTest.cpp Tests/LinearSolvers/LeastSquares/README.md Tests/LinearSolvers/LeastSquares/initData.cpp Tests/LinearSolvers/LeastSquares/initTriangleWaveData.cpp Tests/LinearSolvers/LeastSquares/initTriangleWaveDataFor2D.cpp Tests/LinearSolvers/LeastSquares/inputs.2d.trianglewave commit eaad7627ff925f8ae9b290a7d6a2d5f2fba54239 Author: asalmgren Date: Fri Apr 16 15:52:15 2021 -0700 level_0_gome --> level_0_geom (#1953) Src/AmrCore/AMReX_AmrCore.cpp commit 029a8b0608beab17f440718b5b147dc5dd24b67c Author: Weiqun Zhang Date: Fri Apr 16 13:51:19 2021 -0700 Port FluxRegister::ClearInternalBorders to GPU (#1952) Src/AmrCore/AMReX_FluxRegister.cpp commit 42da8fb5ac8df41566a045bb3332df6ad24126d8 Author: Weiqun Zhang Date: Thu Apr 15 19:19:25 2021 -0700 Fix the return of MLPoisson::makeNLinOp (#1950) Src/LinearSolvers/MLMG/AMReX_MLPoisson.cpp commit 66f99da25c6a305fae47f0b952ebe670af867b19 Author: Andrew Myers Date: Thu Apr 15 18:07:28 2021 -0700 Skip temporary in ParticleToMesh when it is not needed. (#1949) Src/AmrCore/AMReX_AmrParticles.H Src/Particle/AMReX_ParticleMesh.H commit d2e37b06bed136e001ae62718ba5754720a01f0d Author: Andrew Myers Date: Thu Apr 15 11:32:42 2021 -0700 Multilevel version of ParticleToMesh (#1945) This uses the same algorithm as the AssignDensity method in `AmrCore/AMReX_AmrParticles.H` to handle the coarse / fine boundaries. However, it allows a user-specified lambda function to define the interpolation operation, as in the single-level `ParticleToMesh` routine. Src/AmrCore/AMReX_AmrParticles.H Src/Particle/AMReX_ParticleMesh.H Src/Particle/AMReX_ParticleUtil.H Tests/Particles/ParticleMesh/main.cpp Tests/Particles/ParticleMeshMultiLevel/CMakeLists.txt Tests/Particles/ParticleMeshMultiLevel/GNUmakefile Tests/Particles/ParticleMeshMultiLevel/Make.package Tests/Particles/ParticleMeshMultiLevel/inputs Tests/Particles/ParticleMeshMultiLevel/main.cpp commit b1d05dec0da3b6d0074af8850163837c53488ebf Author: Erik Schnetter Date: Thu Apr 15 13:32:54 2021 -0400 Correct typo in comment (#1946) Tools/CMake/AMReXOptions.cmake commit 914459f443f5ed54b5e33551802d10521042af9b Author: Don E. Willcox Date: Wed Apr 14 16:13:44 2021 -0700 fix greater->less in docs for abort_on_out_of_gpu_memory (#1944) Docs/sphinx_documentation/source/GPU.rst commit 87ee8f03c3b38c645312331fa75c92b6a932a87b Author: Weiqun Zhang Date: Tue Apr 13 16:45:37 2021 -0700 Use CUB for block reduction (#1943) Also removed some reduction functions that have been deprecated long ago. Src/Base/AMReX_GpuFuse.cpp Src/Base/AMReX_GpuLaunchFunctsG.H Src/Base/AMReX_GpuReduce.H Src/Base/AMReX_Reduce.H commit ba378870a10ad475c647cad3b8650151fc896580 Author: Weiqun Zhang Date: Tue Apr 13 08:24:57 2021 -0700 Reimplementation of PrefixSum with cub (#1942) For CUDA >= 11, use cub to reimplement PrefixSum that can take two lambda functions. Both cub and thrust only provide scan functions that take iterators. Src/Base/AMReX_Reduce.H Src/Base/AMReX_Scan.H commit 58c3ba169a2b8fc311395a959724ea63dc37aa3c Author: Weiqun Zhang Date: Sun Apr 11 17:03:22 2021 -0700 Avoid AMREX_NO_UNIQUE_ADDRESS (#1941) * Avoid AMREX_NO_UNIQUE_ADDRESS because it causes gcc 9.3 to crash. * Fix AMREX_NO_UNIQUE_ADDRESS and AMREX_NODISCARD macros. * Adjust compiler flag in CI to allow for C++17 extension when compiling with C++14. .github/workflows/linux.yml Src/Base/AMReX_Extension.H Src/Base/AMReX_NonLocalBC.H commit 18c5e96e04c982ef886d8aa9f6e65ebee1564f35 Author: Weiqun Zhang Date: Thu Apr 8 12:58:09 2021 -0700 Rename variable to fix a shadow warning (#1939) Src/Particle/AMReX_ParticleUtil.H commit 51ba2f7ebbc2dc759845fdab073d353ab175507f Author: Weiqun Zhang Date: Thu Apr 8 11:55:51 2021 -0700 Workaround a compiler bug(?) reported in #1937 (#1938) Src/Base/AMReX_Arena.cpp commit 7c259bab6cc583a92a8d7f6b640af6618e6206f3 Author: Weiqun Zhang Date: Wed Apr 7 17:52:04 2021 -0700 Cell-Centered Overset Solver Coarsening (#1915) Currently for the cell-centered overset multigrid solver, coarsening stops when a coarsened cell would contain both regular and masked-out cells. Here, we further coarsen the grids by ignoring the overset mask and giving the masked-out cells a huge value of alpha so that their values are essentially frozen. Src/LinearSolvers/MLMG/AMReX_MLABecLaplacian.H Src/LinearSolvers/MLMG/AMReX_MLABecLaplacian.cpp Src/LinearSolvers/MLMG/AMReX_MLCellABecLap.H Src/LinearSolvers/MLMG/AMReX_MLCellABecLap.cpp Src/LinearSolvers/MLMG/AMReX_MLLinOp.H Src/LinearSolvers/MLMG/AMReX_MLMG.cpp Src/LinearSolvers/MLMG/AMReX_MLPoisson.H Src/LinearSolvers/MLMG/AMReX_MLPoisson.cpp commit db9303290d4912f15230c792f4e1c91bce0fb0ce Author: Weiqun Zhang Date: Wed Apr 7 15:38:40 2021 -0700 Hypre Nodal Setup (#1906) * Hypre Nodal Setup * Allow CoarseningStrategy::Sigma in Hypre nodal solver. * Port Hypre nodal setup to GPU. * For EB nodal solver, label a node ad Dirichlet if its diagonal component of the stencil is zero. * remove EB stuff in hypre nodal setup because it's no longer needed due to the change in dirichlet mask Src/Base/AMReX_FBI.H Src/Base/AMReX_GpuAtomic.H Src/Base/AMReX_GpuDevice.cpp Src/Base/AMReX_GpuLaunchFunctsC.H Src/Base/AMReX_Scan.H Src/EB/AMReX_EB2_Level.cpp Src/Extern/HYPRE/AMReX_HypreNodeLap.H Src/Extern/HYPRE/AMReX_HypreNodeLap.cpp Src/LinearSolvers/MLMG/AMReX_MLNodeLap_2D_K.H Src/LinearSolvers/MLMG/AMReX_MLNodeLap_3D_K.H Src/LinearSolvers/MLMG/AMReX_MLNodeLap_K.H Src/LinearSolvers/MLMG/AMReX_MLNodeLaplacian.H Src/LinearSolvers/MLMG/AMReX_MLNodeLaplacian.cpp Src/LinearSolvers/MLMG/AMReX_MLNodeLinOp.H Src/LinearSolvers/MLMG/AMReX_MLNodeTensorLap_2D_K.H Src/LinearSolvers/MLMG/AMReX_MLNodeTensorLap_3D_K.H Src/LinearSolvers/MLMG/AMReX_MLNodeTensorLaplacian.H Src/LinearSolvers/MLMG/AMReX_MLNodeTensorLaplacian.cpp commit 8f9589682e56d7f443e42de032b027bd7b1944fa Author: mic84 Date: Wed Apr 7 14:15:50 2021 -0700 CMake: add support for BOUND_CHECK (#1934) Docs/sphinx_documentation/source/BuildingAMReX.rst Tools/CMake/AMReXOptions.cmake Tools/CMake/AMReXSetDefines.cmake Tools/CMake/AMReX_Config.H.in commit d5e4a3882b263c858221de1130f73e04825a2e1e Author: Luca Fedeli Date: Wed Apr 7 16:54:28 2021 +0200 Fix other missing Real() (#1933) This PR fixes few missing `Real()` in `AMReX_EB2_3D_C.cpp`, which currently prevent the compilation of some methods in single precision. Src/EB/AMReX_EB2_3D_C.cpp commit 3f3a9174286aa06d7826367b3d07b29c339d428a Author: Luca Fedeli Date: Wed Apr 7 04:13:13 2021 +0200 fix missing Real (#1932) Src/EB/AMReX_EB2_3D_C.H commit 4123e87ce3779855a638663008de8dcec5ccb99f Author: Andrew Myers Date: Tue Apr 6 12:08:23 2021 -0400 Alias HostVector to PinnedVector and update documentation. (#1931) Src/Base/AMReX_GpuContainers.H commit 4cb1c5b79906fe566d1b80777315cf4817924b1a Author: Weiqun Zhang Date: Mon Apr 5 16:56:23 2021 -0700 Make CNS work on CPU too (#1930) Src/Base/AMReX_BaseFab.H Src/Base/AMReX_FabConv.cpp Src/Base/AMReX_IArrayBox.cpp Tests/GPU/CNS/Exec/RT/cns_prob.cpp Tests/GPU/CNS/Exec/Sod/cns_prob.cpp Tests/GPU/CNS/Source/CNS.cpp commit 7c37a75e23eabc317c7a969e21153183db616fd1 Author: Axel Huebl Date: Mon Apr 5 16:15:02 2021 -0700 std::isfinite/inf for SYCL (#1929) Wrappity wrap the non-constexpr functions: https://github.com/intel/llvm/blob/sycl/sycl/doc/extensions/C-CXX-StandardLibrary/C-CXX-StandardLibrary.rst Note that `std::isnan` works by accident/virtue of being constexpr. Src/Base/AMReX_Math.H commit 1ee06e64345a52259315181ac13a821a4ac64f09 Author: Weiqun Zhang Date: Mon Apr 5 14:10:12 2021 -0700 Update RNG for latest oneAPI (#1928) MKL now has device support for uniform distribution of unsigned int and Poisson distribution. Src/Base/AMReX_Random.H commit deddea69d9f7e2b5cac2727cadfe77e0806c194b Author: Weiqun Zhang Date: Mon Apr 5 10:13:30 2021 -0700 Use rocprim for scan (#1925) .github/workflows/dependencies/dependencies_hip.sh .github/workflows/linux.yml Src/Base/AMReX_Scan.H commit 06cec5dd8ede26ea3c0646368bdbdb32f35842ae Author: Weiqun Zhang Date: Mon Apr 5 09:31:25 2021 -0700 Use pinned memory for data transfer between host and device (#1923) The amount of data to be transferred in TagBox and STLtools is likely to be big. So pinned memory is used instead of pageable. AsyncArray is sometimes used to transfer a large amount of data. So pinned memory is used. Src/AmrCore/AMReX_AmrMesh.cpp Src/AmrCore/AMReX_TagBox.H Src/AmrCore/AMReX_TagBox.cpp Src/Base/AMReX_GpuAsyncArray.H Src/Base/AMReX_GpuAsyncArray.cpp Src/EB/AMReX_EB_STL_utils.H commit d3a24fe0519d8eb7cfd7043be2907e08ee534352 Author: Weiqun Zhang Date: Mon Apr 5 09:16:28 2021 -0700 Use cub::DeviceScan for CUDA >= 11 (#1924) Src/Base/AMReX_Scan.H Tools/GNUMake/Make.defs Tools/GNUMake/comps/hip.mak Tools/GNUMake/comps/nvcc.mak commit ddfbf3de3bdf5ced8f42d0828ed52d95fcea22ec Author: Weiqun Zhang Date: Mon Apr 5 09:16:07 2021 -0700 Fix compilation broken in #1920 (#1926) Tests/LinearSolvers/CellEB/MyTest.cpp Tests/LinearSolvers/CellEB2/MyTest.cpp Tests/LinearSolvers/EBConvergenceTest/MyTest.cpp Tests/LinearSolvers/EBflux_grad/MyTest.cpp Tests/LinearSolvers/LeastSquares/initData.cpp Tests/LinearSolvers/NodeEB/MyTest.cpp commit 5ec3d3799ff238208ae41949ce506d3491b37208 Author: Weiqun Zhang Date: Fri Apr 2 19:03:42 2021 -0700 Move Tests/CNS to Tests/EB/CNS (#1922) Tests/EB/CNS/CMakeLists.txt Tests/EB/CNS/Exec/Combustor/GNUmakefile Tests/EB/CNS/Exec/Combustor/Make.package Tests/EB/CNS/Exec/Combustor/bc_fill_nd.F90 Tests/EB/CNS/Exec/Combustor/bc_fill_nd.F90_jbb Tests/EB/CNS/Exec/Combustor/cns_prob.F90 Tests/EB/CNS/Exec/Combustor/cns_prob.F90_jbb Tests/EB/CNS/Exec/Combustor/inputs Tests/EB/CNS/Exec/Combustor/inputs.regt Tests/EB/CNS/Exec/Make.CNS Tests/EB/CNS/Exec/Pulse/GNUmakefile Tests/EB/CNS/Exec/Pulse/Make.package Tests/EB/CNS/Exec/Pulse/cns_prob.F90 Tests/EB/CNS/Exec/Pulse/inputs Tests/EB/CNS/Exec/Pulse/inputs.regt Tests/EB/CNS/Exec/ShockRef/GNUmakefile Tests/EB/CNS/Exec/ShockRef/Make.package Tests/EB/CNS/Exec/ShockRef/cns_prob.F90 Tests/EB/CNS/Exec/ShockRef/inputs Tests/EB/CNS/Exec/ShockRef/inputs.amr Tests/EB/CNS/Exec/ShockRef/inputs.regt Tests/EB/CNS/Exec/Sod/GNUmakefile Tests/EB/CNS/Exec/Sod/Make.package Tests/EB/CNS/Exec/Sod/cns_prob.F90 Tests/EB/CNS/Exec/Sod/inputs Tests/EB/CNS/Source/CNS.H Tests/EB/CNS/Source/CNS.cpp Tests/EB/CNS/Source/CNSBld.cpp Tests/EB/CNS/Source/CNS_F.H Tests/EB/CNS/Source/CNS_advance.cpp Tests/EB/CNS/Source/CNS_init_eb2.cpp Tests/EB/CNS/Source/CNS_io.cpp Tests/EB/CNS/Source/CNS_setup.cpp Tests/EB/CNS/Source/Make.package Tests/EB/CNS/Source/diffusion/Make.package Tests/EB/CNS/Source/diffusion/cns_diff_mod.F90 Tests/EB/CNS/Source/diffusion/cns_eb_diff_mod.F90 Tests/EB/CNS/Source/diffusion/cns_eb_diff_wall.F90 Tests/EB/CNS/Source/diffusion/diff_coef_mod.F90 Tests/EB/CNS/Source/fortran/CNS_derive.F90 Tests/EB/CNS/Source/fortran/CNS_divop.F90 Tests/EB/CNS/Source/fortran/CNS_dudt.F90 Tests/EB/CNS/Source/fortran/CNS_f.F90 Tests/EB/CNS/Source/fortran/CNS_nd.F90 Tests/EB/CNS/Source/fortran/CNS_physics.F90 Tests/EB/CNS/Source/fortran/CNS_tagging.F90 Tests/EB/CNS/Source/fortran/Make.package Tests/EB/CNS/Source/fortran/bc_fill_nd.F90 Tests/EB/CNS/Source/hydro/Hyp_gamma_MOL.F90 Tests/EB/CNS/Source/hydro/Hyp_gamma_MOL_EB.F90 Tests/EB/CNS/Source/hydro/Make.package Tests/EB/CNS/Source/hydro/analriem3d.F90 Tests/EB/CNS/Source/hydro/cns_eb_hyp_wall.F90 Tests/EB/CNS/Source/hydro/slope_mol_3d_gamma.F90 Tests/EB/CNS/Source/hydro/slope_mol_3d_gamma_EB.F90 Tests/EB/CNS/Source/main.cpp commit 42063585bb299c5dd220aa5b35be9e3f05594da5 Author: Weiqun Zhang Date: Fri Apr 2 18:07:50 2021 -0700 No need for AMREX_IS_TRIVIALLY_COPYABLE && AMREX_IS_TRIVIALLY_DEFAULT_CONSTRUCTIBLE (#1921) They are not needed because gcc 4 is no longer supported. Src/Base/AMReX_BaseFab.H Src/Base/AMReX_CudaGraph.H Src/Base/AMReX_GpuAsyncArray.H Src/Base/AMReX_GpuBuffer.H Src/Base/AMReX_GpuContainers.H Src/Base/AMReX_GpuFuse.H Src/Base/AMReX_GpuMemory.H Src/Base/AMReX_PODVector.H Src/Base/AMReX_ParallelDescriptor.cpp Src/Base/AMReX_Reduce.H Src/Base/AMReX_TypeTraits.H commit 77fb0251e3ad99af069d5d0bcaa80128cc50062b Author: Weiqun Zhang Date: Fri Apr 2 10:30:42 2021 -0700 Use std::make_unique instead of reset since we are using C++14 (#1920) Docs/sphinx_documentation/source/IO.rst Docs/sphinx_documentation/source/LinearSolvers.rst Src/Amr/AMReX_Amr.cpp Src/Amr/AMReX_AmrLevel.cpp Src/Amr/AMReX_StateData.cpp Src/Amr/AMReX_StateDescriptor.cpp Src/AmrCore/AMReX_AmrCore.cpp Src/AmrCore/AMReX_AmrParticles.H Src/AmrCore/AMReX_ErrorList.cpp Src/AmrCore/AMReX_FillPatchUtil_I.H Src/Base/AMReX_AsyncOut.cpp Src/Base/AMReX_BackgroundThread.cpp Src/Base/AMReX_BoxArray.cpp Src/Base/AMReX_CuptiTrace.cpp Src/Base/AMReX_DistributionMapping.cpp Src/Base/AMReX_FabArrayBase.cpp Src/Base/AMReX_ForkJoin.cpp Src/Base/AMReX_GpuDevice.cpp Src/Base/AMReX_GpuFuse.cpp Src/Base/AMReX_MFIter.cpp Src/Base/AMReX_Machine.cpp Src/Base/AMReX_MemPool.cpp Src/Base/AMReX_MemProfiler.cpp Src/Base/AMReX_MultiFab.cpp Src/Base/AMReX_ParallelContext.cpp Src/Base/AMReX_PlotFileDataImpl.cpp Src/Base/AMReX_PlotFileUtil.cpp Src/Base/AMReX_iMultiFab.cpp Src/EB/AMReX_EB2_IndexSpaceI.H Src/EB/AMReX_EBFabFactory.cpp Src/Extern/HYPRE/AMReX_Hypre.cpp Src/Extern/HYPRE/AMReX_HypreABecLap3.cpp Src/Extern/HYPRE/AMReX_HypreNodeLap.cpp Src/Extern/PETSc/AMReX_PETSc.cpp Src/Extern/amrdata/AMReX_AmrData.cpp Src/F_Interfaces/Octree/AMReX_octree_fi.cpp Src/LinearSolvers/MLMG/AMReX_MLCellABecLap.cpp Src/LinearSolvers/MLMG/AMReX_MLCellLinOp.cpp Src/LinearSolvers/MLMG/AMReX_MLEBABecLap.cpp Src/LinearSolvers/MLMG/AMReX_MLLinOp.H Src/LinearSolvers/MLMG/AMReX_MLLinOp.cpp Src/LinearSolvers/MLMG/AMReX_MLMG.cpp Src/LinearSolvers/MLMG/AMReX_MLNodeLaplacian.cpp Src/LinearSolvers/MLMG/AMReX_MLNodeLinOp.cpp Src/LinearSolvers/MLMG/AMReX_MLPoisson.cpp Src/LinearSolvers/Projections/AMReX_MacProjector.cpp Src/LinearSolvers/Projections/AMReX_NodalProjector.cpp Src/Particle/AMReX_NeighborParticlesGPUImpl.H Src/Particle/AMReX_NeighborParticlesI.H Src/Particle/AMReX_ParticleContainerBase.cpp Src/Particle/AMReX_TracerParticles.cpp Tests/Amr/Advection_AmrCore/Source/AmrCoreAdv.cpp Tests/Amr/Advection_AmrLevel/Source/AmrLevelAdv.cpp Tests/FillBoundaryComparison/main.cpp Tests/GPU/CNS/Source/CNS.cpp Tests/GPU/CNS/Source/CNS_io.cpp Tests/HDF5Benchmark/main.cpp Tests/LinearSolvers/CellEB/MyTest.cpp Tests/LinearSolvers/CellEB2/MyTest.cpp Tests/LinearSolvers/CellOverset/MyTest.cpp Tests/LinearSolvers/EBConvergenceTest/MyTest.cpp Tests/LinearSolvers/EBflux_grad/MyTest.cpp Tests/LinearSolvers/LeastSquares/initData.cpp Tests/LinearSolvers/MultiComponent/MCNodalLinOp.cpp Tests/LinearSolvers/NodeEB/MyTest.cpp Tests/LinearSolvers/TensorOverset/MyTest.cpp Tests/Particles/AssignMultiLevelDensity/main.cpp Tests/Particles/AsyncIO/main.cpp commit 57d41ef9cc3f0db4b5d644d0169311b4f9f68a92 Author: Weiqun Zhang Date: Thu Apr 1 08:37:00 2021 -0700 Update CHANGES for 21.04 (#1919) CHANGES commit 622fbf998d44b77f44d1c682b9b33e018402051f Author: Andrew Myers Date: Thu Apr 1 11:04:12 2021 -0400 Docs typo fix. (#1918) Docs/sphinx_documentation/source/LinearSolvers_Chapter.rst commit ecf7e9812760feff4cdb2c53deb3402b68383fc8 Author: asalmgren Date: Wed Mar 31 14:02:27 2021 -0700 Allow us to set the "normalization threshold" variable at run-time (#1917) through the NodalProjector class Src/LinearSolvers/Projections/AMReX_NodalProjector.cpp commit b8559fe999715afd3056cb94371bf2399a461863 Author: Weiqun Zhang Date: Tue Mar 30 17:01:58 2021 -0700 Fix a bug in BaseFab::resize. (#1916) Previously if `BaseFab::resize`'s optional `Arena*` argument is `nullptr` (which is the default) while `BaseFab`'s `Arena*` member is a different `Arena`, the `nullptr` will be interpreted as the default `Arena`, `The_Arena`, ignoring the `BaseFab`'s `Arena` has already been set. This causes the following code to allocate data using `The_Arena`, not `The_Cpu_Arena`. BaseFab fab(The_Cpu_Arena()); fab.resize(box); This PR fixes the bug. Src/Base/AMReX_BaseFab.H commit 81982da08fb25cd266b3666d358d1cbf09bb4de5 Author: asalmgren Date: Tue Mar 30 12:47:49 2021 -0700 Change the default value of m_normalization_threshold from 1e-10 to 1e-8 (#1914) This is motivated by an EB nodal BiCG bottom solver failing to converge with 1e-10 while converging with 1e-8. Src/LinearSolvers/MLMG/AMReX_MLNodeLaplacian.H commit 92945ad3a3560031c43fe7f02b9cc252f8330708 Author: Andrew Myers Date: Tue Mar 30 12:34:53 2021 -0400 Dont zero out input multifab in amrex::ParticleToMesh (#1910) Src/Particle/AMReX_ParticleContainerI.H Src/Particle/AMReX_ParticleMesh.H commit 78eae52af97d4c6b993feec9bf76f21b9ef7288a Author: asalmgren Date: Mon Mar 29 17:00:17 2021 -0700 Revert the change which zeroed out the center coefficient if it has the wrong (#1911) sign. This breaks more cases than it fixes. Src/LinearSolvers/MLMG/AMReX_MLNodeLap_2D_K.H Src/LinearSolvers/MLMG/AMReX_MLNodeLap_3D_K.H commit 227e378901d904040340ad38971db5142f008d15 Author: vzendejasl <59891999+vzendejasl@users.noreply.github.com> Date: Mon Mar 29 13:56:25 2021 -0700 Slopes test (#1859) Slope computation on cell centroids using the Least Square method. These tests are based from the LeastSquares Test (#1707) but instead of calculating scalar values on face centroids they are calculated on cell centroids using a linear function rather than a second order polynomial (#1707). Tests/Slopes/GNUmakefile Tests/Slopes/Make.package Tests/Slopes/MyEB.H Tests/Slopes/MyTest.H Tests/Slopes/MyTest.cpp Tests/Slopes/README.md Tests/Slopes/initData.cpp Tests/Slopes/initEB.cpp Tests/Slopes/initLinearData.cpp Tests/Slopes/initLinearDataFor2D.cpp Tests/Slopes/initLinearDataFor3D.cpp Tests/Slopes/inputs Tests/Slopes/inputs.2d.askew Tests/Slopes/inputs.2d.base Tests/Slopes/inputs.2d.fullyrotated Tests/Slopes/inputs.3d.linear.aligned.xy-x Tests/Slopes/inputs.3d.linear.aligned.xy-y Tests/Slopes/inputs.3d.linear.aligned.xz-x Tests/Slopes/inputs.3d.linear.aligned.xz-z Tests/Slopes/inputs.3d.linear.aligned.yz-y Tests/Slopes/inputs.3d.linear.aligned.yz-z Tests/Slopes/inputs.3d.linear.askew-all Tests/Slopes/inputs.3d.linear.askew-xy Tests/Slopes/inputs.3d.linear.askew-xz Tests/Slopes/inputs.3d.linear.askew-yz Tests/Slopes/main.cpp commit 0ab63f3f93950b5a90fb091c6eabbf3239886fcd Author: Weiqun Zhang Date: Mon Mar 29 09:32:23 2021 -0700 Fix bug in nodal solver that uses harmonic averageing (#1902) This is a minor bug that does not affect the final result. The convergence rate might be slightly affected. Src/LinearSolvers/MLMG/AMReX_MLNodeLap_2D_K.H commit dc8f72702c1ed4bb4443916ded01d9daff4f811f Author: asalmgren Date: Fri Mar 26 12:40:02 2021 -0700 Ensure center coefficient in an EB nodal solve has correct sign (#1905) It is possible for the center coefficient in an EB nodal solve to have the incorrect sign after coarsening -- here we protect against that by setting s0 to the max of (s0,0) -- this enables the BiCG to converge. Src/LinearSolvers/MLMG/AMReX_MLNodeLap_2D_K.H Src/LinearSolvers/MLMG/AMReX_MLNodeLap_3D_K.H commit 9f297b9bc3a233553925feb5c0e2db784f1ddf9e Author: mic84 Date: Fri Mar 26 12:00:44 2021 -0700 CMake: add test_install (#1894) * CMake: add test_install * CMake: forgot to add test dir * CMake: specify compiler to use when running test_install * Update Doc .github/workflows/linux.yml CMakeLists.txt Docs/sphinx_documentation/source/BuildingAMReX.rst Tests/CMakeTestInstall/CMakeLists.txt Tools/CMake/AMReXInstallHelpers.cmake commit 146fc573967dbb2328be15d8e499a479f218c4e8 Author: jmsexton03 Date: Thu Mar 25 16:16:33 2021 -0700 Update Make.CNS and similar files to reflect Tutorials to Tests move (#1903) Tests/Amr/Advection_AmrCore/README Tests/Amr/Advection_AmrCore/README.md Tests/Amr/Advection_AmrLevel/Exec/Make.Adv Tests/CNS/Exec/Make.CNS Tests/FortranInterface/Advection_F/Exec/Make.Adv Tests/FortranInterface/Advection_octree_F/Exec/Make.Adv Tests/GPU/CNS/Exec/Make.CNS Tests/LinearSolvers/MultiComponent/MCNodalLinOp.H Tests/LinearSolvers/MultiComponent/MCNodalLinOp.cpp Tests/LinearSolvers/MultiComponent/main.cpp commit f9216a341e51404e935733a4c953525a6f503d47 Author: Andrew Myers Date: Thu Mar 25 14:42:37 2021 -0400 Fix AMREX_EXPORT (#1901) Src/EB/AMReX_EB2.H commit e57c1e8294c7c74a5e3a71821639a4cc1e48d895 Author: Andrew Myers Date: Thu Mar 25 13:52:56 2021 -0400 reimplement increment and incrementwithtotal to take advantage of GPUs and OpenMP (#1899) Src/Particle/AMReX_ParticleContainerI.H Src/Particle/AMReX_Particles.H commit 62b3d7e4cafbc5b2f6453782bd9f4500014c4b63 Author: Andrew Myers Date: Thu Mar 25 13:46:49 2021 -0400 Export more global variables for windows support (#1900) Src/EB/AMReX_EB2.cpp commit 87c81d0ca70e65c9209386c03b3c85ab204667bb Author: Axel Huebl Date: Wed Mar 24 16:48:05 2021 -0700 CI: HIPCC as C Compiler (#1897) Since `hipcc` is a C++ compiler, we should assign it with `-x c` if used as C compiler. Or... we just use the corresponding `clang` for C files. .github/workflows/linux.yml commit f2443e0bc4aa433baa2154cf9b3b7ca4a672bf65 Author: Andrew Myers Date: Wed Mar 24 18:09:52 2021 -0400 Return const& in rdata and idata and disallow returning copies for rvalues. (#1895) Src/Particle/AMReX_Particle.H commit d226b807024fedd11eec4d751367524f75c01178 Author: Axel Huebl Date: Wed Mar 24 14:36:25 2021 -0700 CI: HIP w/o MPI (#1896) HIP 4.1 introduces issues with C targets which break the MPI feature test in CMake's `FindMPI.cmake` for C. .github/workflows/linux.yml commit 7c3e95210a1877c67139ea57c62fbcc717acc1c3 Author: Andrew Myers Date: Mon Mar 22 18:13:14 2021 -0400 Remove unnecessary abort. (#1892) Src/EB/AMReX_distFcnElement.cpp commit f3ddeadbb53049034f817d8bb9a8c21951ff494a Author: Weiqun Zhang Date: Mon Mar 22 14:53:44 2021 -0700 Update ALaplacian for multi-component and reuse (#1891) Src/LinearSolvers/MLMG/AMReX_MLABecLap_1D_K.H Src/LinearSolvers/MLMG/AMReX_MLABecLaplacian.cpp Src/LinearSolvers/MLMG/AMReX_MLALap_1D_K.H Src/LinearSolvers/MLMG/AMReX_MLALap_2D_K.H Src/LinearSolvers/MLMG/AMReX_MLALap_3D_K.H Src/LinearSolvers/MLMG/AMReX_MLALaplacian.H Src/LinearSolvers/MLMG/AMReX_MLALaplacian.cpp commit d1468003318845a83fb59692bb4f06be95eb319a Author: Andrew Myers Date: Mon Mar 22 13:28:37 2021 -0400 Move tutorials (#1876) .github/workflows/linux.yml .github/workflows/macos.yml .github/workflows/windows.yml CMakeLists.txt Docs/sphinx_documentation/source/BuildingAMReX.rst Src/Base/AMReX_FabArrayBase.H Src/Base/AMReX_NFiles.H Src/Particle/AMReX_ParticleContainerI.H Src/Particle/AMReX_ParticleUtil.H Tests/Amr/Advection_AmrCore/CMakeLists.txt Tests/Amr/Advection_AmrCore/Exec/GNUmakefile Tests/Amr/Advection_AmrCore/Exec/GNUmakefile_movie Tests/Amr/Advection_AmrCore/Exec/Make.Adv Tests/Amr/Advection_AmrCore/Exec/Make.package Tests/Amr/Advection_AmrCore/Exec/Prob.H Tests/Amr/Advection_AmrCore/Exec/inputs Tests/Amr/Advection_AmrCore/Exec/inputs_for_scaling Tests/Amr/Advection_AmrCore/Exec/paraview_amr101.py Tests/Amr/Advection_AmrCore/README Tests/Amr/Advection_AmrCore/README.md Tests/Amr/Advection_AmrCore/Source/AdvancePhiAllLevels.cpp Tests/Amr/Advection_AmrCore/Source/AdvancePhiAtLevel.cpp Tests/Amr/Advection_AmrCore/Source/AmrCoreAdv.H Tests/Amr/Advection_AmrCore/Source/AmrCoreAdv.cpp Tests/Amr/Advection_AmrCore/Source/DefineVelocity.cpp Tests/Amr/Advection_AmrCore/Source/Kernels.H Tests/Amr/Advection_AmrCore/Source/Make.package Tests/Amr/Advection_AmrCore/Source/Src_K/Adv_K.H Tests/Amr/Advection_AmrCore/Source/Src_K/Make.package Tests/Amr/Advection_AmrCore/Source/Src_K/compute_flux_2D_K.H Tests/Amr/Advection_AmrCore/Source/Src_K/compute_flux_3D_K.H Tests/Amr/Advection_AmrCore/Source/Src_K/slope_K.H Tests/Amr/Advection_AmrCore/Source/Tagging.H Tests/Amr/Advection_AmrCore/Source/bc_fill.H Tests/Amr/Advection_AmrCore/Source/face_velocity.H Tests/Amr/Advection_AmrCore/Source/main.cpp Tests/Amr/Advection_AmrLevel/CMakeLists.txt Tests/Amr/Advection_AmrLevel/Exec/Make.Adv Tests/Amr/Advection_AmrLevel/Exec/SingleVortex/GNUmakefile Tests/Amr/Advection_AmrLevel/Exec/SingleVortex/Make.package Tests/Amr/Advection_AmrLevel/Exec/SingleVortex/Prob.f90 Tests/Amr/Advection_AmrLevel/Exec/SingleVortex/face_velocity_2d.f90 Tests/Amr/Advection_AmrLevel/Exec/SingleVortex/face_velocity_3d.f90 Tests/Amr/Advection_AmrLevel/Exec/SingleVortex/inputs Tests/Amr/Advection_AmrLevel/Exec/SingleVortex/inputs.tracers Tests/Amr/Advection_AmrLevel/Exec/SingleVortex/probin Tests/Amr/Advection_AmrLevel/Exec/UniformVelocity/GNUmakefile Tests/Amr/Advection_AmrLevel/Exec/UniformVelocity/Make.package Tests/Amr/Advection_AmrLevel/Exec/UniformVelocity/Prob.f90 Tests/Amr/Advection_AmrLevel/Exec/UniformVelocity/face_velocity_2d.f90 Tests/Amr/Advection_AmrLevel/Exec/UniformVelocity/face_velocity_3d.f90 Tests/Amr/Advection_AmrLevel/Exec/UniformVelocity/inputs Tests/Amr/Advection_AmrLevel/Exec/UniformVelocity/inputs.regt Tests/Amr/Advection_AmrLevel/Exec/UniformVelocity/probdata.f90 Tests/Amr/Advection_AmrLevel/Exec/UniformVelocity/probin Tests/Amr/Advection_AmrLevel/README Tests/Amr/Advection_AmrLevel/Source/Adv_F.H Tests/Amr/Advection_AmrLevel/Source/AmrLevelAdv.H Tests/Amr/Advection_AmrLevel/Source/AmrLevelAdv.cpp Tests/Amr/Advection_AmrLevel/Source/LevelBldAdv.cpp Tests/Amr/Advection_AmrLevel/Source/Make.package Tests/Amr/Advection_AmrLevel/Source/Src_2d/Adv_2d.f90 Tests/Amr/Advection_AmrLevel/Source/Src_2d/Make.package Tests/Amr/Advection_AmrLevel/Source/Src_2d/compute_flux_2d.f90 Tests/Amr/Advection_AmrLevel/Source/Src_2d/slope_2d.f90 Tests/Amr/Advection_AmrLevel/Source/Src_3d/Adv_3d.f90 Tests/Amr/Advection_AmrLevel/Source/Src_3d/Make.package Tests/Amr/Advection_AmrLevel/Source/Src_3d/compute_flux_3d.f90 Tests/Amr/Advection_AmrLevel/Source/Src_3d/slope_3d.f90 Tests/Amr/Advection_AmrLevel/Source/Src_nd/Adv_nd.f90 Tests/Amr/Advection_AmrLevel/Source/Src_nd/Make.package Tests/Amr/Advection_AmrLevel/Source/Src_nd/Tagging_nd.f90 Tests/Amr/Advection_AmrLevel/Source/Src_nd/tagging_params.f90 Tests/Amr/Advection_AmrLevel/Source/main.cpp Tests/CMakeLists.txt Tests/CNS/CMakeLists.txt Tests/CNS/Exec/Combustor/GNUmakefile Tests/CNS/Exec/Combustor/Make.package Tests/CNS/Exec/Combustor/bc_fill_nd.F90 Tests/CNS/Exec/Combustor/bc_fill_nd.F90_jbb Tests/CNS/Exec/Combustor/cns_prob.F90 Tests/CNS/Exec/Combustor/cns_prob.F90_jbb Tests/CNS/Exec/Combustor/inputs Tests/CNS/Exec/Combustor/inputs.regt Tests/CNS/Exec/Make.CNS Tests/CNS/Exec/Pulse/GNUmakefile Tests/CNS/Exec/Pulse/Make.package Tests/CNS/Exec/Pulse/cns_prob.F90 Tests/CNS/Exec/Pulse/inputs Tests/CNS/Exec/Pulse/inputs.regt Tests/CNS/Exec/ShockRef/GNUmakefile Tests/CNS/Exec/ShockRef/Make.package Tests/CNS/Exec/ShockRef/cns_prob.F90 Tests/CNS/Exec/ShockRef/inputs Tests/CNS/Exec/ShockRef/inputs.amr Tests/CNS/Exec/ShockRef/inputs.regt Tests/CNS/Exec/Sod/GNUmakefile Tests/CNS/Exec/Sod/Make.package Tests/CNS/Exec/Sod/cns_prob.F90 Tests/CNS/Exec/Sod/inputs Tests/CNS/Source/CNS.H Tests/CNS/Source/CNS.cpp Tests/CNS/Source/CNSBld.cpp Tests/CNS/Source/CNS_F.H Tests/CNS/Source/CNS_advance.cpp Tests/CNS/Source/CNS_init_eb2.cpp Tests/CNS/Source/CNS_io.cpp Tests/CNS/Source/CNS_setup.cpp Tests/CNS/Source/Make.package Tests/CNS/Source/diffusion/Make.package Tests/CNS/Source/diffusion/cns_diff_mod.F90 Tests/CNS/Source/diffusion/cns_eb_diff_mod.F90 Tests/CNS/Source/diffusion/cns_eb_diff_wall.F90 Tests/CNS/Source/diffusion/diff_coef_mod.F90 Tests/CNS/Source/fortran/CNS_derive.F90 Tests/CNS/Source/fortran/CNS_divop.F90 Tests/CNS/Source/fortran/CNS_dudt.F90 Tests/CNS/Source/fortran/CNS_f.F90 Tests/CNS/Source/fortran/CNS_nd.F90 Tests/CNS/Source/fortran/CNS_physics.F90 Tests/CNS/Source/fortran/CNS_tagging.F90 Tests/CNS/Source/fortran/Make.package Tests/CNS/Source/fortran/bc_fill_nd.F90 Tests/CNS/Source/hydro/Hyp_gamma_MOL.F90 Tests/CNS/Source/hydro/Hyp_gamma_MOL_EB.F90 Tests/CNS/Source/hydro/Make.package Tests/CNS/Source/hydro/analriem3d.F90 Tests/CNS/Source/hydro/cns_eb_hyp_wall.F90 Tests/CNS/Source/hydro/slope_mol_3d_gamma.F90 Tests/CNS/Source/hydro/slope_mol_3d_gamma_EB.F90 Tests/CNS/Source/main.cpp Tests/FortranInterface/Advection_F/CMakeLists.txt Tests/FortranInterface/Advection_F/Exec/Make.Adv Tests/FortranInterface/Advection_F/Exec/SingleVortex/GNUmakefile Tests/FortranInterface/Advection_F/Exec/SingleVortex/Make.package Tests/FortranInterface/Advection_F/Exec/SingleVortex/Prob_2d.f90 Tests/FortranInterface/Advection_F/Exec/SingleVortex/Prob_3d.f90 Tests/FortranInterface/Advection_F/Exec/SingleVortex/face_velocity_2d.F90 Tests/FortranInterface/Advection_F/Exec/SingleVortex/face_velocity_3d.F90 Tests/FortranInterface/Advection_F/Exec/SingleVortex/inputs Tests/FortranInterface/Advection_F/Exec/SingleVortex/inputs.physbc Tests/FortranInterface/Advection_F/Exec/SingleVortex/inputs.rt Tests/FortranInterface/Advection_F/README Tests/FortranInterface/Advection_F/Source/Make.package Tests/FortranInterface/Advection_F/Source/Src_2d/Make.package Tests/FortranInterface/Advection_F/Source/Src_2d/advect_2d_mod.F90 Tests/FortranInterface/Advection_F/Source/Src_2d/compute_flux_2d.f90 Tests/FortranInterface/Advection_F/Source/Src_2d/slope_2d.f90 Tests/FortranInterface/Advection_F/Source/Src_3d/Make.package Tests/FortranInterface/Advection_F/Source/Src_3d/advect_3d_mod.F90 Tests/FortranInterface/Advection_F/Source/Src_3d/compute_flux_3d.f90 Tests/FortranInterface/Advection_F/Source/Src_3d/slope_3d.f90 Tests/FortranInterface/Advection_F/Source/amr_data_mod.F90 Tests/FortranInterface/Advection_F/Source/averagedown_mod.F90 Tests/FortranInterface/Advection_F/Source/bc_mod.F90 Tests/FortranInterface/Advection_F/Source/compute_dt_mod.F90 Tests/FortranInterface/Advection_F/Source/evolve_mod.F90 Tests/FortranInterface/Advection_F/Source/fillpatch_mod.F90 Tests/FortranInterface/Advection_F/Source/fmain.F90 Tests/FortranInterface/Advection_F/Source/initdata.F90 Tests/FortranInterface/Advection_F/Source/my_amr_mod.F90 Tests/FortranInterface/Advection_F/Source/plotfile_mod.F90 Tests/FortranInterface/Advection_F/Source/tagging_mod.F90 Tests/FortranInterface/Advection_octree_F/CMakeLists.txt Tests/FortranInterface/Advection_octree_F/Exec/Make.Adv Tests/FortranInterface/Advection_octree_F/Exec/SingleVortex/GNUmakefile Tests/FortranInterface/Advection_octree_F/Exec/SingleVortex/Make.package Tests/FortranInterface/Advection_octree_F/Exec/SingleVortex/Prob.f90 Tests/FortranInterface/Advection_octree_F/Exec/SingleVortex/face_velocity_2d.F90 Tests/FortranInterface/Advection_octree_F/Exec/SingleVortex/inputs Tests/FortranInterface/Advection_octree_F/Exec/SingleVortex/inputs.rt Tests/FortranInterface/Advection_octree_F/README Tests/FortranInterface/Advection_octree_F/Source/Make.package Tests/FortranInterface/Advection_octree_F/Source/Src_2d/Make.package Tests/FortranInterface/Advection_octree_F/Source/Src_2d/advect_2d_mod.F90 Tests/FortranInterface/Advection_octree_F/Source/Src_2d/compute_flux_2d.f90 Tests/FortranInterface/Advection_octree_F/Source/Src_2d/slope_2d.f90 Tests/FortranInterface/Advection_octree_F/Source/amr_data_mod.F90 Tests/FortranInterface/Advection_octree_F/Source/averagedown_mod.F90 Tests/FortranInterface/Advection_octree_F/Source/bc_mod.F90 Tests/FortranInterface/Advection_octree_F/Source/compute_dt_mod.F90 Tests/FortranInterface/Advection_octree_F/Source/evolve_mod.F90 Tests/FortranInterface/Advection_octree_F/Source/fillpatch_mod.F90 Tests/FortranInterface/Advection_octree_F/Source/fmain.F90 Tests/FortranInterface/Advection_octree_F/Source/initdata.F90 Tests/FortranInterface/Advection_octree_F/Source/my_amr_mod.F90 Tests/FortranInterface/Advection_octree_F/Source/plotfile_mod.F90 Tests/FortranInterface/Advection_octree_F/Source/tagging_mod.F90 Tests/GPU/CNS/CMakeLists.txt Tests/GPU/CNS/Exec/Make.CNS Tests/GPU/CNS/Exec/RT/GNUmakefile Tests/GPU/CNS/Exec/RT/Make.package Tests/GPU/CNS/Exec/RT/cns_prob.H Tests/GPU/CNS/Exec/RT/cns_prob.cpp Tests/GPU/CNS/Exec/RT/cns_prob_parm.H Tests/GPU/CNS/Exec/RT/inputs Tests/GPU/CNS/Exec/RT/inputs-rt Tests/GPU/CNS/Exec/Sod/GNUmakefile Tests/GPU/CNS/Exec/Sod/Make.package Tests/GPU/CNS/Exec/Sod/cns_prob.H Tests/GPU/CNS/Exec/Sod/cns_prob.cpp Tests/GPU/CNS/Exec/Sod/cns_prob_parm.H Tests/GPU/CNS/Exec/Sod/inputs Tests/GPU/CNS/Exec/Sod/inputs-rt Tests/GPU/CNS/Source/CNS.H Tests/GPU/CNS/Source/CNS.cpp Tests/GPU/CNS/Source/CNSBld.cpp Tests/GPU/CNS/Source/CNS_K.H Tests/GPU/CNS/Source/CNS_advance.cpp Tests/GPU/CNS/Source/CNS_bcfill.cpp Tests/GPU/CNS/Source/CNS_derive.H Tests/GPU/CNS/Source/CNS_derive.cpp Tests/GPU/CNS/Source/CNS_index_macros.H Tests/GPU/CNS/Source/CNS_io.cpp Tests/GPU/CNS/Source/CNS_parm.H Tests/GPU/CNS/Source/CNS_parm.cpp Tests/GPU/CNS/Source/CNS_setup.cpp Tests/GPU/CNS/Source/CNS_tagging.H Tests/GPU/CNS/Source/Make.package Tests/GPU/CNS/Source/diffusion/Make.package Tests/GPU/CNS/Source/hydro/CNS_hydro_K.H Tests/GPU/CNS/Source/hydro/Make.package Tests/GPU/CNS/Source/main.cpp Tests/GPU/Vector/inputs Tests/HDF5Benchmark/inputs Tests/LinearSolvers/ABecLaplacian_C/CMakeLists.txt Tests/LinearSolvers/ABecLaplacian_F/CMakeLists.txt Tests/LinearSolvers/MAC_Projection_EB/CMakeLists.txt Tests/LinearSolvers/NodalPoisson/CMakeLists.txt Tests/LinearSolvers/Nodal_Projection_EB/CMakeLists.txt Tests/LinearSolvers/NodeTensorLap/CMakeLists.txt Tests/MultiBlock/IndexType/main.cpp Tests/Particles/ParallelContext/CMakeLists.txt Tests/Particles/ParallelContext/inputs.rt Tests/Particles/ParallelContext/main.cpp Tests/Particles/Redistribute/inputs Tests/Particles/Redistribute/inputs.rt Tools/RegressionTesting/AMReX-cuda-tests.ini Tools/RegressionTesting/AMReX-dpcpp-tests.ini Tools/RegressionTesting/AMReX-hip-tests.ini Tools/RegressionTesting/AMReX-tests.ini Tutorials/Basic/Build_with_libamrex/GNUmakefile Tutorials/Basic/Build_with_libamrex/MyParams.H Tutorials/Basic/Build_with_libamrex/main.cpp Tutorials/Basic/Build_with_libamrex/my_func.f90 Tutorials/Basic/Build_with_libamrex/test_parameters.cpp Tutorials/Basic/HeatEquation_EX1_C/CMakeLists.txt Tutorials/Basic/HeatEquation_EX1_C/Exec/GNUmakefile Tutorials/Basic/HeatEquation_EX1_C/Exec/inputs Tutorials/Basic/HeatEquation_EX1_C/Source/Make.package Tutorials/Basic/HeatEquation_EX1_C/Source/main.cpp Tutorials/Basic/HeatEquation_EX1_C/Source/myfunc.H Tutorials/Basic/HeatEquation_EX1_C/Source/myfunc.cpp Tutorials/Basic/HeatEquation_EX1_C/Source/mykernel.H Tutorials/Basic/HeatEquation_EX1_CF/CMakeLists.txt Tutorials/Basic/HeatEquation_EX1_CF/Exec/GNUmakefile Tutorials/Basic/HeatEquation_EX1_CF/Exec/inputs_2d Tutorials/Basic/HeatEquation_EX1_CF/Exec/inputs_3d Tutorials/Basic/HeatEquation_EX1_CF/Source/Make.package Tutorials/Basic/HeatEquation_EX1_CF/Source/advance.cpp Tutorials/Basic/HeatEquation_EX1_CF/Source/advance_2d.f90 Tutorials/Basic/HeatEquation_EX1_CF/Source/advance_3d.f90 Tutorials/Basic/HeatEquation_EX1_CF/Source/init_phi_2d.f90 Tutorials/Basic/HeatEquation_EX1_CF/Source/init_phi_3d.f90 Tutorials/Basic/HeatEquation_EX1_CF/Source/main.cpp Tutorials/Basic/HeatEquation_EX1_CF/Source/myfunc.H Tutorials/Basic/HeatEquation_EX1_CF/Source/myfunc_F.H Tutorials/Basic/HeatEquation_EX1_F/CMakeLists.txt Tutorials/Basic/HeatEquation_EX1_F/GNUmakefile Tutorials/Basic/HeatEquation_EX1_F/Make.package Tutorials/Basic/HeatEquation_EX1_F/advance.f90 Tutorials/Basic/HeatEquation_EX1_F/fmain.F90 Tutorials/Basic/HeatEquation_EX1_F/init_phi.f90 Tutorials/Basic/HeatEquation_EX1_F/inputs Tutorials/Basic/HeatEquation_EX2_C/CMakeLists.txt Tutorials/Basic/HeatEquation_EX2_C/Exec/GNUmakefile Tutorials/Basic/HeatEquation_EX2_C/Exec/inputs Tutorials/Basic/HeatEquation_EX2_C/Source/Make.package Tutorials/Basic/HeatEquation_EX2_C/Source/main.cpp Tutorials/Basic/HeatEquation_EX2_C/Source/myfunc.H Tutorials/Basic/HeatEquation_EX2_C/Source/myfunc.cpp Tutorials/Basic/HeatEquation_EX2_C/Source/mykernel.H Tutorials/Basic/HeatEquation_EX2_CF/CMakeLists.txt Tutorials/Basic/HeatEquation_EX2_CF/Exec/GNUmakefile Tutorials/Basic/HeatEquation_EX2_CF/Exec/inputs_2d Tutorials/Basic/HeatEquation_EX2_CF/Exec/inputs_3d Tutorials/Basic/HeatEquation_EX2_CF/Source/Make.package Tutorials/Basic/HeatEquation_EX2_CF/Source/advance.cpp Tutorials/Basic/HeatEquation_EX2_CF/Source/advance_2d.f90 Tutorials/Basic/HeatEquation_EX2_CF/Source/advance_3d.f90 Tutorials/Basic/HeatEquation_EX2_CF/Source/init_phi_2d.f90 Tutorials/Basic/HeatEquation_EX2_CF/Source/init_phi_3d.f90 Tutorials/Basic/HeatEquation_EX2_CF/Source/main.cpp Tutorials/Basic/HeatEquation_EX2_CF/Source/myfunc.H Tutorials/Basic/HeatEquation_EX2_CF/Source/myfunc_F.H Tutorials/Basic/HeatEquation_EX3_C/CMakeLists.txt Tutorials/Basic/HeatEquation_EX3_C/Exec/GNUmakefile Tutorials/Basic/HeatEquation_EX3_C/Exec/inputs_2d Tutorials/Basic/HeatEquation_EX3_C/Exec/inputs_3d Tutorials/Basic/HeatEquation_EX3_C/Source/Make.package Tutorials/Basic/HeatEquation_EX3_C/Source/advance.cpp Tutorials/Basic/HeatEquation_EX3_C/Source/init_phi_2d.f90 Tutorials/Basic/HeatEquation_EX3_C/Source/init_phi_3d.f90 Tutorials/Basic/HeatEquation_EX3_C/Source/main.cpp Tutorials/Basic/HeatEquation_EX3_C/Source/myfunc.H Tutorials/Basic/HeatEquation_EX3_C/Source/myfunc_F.H Tutorials/Basic/HelloWorld_C/CMakeLists.txt Tutorials/Basic/HelloWorld_C/GNUmakefile Tutorials/Basic/HelloWorld_C/Make.package Tutorials/Basic/HelloWorld_C/main.cpp Tutorials/Basic/HelloWorld_F/CMakeLists.txt Tutorials/Basic/HelloWorld_F/GNUmakefile Tutorials/Basic/HelloWorld_F/Make.package Tutorials/Basic/HelloWorld_F/fmain.f90 Tutorials/Basic/PrefixSum_MultiFab/CMakeLists.txt Tutorials/Basic/PrefixSum_MultiFab/GNUmakefile Tutorials/Basic/PrefixSum_MultiFab/Make.package Tutorials/Basic/PrefixSum_MultiFab/Parallel-Prefix-Sum.ipynb Tutorials/Basic/PrefixSum_MultiFab/README.md Tutorials/Basic/PrefixSum_MultiFab/inputs Tutorials/Basic/PrefixSum_MultiFab/main.cpp Tutorials/Basic/main_C/CMakeLists.txt Tutorials/Basic/main_C/GNUmakefile Tutorials/Basic/main_C/Make.package Tutorials/Basic/main_C/main.cpp Tutorials/Basic/main_F/CMakeLists.txt Tutorials/Basic/main_F/GNUmakefile Tutorials/Basic/main_F/Make.package Tutorials/Basic/main_F/main.F90 Tutorials/Blueprint/AssignMultiLevelDensity/CMakeLists.txt Tutorials/Blueprint/AssignMultiLevelDensity/GNUmakefile Tutorials/Blueprint/AssignMultiLevelDensity/Make.package Tutorials/Blueprint/AssignMultiLevelDensity/inputs Tutorials/Blueprint/AssignMultiLevelDensity/main.cpp Tutorials/Blueprint/CellSortedParticles/CMakeLists.txt Tutorials/Blueprint/CellSortedParticles/CellSortedPC.H Tutorials/Blueprint/CellSortedParticles/CellSortedPC.cpp Tutorials/Blueprint/CellSortedParticles/GNUmakefile Tutorials/Blueprint/CellSortedParticles/Make.package Tutorials/Blueprint/CellSortedParticles/cell_sorted_3d.F90 Tutorials/Blueprint/CellSortedParticles/cell_sorted_F.H Tutorials/Blueprint/CellSortedParticles/inputs Tutorials/Blueprint/CellSortedParticles/main.cpp Tutorials/Blueprint/HeatEquation_EX1_C/CMakeLists.txt Tutorials/Blueprint/HeatEquation_EX1_C/Exec/GNUmakefile Tutorials/Blueprint/HeatEquation_EX1_C/Exec/inputs_2d Tutorials/Blueprint/HeatEquation_EX1_C/Exec/inputs_3d Tutorials/Blueprint/HeatEquation_EX1_C/Source/Make.package Tutorials/Blueprint/HeatEquation_EX1_C/Source/advance.cpp Tutorials/Blueprint/HeatEquation_EX1_C/Source/advance_2d.f90 Tutorials/Blueprint/HeatEquation_EX1_C/Source/advance_3d.f90 Tutorials/Blueprint/HeatEquation_EX1_C/Source/init_phi_2d.f90 Tutorials/Blueprint/HeatEquation_EX1_C/Source/init_phi_3d.f90 Tutorials/Blueprint/HeatEquation_EX1_C/Source/main.cpp Tutorials/Blueprint/HeatEquation_EX1_C/Source/myfunc.H Tutorials/Blueprint/HeatEquation_EX1_C/Source/myfunc_F.H Tutorials/Blueprint/README.txt Tutorials/CMakeLists.txt Tutorials/EB/GeometryGeneration/CMakeLists.txt Tutorials/EB/GeometryGeneration/GNUmakefile Tutorials/EB/GeometryGeneration/Make.package Tutorials/EB/GeometryGeneration/main.cpp Tutorials/EB/MacProj/CMakeLists.txt Tutorials/EB/MacProj/GNUmakefile Tutorials/EB/MacProj/Make.package Tutorials/EB/MacProj/inputs Tutorials/EB/MacProj/main.cpp Tutorials/EB/Poisson/CMakeLists.txt Tutorials/EB/Poisson/GNUmakefile Tutorials/EB/Poisson/Make.package Tutorials/EB/Poisson/Poisson.H Tutorials/EB/Poisson/Poisson.cpp Tutorials/EB/Poisson/inputs Tutorials/EB/Poisson/main.cpp Tutorials/EB/STLtest/GNUmakefile Tutorials/EB/STLtest/Make.package Tutorials/EB/STLtest/airfoil.stl Tutorials/EB/STLtest/inputs Tutorials/EB/STLtest/main.cpp Tutorials/ForkJoin/MLMG/CMakeLists.txt Tutorials/ForkJoin/MLMG/GNUmakefile Tutorials/ForkJoin/MLMG/Make.package Tutorials/ForkJoin/MLMG/ff.f90 Tutorials/ForkJoin/MLMG/inputs Tutorials/ForkJoin/MLMG/main.cpp Tutorials/ForkJoin/Simple/CMakeLists.txt Tutorials/ForkJoin/Simple/GNUmakefile Tutorials/ForkJoin/Simple/Make.package Tutorials/ForkJoin/Simple/MyTest.H Tutorials/ForkJoin/Simple/MyTest.cpp Tutorials/ForkJoin/Simple/MyTest_F.H Tutorials/ForkJoin/Simple/inputs Tutorials/ForkJoin/Simple/main.cpp Tutorials/FortranInterface/Advection_octree_F2/CMakeLists.txt Tutorials/FortranInterface/Advection_octree_F2/Exec/Make.Adv Tutorials/FortranInterface/Advection_octree_F2/Exec/SingleVortex/GNUmakefile Tutorials/FortranInterface/Advection_octree_F2/Exec/SingleVortex/Make.package Tutorials/FortranInterface/Advection_octree_F2/Exec/SingleVortex/Prob.f90 Tutorials/FortranInterface/Advection_octree_F2/Exec/SingleVortex/face_velocity_2d.F90 Tutorials/FortranInterface/Advection_octree_F2/Exec/SingleVortex/inputs Tutorials/FortranInterface/Advection_octree_F2/README Tutorials/FortranInterface/Advection_octree_F2/Source/Make.package Tutorials/FortranInterface/Advection_octree_F2/Source/Src_2d/Make.package Tutorials/FortranInterface/Advection_octree_F2/Source/Src_2d/advect_2d_mod.F90 Tutorials/FortranInterface/Advection_octree_F2/Source/Src_2d/compute_flux_2d.f90 Tutorials/FortranInterface/Advection_octree_F2/Source/Src_2d/slope_2d.f90 Tutorials/FortranInterface/Advection_octree_F2/Source/amr_data_mod.F90 Tutorials/FortranInterface/Advection_octree_F2/Source/averagedown_mod.F90 Tutorials/FortranInterface/Advection_octree_F2/Source/bc_mod.F90 Tutorials/FortranInterface/Advection_octree_F2/Source/compute_dt_mod.F90 Tutorials/FortranInterface/Advection_octree_F2/Source/evolve_mod.F90 Tutorials/FortranInterface/Advection_octree_F2/Source/fillpatch_mod.F90 Tutorials/FortranInterface/Advection_octree_F2/Source/fmain.F90 Tutorials/FortranInterface/Advection_octree_F2/Source/initdata.F90 Tutorials/FortranInterface/Advection_octree_F2/Source/my_amr_mod.F90 Tutorials/FortranInterface/Advection_octree_F2/Source/plotfile_mod.F90 Tutorials/FortranInterface/Advection_octree_F2/Source/tagging_mod.F90 Tutorials/GPU/EBCNS/CMakeLists.txt Tutorials/GPU/EBCNS/Exec/Make.CNS Tutorials/GPU/EBCNS/Exec/Sod/GNUmakefile Tutorials/GPU/EBCNS/Exec/Sod/Make.package Tutorials/GPU/EBCNS/Exec/Sod/cns_prob.H Tutorials/GPU/EBCNS/Exec/Sod/cns_prob.cpp Tutorials/GPU/EBCNS/Exec/Sod/cns_prob_parm.H Tutorials/GPU/EBCNS/Exec/Sod/inputs Tutorials/GPU/EBCNS/README Tutorials/GPU/EBCNS/Source/CNS.H Tutorials/GPU/EBCNS/Source/CNS.cpp Tutorials/GPU/EBCNS/Source/CNSBld.cpp Tutorials/GPU/EBCNS/Source/CNS_K.H Tutorials/GPU/EBCNS/Source/CNS_advance.cpp Tutorials/GPU/EBCNS/Source/CNS_bcfill.cpp Tutorials/GPU/EBCNS/Source/CNS_derive.H Tutorials/GPU/EBCNS/Source/CNS_derive.cpp Tutorials/GPU/EBCNS/Source/CNS_index_macros.H Tutorials/GPU/EBCNS/Source/CNS_init_eb2.cpp Tutorials/GPU/EBCNS/Source/CNS_io.cpp Tutorials/GPU/EBCNS/Source/CNS_parm.H Tutorials/GPU/EBCNS/Source/CNS_parm.cpp Tutorials/GPU/EBCNS/Source/CNS_setup.cpp Tutorials/GPU/EBCNS/Source/CNS_tagging.H Tutorials/GPU/EBCNS/Source/Make.package Tutorials/GPU/EBCNS/Source/diffusion/Make.package Tutorials/GPU/EBCNS/Source/hydro/CNS_hydro_K.H Tutorials/GPU/EBCNS/Source/hydro/Make.package Tutorials/GPU/EBCNS/Source/main.cpp Tutorials/GPU/Launch/CMakeLists.txt Tutorials/GPU/Launch/GNUmakefile Tutorials/GPU/Launch/Make.package Tutorials/GPU/Launch/MyKernel_F.F90 Tutorials/GPU/Launch/MyKernel_F.H Tutorials/GPU/Launch/Readme.md Tutorials/GPU/Launch/main.cpp Tutorials/GPU/ParallelReduce/CMakeLists.txt Tutorials/GPU/ParallelReduce/GNUmakefile Tutorials/GPU/ParallelReduce/Make.package Tutorials/GPU/ParallelReduce/main.cpp Tutorials/GPU/ParallelScan/CMakeLists.txt Tutorials/GPU/ParallelScan/GNUmakefile Tutorials/GPU/ParallelScan/Make.package Tutorials/GPU/ParallelScan/main.cpp Tutorials/GPU/run.corigpu Tutorials/GPU/run.summit Tutorials/MUI/Exec_01/GNUmakefile Tutorials/MUI/Exec_02/GNUmakefile Tutorials/MUI/Exec_coupled/cmd_mpirun Tutorials/MUI/Exec_coupled/inputs Tutorials/MUI/Source_01/Make.package Tutorials/MUI/Source_01/init_phi_3d.f90 Tutorials/MUI/Source_01/main_01.cpp Tutorials/MUI/Source_01/myfunc.H Tutorials/MUI/Source_01/myfunc_F.H Tutorials/MUI/Source_02/Make.package Tutorials/MUI/Source_02/main_02.cpp Tutorials/MUI/Source_02/myfunc.H Tutorials/MUI/doc/GNUmakefile Tutorials/MUI/doc/MUIcouplingNotes.tex Tutorials/MUI/doc/iface_rect.png Tutorials/MUI/doc/vis_interface.m Tutorials/Particles/CellSortedParticles/CMakeLists.txt Tutorials/Particles/CellSortedParticles/CellSortedPC.H Tutorials/Particles/CellSortedParticles/CellSortedPC.cpp Tutorials/Particles/CellSortedParticles/GNUmakefile Tutorials/Particles/CellSortedParticles/Make.package Tutorials/Particles/CellSortedParticles/cell_sorted_3d.F90 Tutorials/Particles/CellSortedParticles/cell_sorted_F.H Tutorials/Particles/CellSortedParticles/inputs Tutorials/Particles/CellSortedParticles/main.cpp Tutorials/Particles/ElectromagneticPIC/CMakeLists.txt Tutorials/Particles/ElectromagneticPIC/Exec/CUDA/EMParticleContainer.cpp Tutorials/Particles/ElectromagneticPIC/Exec/CUDA/Evolve.cpp Tutorials/Particles/ElectromagneticPIC/Exec/CUDA/GNUmakefile Tutorials/Particles/ElectromagneticPIC/Exec/CUDA/Make.package Tutorials/Particles/ElectromagneticPIC/Exec/CUDA/em_pic_K.H Tutorials/Particles/ElectromagneticPIC/Exec/CUDA/inputs Tutorials/Particles/ElectromagneticPIC/Exec/OpenACC/EMParticleContainer.cpp Tutorials/Particles/ElectromagneticPIC/Exec/OpenACC/Evolve.cpp Tutorials/Particles/ElectromagneticPIC/Exec/OpenACC/GNUmakefile Tutorials/Particles/ElectromagneticPIC/Exec/OpenACC/Make.package Tutorials/Particles/ElectromagneticPIC/Exec/OpenACC/em_pic_3d.F90 Tutorials/Particles/ElectromagneticPIC/Exec/OpenACC/em_pic_F.H Tutorials/Particles/ElectromagneticPIC/Exec/OpenACC/inputs Tutorials/Particles/ElectromagneticPIC/Exec/OpenACC/script.sh Tutorials/Particles/ElectromagneticPIC/Exec/OpenMP/EMParticleContainer.cpp Tutorials/Particles/ElectromagneticPIC/Exec/OpenMP/Evolve.cpp Tutorials/Particles/ElectromagneticPIC/Exec/OpenMP/GNUmakefile Tutorials/Particles/ElectromagneticPIC/Exec/OpenMP/GNUmakefile.libamrex Tutorials/Particles/ElectromagneticPIC/Exec/OpenMP/Make.package Tutorials/Particles/ElectromagneticPIC/Exec/OpenMP/em_pic_3d.F90 Tutorials/Particles/ElectromagneticPIC/Exec/OpenMP/em_pic_F.H Tutorials/Particles/ElectromagneticPIC/Exec/OpenMP/inputs Tutorials/Particles/ElectromagneticPIC/Exec/OpenMP/run.summitdev Tutorials/Particles/ElectromagneticPIC/Make.EMPIC Tutorials/Particles/ElectromagneticPIC/Source/Constants.H Tutorials/Particles/ElectromagneticPIC/Source/EMParticleContainer.H Tutorials/Particles/ElectromagneticPIC/Source/EMParticleContainerInit.cpp Tutorials/Particles/ElectromagneticPIC/Source/Evolve.H Tutorials/Particles/ElectromagneticPIC/Source/IO.H Tutorials/Particles/ElectromagneticPIC/Source/IO.cpp Tutorials/Particles/ElectromagneticPIC/Source/Make.package Tutorials/Particles/ElectromagneticPIC/Source/NodalFlags.H Tutorials/Particles/ElectromagneticPIC/Source/NodalFlags.cpp Tutorials/Particles/ElectromagneticPIC/Source/main.cpp Tutorials/Particles/ElectrostaticPIC/CMakeLists.txt Tutorials/Particles/ElectrostaticPIC/ElectrostaticParticleContainer.H Tutorials/Particles/ElectrostaticPIC/ElectrostaticParticleContainer.cpp Tutorials/Particles/ElectrostaticPIC/GNUmakefile Tutorials/Particles/ElectrostaticPIC/Make.package Tutorials/Particles/ElectrostaticPIC/PhysConst.H Tutorials/Particles/ElectrostaticPIC/electrostatic_pic_2d.f90 Tutorials/Particles/ElectrostaticPIC/electrostatic_pic_3d.f90 Tutorials/Particles/ElectrostaticPIC/electrostatic_pic_F.H Tutorials/Particles/ElectrostaticPIC/inputs Tutorials/Particles/ElectrostaticPIC/main.cpp Tutorials/Particles/NeighborList/CMakeLists.txt Tutorials/Particles/NeighborList/CheckPair.H Tutorials/Particles/NeighborList/Constants.H Tutorials/Particles/NeighborList/GNUmakefile Tutorials/Particles/NeighborList/MDParticleContainer.H Tutorials/Particles/NeighborList/MDParticleContainer.cpp Tutorials/Particles/NeighborList/Make.package Tutorials/Particles/NeighborList/README.md Tutorials/Particles/NeighborList/inputs Tutorials/Particles/NeighborList/main.cpp Tutorials/Particles/NeighborList/script.sh Tutorials/README.md Tutorials/SDC/MISDC_ADR_2d/Exec/GNUmakefile Tutorials/SDC/MISDC_ADR_2d/Exec/inputs_2d Tutorials/SDC/MISDC_ADR_2d/README Tutorials/SDC/MISDC_ADR_2d/Source/Make.package Tutorials/SDC/MISDC_ADR_2d/Source/SDC_sweeper.cpp Tutorials/SDC/MISDC_ADR_2d/Source/functions_2d.f90 Tutorials/SDC/MISDC_ADR_2d/Source/init_phi_2d.f90 Tutorials/SDC/MISDC_ADR_2d/Source/main.cpp Tutorials/SDC/MISDC_ADR_2d/Source/myfunc.H Tutorials/SDC/MISDC_ADR_2d/Source/myfunc_F.H Tutorials/SWFFT/SWFFT_poisson/GNUmakefile Tutorials/SWFFT/SWFFT_poisson/Make.package Tutorials/SWFFT/SWFFT_poisson/README Tutorials/SWFFT/SWFFT_poisson/SWFFT_Test.H Tutorials/SWFFT/SWFFT_poisson/SWFFT_Test.cpp Tutorials/SWFFT/SWFFT_poisson/SWFFT_Test_F.F90 Tutorials/SWFFT/SWFFT_poisson/SWFFT_Test_F.H Tutorials/SWFFT/SWFFT_poisson/inputs.128 Tutorials/SWFFT/SWFFT_poisson/inputs.32 Tutorials/SWFFT/SWFFT_poisson/inputs.64 Tutorials/SWFFT/SWFFT_poisson/main.cpp Tutorials/SWFFT/SWFFT_poisson/run_me Tutorials/SWFFT/SWFFT_poisson/swfft_solver.cpp Tutorials/SWFFT/SWFFT_simple/GNUmakefile Tutorials/SWFFT/SWFFT_simple/Make.package Tutorials/SWFFT/SWFFT_simple/README Tutorials/SWFFT/SWFFT_simple/SWFFT_Test.H Tutorials/SWFFT/SWFFT_simple/SWFFT_Test.cpp Tutorials/SWFFT/SWFFT_simple/SWFFT_Test_F.F90 Tutorials/SWFFT/SWFFT_simple/SWFFT_Test_F.H Tutorials/SWFFT/SWFFT_simple/inputs.multipleGrids Tutorials/SWFFT/SWFFT_simple/inputs.oneGrid Tutorials/SWFFT/SWFFT_simple/main.cpp Tutorials/SWFFT/SWFFT_simple/run_me_2d Tutorials/SWFFT/SWFFT_simple/run_me_3d Tutorials/SWFFT/SWFFT_simple/swfft_compute.cpp commit 384ca2341889a8f1db78e87cb4b1735bd2d58554 Author: Axel Huebl Date: Mon Mar 22 10:09:53 2021 -0700 CMake: IPO/LTO (#1890) Add optional interprocedural optimization (IPO) aka link-time optimization (LTO) to AMReX. In my WarpX tests recently on x86_64 and ppc64le, using those options incurred a 2% performance hit (although one would expect a slight increase). Thus, they are not enabled by default. Also, enabling them with some toolchains needs users to set the linker properly. Nontheless, enabling IPO shrinks the binary size a lot. This is very valuable and worth the runtime penalty for our binary deployments, e.g. when shipping generic x86_64, aarch64 and ppc64le binaries on conda-forge to users. WarpX binary size (default build): * no IPO: 217MiB * WarpX IPO, AMReX no IPO: 155MiB * both WarpX and AMReX with IPO: 127M Docs/sphinx_documentation/source/BuildingAMReX.rst Tools/CMake/AMReXBuildInfo.cmake Tools/CMake/AMReXOptions.cmake Tools/CMake/AMReXTypecheck.cmake Tools/CMake/AMReX_Config.cmake commit ef0eb9f5c4b065b38e3a3242a43263aba43aa874 Author: Weiqun Zhang Date: Sun Mar 21 14:35:07 2021 -0700 Header file order and iosfwd (#1883) Include amrex headers before standard C++ headers. Use iosfwd instead of iostream if we can. Src/Amr/AMReX_Amr.H Src/Amr/AMReX_Amr.cpp Src/Amr/AMReX_AmrLevel.cpp Src/Amr/AMReX_Derive.H Src/Amr/AMReX_Derive.cpp Src/Amr/AMReX_StateData.H Src/Amr/AMReX_StateData.cpp Src/Amr/AMReX_StateDescriptor.H Src/Amr/AMReX_StateDescriptor.cpp Src/AmrCore/AMReX_AmrCore.H Src/AmrCore/AMReX_AmrCore.cpp Src/AmrCore/AMReX_Cluster.H Src/AmrCore/AMReX_Cluster.cpp Src/AmrCore/AMReX_ErrorList.H Src/AmrCore/AMReX_ErrorList.cpp Src/AmrCore/AMReX_FillPatchUtil.H Src/AmrCore/AMReX_Interpolater.cpp Src/AmrCore/AMReX_TagBox.cpp Src/Base/AMReX.H Src/Base/AMReX_Array.H Src/Base/AMReX_Array4.H Src/Base/AMReX_BCRec.cpp Src/Base/AMReX_BLBackTrace.H Src/Base/AMReX_BLProfiler.H Src/Base/AMReX_BLProfiler.cpp Src/Base/AMReX_BaseFab.cpp Src/Base/AMReX_BaseUmap.H Src/Base/AMReX_BoxArray.H Src/Base/AMReX_BoxArray.cpp Src/Base/AMReX_BoxDomain.H Src/Base/AMReX_BoxDomain.cpp Src/Base/AMReX_BoxIterator.H Src/Base/AMReX_BoxIterator.cpp Src/Base/AMReX_BoxList.H Src/Base/AMReX_BoxList.cpp Src/Base/AMReX_CoordSys.H Src/Base/AMReX_CoordSys.cpp Src/Base/AMReX_Dim3.H Src/Base/AMReX_Dim3.cpp Src/Base/AMReX_DistributionMapping.H Src/Base/AMReX_FACopyDescriptor.H Src/Base/AMReX_FArrayBox.cpp Src/Base/AMReX_FabArray.H Src/Base/AMReX_FabArrayBase.H Src/Base/AMReX_FabArrayBase.cpp Src/Base/AMReX_FabConv.H Src/Base/AMReX_Geometry.H Src/Base/AMReX_Geometry.cpp Src/Base/AMReX_GpuAllocators.H Src/Base/AMReX_GpuAsyncArray.H Src/Base/AMReX_GpuComplex.H Src/Base/AMReX_GpuContainers.H Src/Base/AMReX_GpuDevice.cpp Src/Base/AMReX_GpuUtility.H Src/Base/AMReX_IndexType.H Src/Base/AMReX_IndexType.cpp Src/Base/AMReX_IntConv.H Src/Base/AMReX_IntVect.cpp Src/Base/AMReX_MFIter.H Src/Base/AMReX_MemPool.cpp Src/Base/AMReX_MemProfiler.H Src/Base/AMReX_MemProfiler.cpp Src/Base/AMReX_MultiFab.H Src/Base/AMReX_MultiFab.cpp Src/Base/AMReX_NFiles.H Src/Base/AMReX_NFiles.cpp Src/Base/AMReX_Orientation.H Src/Base/AMReX_Orientation.cpp Src/Base/AMReX_ParallelContext.H Src/Base/AMReX_ParallelContext.cpp Src/Base/AMReX_ParallelDescriptor.H Src/Base/AMReX_ParallelDescriptor.cpp Src/Base/AMReX_Periodicity.cpp Src/Base/AMReX_PlotFileDataImpl.H Src/Base/AMReX_PlotFileDataImpl.cpp Src/Base/AMReX_PlotFileUtil.H Src/Base/AMReX_PlotFileUtil.cpp Src/Base/AMReX_Print.H Src/Base/AMReX_Random.cpp Src/Base/AMReX_RealVect.H Src/Base/AMReX_RealVect.cpp Src/Base/AMReX_TinyProfiler.H Src/Base/AMReX_TinyProfiler.cpp Src/Base/AMReX_Tuple.H Src/Base/AMReX_TypeTraits.H Src/Base/AMReX_Utility.H Src/Base/AMReX_Vector.H Src/Base/AMReX_VectorIO.H Src/Base/AMReX_VectorIO.cpp Src/Base/AMReX_VisMF.H Src/Base/AMReX_VisMF.cpp Src/Base/AMReX_iMultiFab.cpp Src/Base/AMReX_parmparse_fi.cpp Src/Base/AMReX_parstream.H Src/Base/AMReX_parstream.cpp Src/Base/CMakeLists.txt Src/Base/Make.package Src/Boundary/AMReX_Mask.cpp Src/EB/AMReX_EB2_IF_Base.H Src/EB/AMReX_EB2_IF_Spline.H Src/EB/AMReX_EB2_Level.H Src/EB/AMReX_EBCellFlag.H Src/EB/AMReX_EBCellFlag.cpp Src/EB/AMReX_EBToPVD.H Src/EB/AMReX_EB_STL_utils.H Src/EB/AMReX_distFcnElement.H Src/Extern/HYPRE/AMReX_Hypre.H Src/Extern/HYPRE/AMReX_HypreABecLap.H Src/Extern/HYPRE/AMReX_HypreABecLap.cpp Src/Extern/HYPRE/AMReX_HypreIJIface.H Src/Extern/HYPRE/AMReX_HypreNodeLap.H Src/F_Interfaces/AmrCore/AMReX_FAmrCore.H Src/Particle/AMReX_Particle.H Src/Particle/AMReX_ParticleMPIUtil.H Src/Particle/AMReX_Particles.H Src/Particle/AMReX_TracerParticle_mod_K.H Src/Particle/AMReX_TracerParticles.cpp commit c984473e021aabdd5f5d2091dcc57481e8637c7c Author: Weiqun Zhang Date: Sun Mar 21 11:38:21 2021 -0700 Remove outdated notes (#1889) Docs/Notes/DPCPPWishlist.md Docs/Notes/HIPIssues.md Docs/Notes/Notes.io_implementation commit 15f5168a1ad9f49d87ff0ee25935e171fe1d0f19 Author: Weiqun Zhang Date: Fri Mar 19 20:32:22 2021 -0700 amrex::EnableIf_t -> std::enable_if_t (#1885) Since we require C++14 now, we can use std::enable_if_t now. Src/AmrCore/AMReX_FillPatchUtil.H Src/AmrCore/AMReX_FillPatchUtil_I.H Src/Base/AMReX_FBI.H Src/Base/AMReX_FabArrayCommI.H Src/Base/AMReX_FabArrayUtility.H Src/Base/AMReX_GpuFuse.H Src/Base/AMReX_GpuLaunchFunctsC.H Src/Base/AMReX_GpuLaunchFunctsG.H Src/Base/AMReX_NonLocalBC.H Src/Base/AMReX_NonLocalBCImpl.H Src/Base/AMReX_Reduce.H Src/Base/AMReX_Scan.H Src/Base/AMReX_Tuple.H Src/Base/AMReX_TypeTraits.H Src/Particle/AMReX_ParticleCommunication.H Src/Particle/AMReX_ParticleContainerI.H Src/Particle/AMReX_ParticleMesh.H Src/Particle/AMReX_ParticleReduce.H Src/Particle/AMReX_ParticleTile.H Src/Particle/AMReX_ParticleTransformation.H Src/Particle/AMReX_ParticleUtil.H Src/Particle/AMReX_Particles.H Src/Particle/AMReX_WriteBinaryParticleData.H commit 1dfb98c29afe574afe30101a8db82190d9eadaa6 Author: Andrew Myers Date: Fri Mar 19 23:31:57 2021 -0400 remove __attribute__((warn_unused_result)) from AMREX_NODISCARD (#1886) Src/Base/AMReX_Extension.H commit 352059135e3e87f3b35e237b1c7fb8961eabffa1 Author: Weiqun Zhang Date: Fri Mar 19 18:11:52 2021 -0700 Make FabArray::setFab safer (#1884) Instead of taking a raw pointer, it takes std::unique_ptr or rvalue FAB, so that the ownership is very clear. It also deletes the old FAB if not null. Src/Base/AMReX_FArrayBox.H Src/Base/AMReX_FabArray.H Src/Base/AMReX_MultiFabUtil.cpp Src/Base/AMReX_ParallelDescriptor.H Src/EB/AMReX_EB2_MultiGFab.cpp Src/EB/AMReX_MultiCutFab.cpp Src/Extern/amrdata/AMReX_AmrData.cpp commit 9e1576fab02e20b19d54105e6e7a434bf6d6f040 Author: Weiqun Zhang Date: Fri Mar 19 17:00:02 2021 -0700 Fix typo in AMREX_DEBUG macro (#1881) Fix the call to CheckRcvStats. Src/Base/AMReX_FabArrayBase.H Src/Base/AMReX_FabArrayBase.cpp Src/Base/AMReX_NonLocalBC.H Src/Base/AMReX_NonLocalBCImpl.H commit ea9ba93abd20c9a4c79f2ae8c392cd62a553119f Author: Maikel Nadolski Date: Fri Mar 19 23:23:51 2021 +0100 Forward declare functions that are used in the implementation (#1882) This PR forward declares the functions local_copy_cpu unpack_recv_buffer_cpu local_copy_gpu unpack_recv_buffer_gpu before its first usage. Some compilers error out if a function is used before it was declared, even in template code. I am not sure, why it worked in the CI so far. Src/Base/AMReX_NonLocalBC.H Src/Base/AMReX_NonLocalBCImpl.H commit ca1839a8d9d805e915d6bd9d385a0c3087ad866d Author: Axel Huebl Date: Fri Mar 19 12:44:33 2021 -0700 Flags: RelWithDebInfo (#1879) Add warning control to CMake Build mode `RelWithDebugInfo` (currently default for WarpX). Tools/CMake/AMReXFlagsTargets.cmake commit 82664e56d3dbdefa75bec47ddd871996778805c3 Author: Axel Huebl Date: Fri Mar 19 08:47:04 2021 -0700 Mask: -Wfinal-dtor-non-final-class (#1880) ## Summary ``` Src/Boundary/AMReX_Mask.H:57:40: warning: class with destructor marked 'final' cannot be inherited from [-Wfinal-dtor-non-final-class] virtual ~Mask () noexcept override final {} ^ Src/Boundary/AMReX_Mask.H:26:7: note: mark 'amrex::Mask' as 'final' to silence this warning class Mask ``` Fix the warning in TagBox, CutFab and EBCellFlagFab as well. ## Additional background Seen with Clang 10.0 on Ubuntu 20.04 LTS. Co-authored-by: Weiqun Zhang Src/AmrCore/AMReX_TagBox.H Src/Boundary/AMReX_Mask.H Src/EB/AMReX_EBCellFlag.H Src/EB/AMReX_MultiCutFab.H commit 0e1acd6c559b8939ac323446963dfafbf7430052 Author: Andrew Myers Date: Fri Mar 19 01:50:12 2021 -0400 Fix Typedescriptor test for Windows (#1878) Tests/Particles/TypeDescriptor/main.cpp commit 281ff0fb583047355e0a726805c08d5d4e20cf67 Author: Axel Huebl Date: Thu Mar 18 11:42:30 2021 -0700 Build: Windows with Shared Libs (.dll) (#1847) * CI: Win with Shared Libs (.dll) Demonstrator of shared library issues on Windows. * Remove: warn-unresolved-symbols / undefined,warning * CMake typecheckobjs lib: export symbols Make this library behave like on Unix: symbol visibility by default. * Globals: dllimport/dllexport (Win) https://stackoverflow.com/questions/54560832/cmake-windows-export-all-symbols-does-not-cover-global-variables/54568678#54568678 .github/workflows/windows.yml Src/AmrCore/AMReX_Interpolater.H Src/Base/AMReX.H Src/Base/AMReX_Extension.H Src/Base/AMReX_MFIter.H Src/Base/AMReX_ParallelContext.H Src/Base/AMReX_ParallelDescriptor.H Src/Base/AMReX_VisMF.H Src/EB/AMReX_EBInterpolater.H Src/Particle/AMReX_ParticleContainerBase.H Tools/CMake/AMReXSetDefines.cmake Tools/CMake/AMReXTypecheck.cmake Tools/CMake/AMReX_Config.H.in Tools/CMake/AMReX_Config.cmake Tools/GNUMake/tools/Make.sensei commit 17371b17ffd5f7794a91236de7fa464d4284fa86 Author: Andrew Myers Date: Thu Mar 18 13:24:45 2021 -0400 Remove tutorials documentation from main repo; it now lives at https://github.com/AMReX-Codes/amrex-tutorials (#1877) Docs/sphinx_tutorials/Makefile Docs/sphinx_tutorials/source/AMR_Tutorial.rst Docs/sphinx_tutorials/source/Basic_Tutorial.rst Docs/sphinx_tutorials/source/Blueprint_Tutorial.rst Docs/sphinx_tutorials/source/EB_Tutorial.rst Docs/sphinx_tutorials/source/ForkJoin_Tutorial.rst Docs/sphinx_tutorials/source/GPU_Tutorial.rst Docs/sphinx_tutorials/source/LinearSolvers_Tutorial.rst Docs/sphinx_tutorials/source/MUI_Tutorial.rst Docs/sphinx_tutorials/source/Particles_Tutorial.rst Docs/sphinx_tutorials/source/SDC_Tutorial.rst Docs/sphinx_tutorials/source/SWFFT/iface_rect.png Docs/sphinx_tutorials/source/SWFFT_Tutorial.rst Docs/sphinx_tutorials/source/_static/theme_overrides.css Docs/sphinx_tutorials/source/conf.py Docs/sphinx_tutorials/source/figs/fork_join_tasks.png Docs/sphinx_tutorials/source/figs/mf_remap_hires.png Docs/sphinx_tutorials/source/figs/nested_fork_join_tasks.png Docs/sphinx_tutorials/source/index.rst build_docs.sh commit b31d050d176c822d51dad418a868d62426c6a1b4 Author: Weiqun Zhang Date: Wed Mar 17 19:13:57 2021 -0700 2D Poisson and ALaplacian Solvers in 3D Build of AMReX (#1805) New capability of solving 2D cell-centered Poisson's and ALaplacian equation with 3D build of amrex. Src/Base/AMReX_Box.H Src/Base/AMReX_Box.cpp Src/Base/AMReX_BoxArray.H Src/Base/AMReX_BoxArray.cpp Src/LinearSolvers/CMakeLists.txt Src/LinearSolvers/MLMG/AMReX_MLALap_2D_K.H Src/LinearSolvers/MLMG/AMReX_MLALap_K.H Src/LinearSolvers/MLMG/AMReX_MLALaplacian.cpp Src/LinearSolvers/MLMG/AMReX_MLCGSolver.cpp Src/LinearSolvers/MLMG/AMReX_MLCellABecLap.cpp Src/LinearSolvers/MLMG/AMReX_MLCellLinOp.cpp Src/LinearSolvers/MLMG/AMReX_MLLinOp.H Src/LinearSolvers/MLMG/AMReX_MLLinOp.cpp Src/LinearSolvers/MLMG/AMReX_MLMG.cpp Src/LinearSolvers/MLMG/AMReX_MLMG_2D_K.H Src/LinearSolvers/MLMG/AMReX_MLMG_K.H Src/LinearSolvers/MLMG/AMReX_MLNodeLinOp.cpp Src/LinearSolvers/MLMG/AMReX_MLPoisson.cpp Src/LinearSolvers/MLMG/AMReX_MLPoisson_2D_K.H Src/LinearSolvers/MLMG/AMReX_MLPoisson_K.H Src/LinearSolvers/MLMG/Make.package commit 14b6bff6954d677b496af065c8c9def6bacc2d43 Author: Weiqun Zhang Date: Wed Mar 17 18:26:39 2021 -0700 Add FabArray::release and Make BaseFab virtual (#1873) Add FabArray::release functions that release the ownership of the FAB at the given index and return a raw pointer to the FAB. Make BaseFab virtual so that it's safe to use the new release functions. Also declare a number of functions `[[nodiscard]]`. Fix some warnings in EB/CNS example. Src/AmrCore/AMReX_TagBox.H Src/Base/AMReX_BaseFab.H Src/Base/AMReX_FArrayBox.H Src/Base/AMReX_FabArray.H Src/Base/AMReX_FabFactory.H Src/Base/AMReX_IArrayBox.H Src/Boundary/AMReX_Mask.H Src/EB/AMReX_EBCellFlag.H Src/EB/AMReX_EBFabFactory.H Src/EB/AMReX_EBFabFactory.cpp Src/EB/AMReX_MultiCutFab.H Tutorials/EB/CNS/Source/CNS.cpp Tutorials/EB/CNS/Source/CNS_advance.cpp Tutorials/EB/CNS/Source/CNS_init_eb2.cpp commit 87b182a718574cc52a93750a4c484c2e5aa4264b Author: Weiqun Zhang Date: Wed Mar 17 15:16:57 2021 -0700 Robin BC in Linear Solver (#1848) Add Robin BC to MLABecLaplacian. There are two steps for setting up Robin BC. The first step is to call setDomainBC, and the second step is to call setLevelBC, which has been modified to take a, b and f in Robin BC, `a*phi + b*dphi/dn = f`. Docs/sphinx_documentation/source/LinearSolvers.rst Src/Boundary/AMReX_LO_BCTYPES.H Src/Boundary/AMReX_LO_BCTYPES.cpp Src/LinearSolvers/MLMG/AMReX_MLABecLaplacian.H Src/LinearSolvers/MLMG/AMReX_MLABecLaplacian.cpp Src/LinearSolvers/MLMG/AMReX_MLCellABecLap.H Src/LinearSolvers/MLMG/AMReX_MLCellABecLap.cpp Src/LinearSolvers/MLMG/AMReX_MLCellLinOp.H Src/LinearSolvers/MLMG/AMReX_MLCellLinOp.cpp Src/LinearSolvers/MLMG/AMReX_MLLinOp.H Src/LinearSolvers/MLMG/AMReX_MLLinOp.cpp Src/LinearSolvers/MLMG/AMReX_MLNodeLinOp.H Src/LinearSolvers/MLMG/AMReX_MLNodeLinOp.cpp commit 5b235f13980b2602636eff689f95d96a99c75d06 Author: Weiqun Zhang Date: Wed Mar 17 10:06:45 2021 -0700 LevelBld (#1852) In Amr class, we need to get LevelBld* and use it to build AmrLevels. Previously, Amr called a function `LevelBld* getLevelBld()` to get that. The problem was that `getLevelBld` was not defined by amrex but it was being called by amrex. This caused issues in building amrex as a shared library on Mac and Windows, because they do not support weak symbols. In this PR, the constructors of Amr now take `LevelBld*` as an argument. This is unfortunately a breaking change. However, it's a compile time failure and it's easy to fix. See the changes in this commit for examples. Src/Amr/AMReX_Amr.H Src/Amr/AMReX_Amr.cpp Src/Amr/AMReX_LevelBld.H Tutorials/Amr/Advection_AmrLevel/Source/main.cpp Tutorials/EB/CNS/Source/main.cpp Tutorials/GPU/CNS/Source/main.cpp Tutorials/GPU/EBCNS/Source/main.cpp commit 5d32c74750b59e06c5731e084a75eec6ab91aad9 Author: drangara <69211175+drangara@users.noreply.github.com> Date: Wed Mar 17 11:24:36 2021 -0400 Fix bug with face-centroid slopes when using multigrid (#1871) When computing the Least Square slopes on face-centroid stencil with multigrid, we need to grow `m_eb_phi` and make additional checks. This adds to the changes made in https://github.com/AMReX-Codes/amrex/pull/1707 Src/EB/AMReX_EB_LeastSquares_2D_K.H Src/EB/AMReX_EB_LeastSquares_3D_K.H Src/LinearSolvers/MLMG/AMReX_MLEBABecLap.cpp Tests/LinearSolvers/LeastSquares/inputs.2d.askew-x.mg Tests/LinearSolvers/LeastSquares/inputs.2d.askew-y.mg Tests/LinearSolvers/LeastSquares/inputs.3d.poiseuille.askew-all.mg commit c28d91c2ee07ad6f970eb10bbc14d00af6adab4a Author: Weiqun Zhang Date: Tue Mar 16 14:59:05 2021 -0700 Avoid conflict with X11 (#1872) Bool -> B to avoid a conflict with an X11 header that defines ``` #define Bool int #define Status int #define True 1 #define False 0 ``` This should fix the compilation issue of Amrvis. Src/Base/AMReX_TypeTraits.H commit 44f77978de2bdaec40cc6136169b854a78ff167e Author: Andrew Myers Date: Tue Mar 16 11:23:18 2021 -0400 Add explanation of parameters to Redistribute docstring (#1868) Src/Particle/AMReX_Particles.H commit e1b123b2302c0377efb9ab99066a60f907c663c9 Author: Weiqun Zhang Date: Mon Mar 15 18:05:17 2021 -0700 Fix sign-compare and shadow warnings (#1867) The warning about shadowed varible in AMReX_NonLocalBC.H seems a compiler bug. Nevertheless, the source is modified to get rid of the warning. Src/Base/AMReX_NonLocalBC.H Src/Particle/AMReX_ParticleCommunication.H Src/Particle/AMReX_ParticleContainerI.H commit 7d833e64ff06477953c9b3a2642b04e61116be93 Author: Weiqun Zhang Date: Mon Mar 15 12:42:05 2021 -0700 EditorConfig and trailing whitespaces (#1866) Trim trailing whitespaces in editorconfig for C/C++ codes and others. Allow md file to have trailing whitespaces because they are syntactically significant. .editorconfig .github/workflows/style/check_trailing_whitespaces.sh commit e5b06d59e3a6cf55c40c6498e42eafe940203cb9 Author: Weiqun Zhang Date: Mon Mar 15 12:28:06 2021 -0700 Optimization of CArena::freeUnused (#1865) Use `find` instead of manully looping over the set. Src/Base/AMReX_CArena.cpp commit fbef03f566fbbe3c583bd96db00b5cee391ad6b8 Author: Axel Huebl Date: Mon Mar 15 12:25:52 2021 -0700 CI: macOS Shared Lib (#1836) ## Summary CI for macOS: Build a shared library. This is the default in Spack. I see issues with AppleClang 12.0 raising: ``` ld: can't use -undefined warning or suppress with -twolevel_namespace clang: error: linker command failed with exit code 1 (use -v to see invocation) ``` @RemiLehe reported this again here: https://github.com/AMReX-Codes/amrex/issues/425#issuecomment-473153152 - [x] reproduce issue - [x] add fix ## Additional background Introduced in 1b6af3cb630 to fix #425 .github/workflows/dependencies/dependencies_mac.sh .github/workflows/macos.yml Src/Amr/AMReX_LevelBld.H Src/Amr/AMReX_PROB_AMR_F.H Src/Base/AMReX_Extension.H Tools/CMake/AMReX_Config.cmake commit 2789d017a7d1e4803fe9ca007dbbad611e42069f Author: Weiqun Zhang Date: Sun Mar 14 14:39:58 2021 -0700 Fix bug in #1861 (#1864) We cannot simply use std::set::erase on the free node list because the operator== only compares the starting block address. Src/Base/AMReX_CArena.cpp commit 107b5d4f36063b110539242928faba914f07e5b3 Author: Weiqun Zhang Date: Sun Mar 14 14:25:49 2021 -0700 Fix CUDA API calls in host callback functions (#1863) CUDA API calls are not allowed in host callback functions. Therefore, we have to move the call to free unused memory from `free` to `alloc`. Src/Base/AMReX_CArena.cpp commit d28caf8685f6c65639aee1d7d1731901480b3a14 Author: Weiqun Zhang Date: Sun Mar 14 14:19:48 2021 -0700 CI for Code Style (#1862) Check for tabs and trailing whitespaces at the end of line in CI. .github/workflows/style.yml .github/workflows/style/check_tabs.sh .github/workflows/style/check_trailing_whitespaces.sh .gitignore commit 486cd438aea970962a02c799c910eb8ddcd7dd71 Author: CCSE@LBNL Date: Sun Mar 14 08:38:31 2021 -0700 Remove tabs and trailing white spaces (#1860) The changes are purely white spaces only. `git diff -w development..HEAD` shows nothing. It was done with ``` find . -type d \( -name .git \ -o -path ./paper \ \) -prune -o \ -type f \( -name "*.H" -o -name "*.h" -o -name "*.hh" -o -name "*.hpp" \ -o -name "*.c" -o -name "*.cc" -o -name "*.cpp" -o -name "*.cxx" \ -o -name "*.f" -o -name "*.F" -o -name "*.f90" -o -name "*.F90" \ -o -name "*.py" \ -o -name "*.md" -o -name "*.rst" \ -o -name "*.sh" \ -o -name "*.tex" \ -o -name "*.txt" \ \) \ -exec grep -Iq . {} \; \ -exec sed -i 's/[[:blank:]]\+$//g' {} + \ -exec sed -i 's/\t/\ \ \ \ \ \ \ \ /g' {} + ``` We will add a CI check as a follow-up. CONTRIBUTING.md Docs/Migration/Migration.md Docs/sphinx_documentation/source/AMReX_Profiling_Tools_Chapter.rst Docs/sphinx_documentation/source/AmrCore.rst Docs/sphinx_documentation/source/AmrLevel.rst Docs/sphinx_documentation/source/Basics.rst Docs/sphinx_documentation/source/DualGrid.rst Docs/sphinx_documentation/source/EB.rst Docs/sphinx_documentation/source/External_Profiling_Tools.rst Docs/sphinx_documentation/source/Fortran.rst Docs/sphinx_documentation/source/GPU.rst Docs/sphinx_documentation/source/GPU_Chapter.rst Docs/sphinx_documentation/source/GettingStarted.rst Docs/sphinx_documentation/source/GridCreation.rst Docs/sphinx_documentation/source/IO.rst Docs/sphinx_documentation/source/InputsLoadBalancing.rst Docs/sphinx_documentation/source/Inputs_Chapter.rst Docs/sphinx_documentation/source/Introduction.rst Docs/sphinx_documentation/source/LinearSolvers.rst Docs/sphinx_documentation/source/LinearSolvers_Chapter.rst Docs/sphinx_documentation/source/LoadBalancing.rst Docs/sphinx_documentation/source/ManagingGridHierarchy_Chapter.rst Docs/sphinx_documentation/source/Particle.rst Docs/sphinx_documentation/source/SWFFT.rst Docs/sphinx_documentation/source/Testing.rst Docs/sphinx_documentation/source/Visualization.rst Docs/sphinx_documentation/source/index.rst Docs/sphinx_tutorials/source/AMR_Tutorial.rst Docs/sphinx_tutorials/source/Basic_Tutorial.rst Docs/sphinx_tutorials/source/Blueprint_Tutorial.rst Docs/sphinx_tutorials/source/GPU_Tutorial.rst Docs/sphinx_tutorials/source/LinearSolvers_Tutorial.rst Docs/sphinx_tutorials/source/MUI_Tutorial.rst Docs/sphinx_tutorials/source/Particles_Tutorial.rst Docs/sphinx_tutorials/source/index.rst README.md Src/Amr/AMReX_Amr.H Src/Amr/AMReX_Amr.cpp Src/Amr/AMReX_AmrLevel.H Src/Amr/AMReX_AmrLevel.cpp Src/Amr/AMReX_AuxBoundaryData.H Src/Amr/AMReX_AuxBoundaryData.cpp Src/Amr/AMReX_Derive.H Src/Amr/AMReX_Derive.cpp Src/Amr/AMReX_StateData.H Src/Amr/AMReX_StateData.cpp Src/Amr/AMReX_StateDescriptor.H Src/Amr/AMReX_StateDescriptor.cpp Src/Amr/AMReX_extrapolater_3D_K.H Src/AmrCore/AMReX_AmrCore.cpp Src/AmrCore/AMReX_AmrMesh.cpp Src/AmrCore/AMReX_AmrParGDB.H Src/AmrCore/AMReX_AmrParticles.H Src/AmrCore/AMReX_Cluster.cpp Src/AmrCore/AMReX_ErrorList.H Src/AmrCore/AMReX_ErrorList.cpp Src/AmrCore/AMReX_FillPatchUtil.H Src/AmrCore/AMReX_FillPatchUtil_2d.F90 Src/AmrCore/AMReX_FillPatchUtil_3d.F90 Src/AmrCore/AMReX_FillPatchUtil_F.H Src/AmrCore/AMReX_FillPatchUtil_I.H Src/AmrCore/AMReX_FluxRegister.cpp Src/AmrCore/AMReX_INTERP_1D.F90 Src/AmrCore/AMReX_INTERP_2D.F90 Src/AmrCore/AMReX_INTERP_F.H Src/AmrCore/AMReX_Interp_1D_C.H Src/AmrCore/AMReX_Interp_2D_C.H Src/AmrCore/AMReX_Interp_3D_C.H Src/AmrCore/AMReX_Interpolater.H Src/AmrCore/AMReX_Interpolater.cpp Src/AmrCore/AMReX_TagBox.cpp Src/Base/AMReX.H Src/Base/AMReX.cpp Src/Base/AMReX_BCRec.cpp Src/Base/AMReX_BLBackTrace.cpp Src/Base/AMReX_BLProfiler.H Src/Base/AMReX_BLProfiler.cpp Src/Base/AMReX_BLutil_F.F90 Src/Base/AMReX_BaseFab.H Src/Base/AMReX_BaseFab.cpp Src/Base/AMReX_BaseUmap_f.H Src/Base/AMReX_BlockMutex.H Src/Base/AMReX_BlockMutex.cpp Src/Base/AMReX_Box.H Src/Base/AMReX_Box.cpp Src/Base/AMReX_BoxArray.H Src/Base/AMReX_BoxArray.cpp Src/Base/AMReX_BoxDomain.cpp Src/Base/AMReX_BoxList.H Src/Base/AMReX_BoxList.cpp Src/Base/AMReX_COORDSYS_1D_C.H Src/Base/AMReX_COORDSYS_2D_C.H Src/Base/AMReX_CoordSys.cpp Src/Base/AMReX_CudaGraph.H Src/Base/AMReX_CuptiTrace.H Src/Base/AMReX_CuptiTrace.cpp Src/Base/AMReX_DistributionMapping.H Src/Base/AMReX_DistributionMapping.cpp Src/Base/AMReX_Extension.H Src/Base/AMReX_FACopyDescriptor.H Src/Base/AMReX_FArrayBox.H Src/Base/AMReX_FArrayBox.cpp Src/Base/AMReX_FBI.H Src/Base/AMReX_FILCC_1D.F90 Src/Base/AMReX_FILCC_2D.F90 Src/Base/AMReX_FILCC_3D.F90 Src/Base/AMReX_FPC.cpp Src/Base/AMReX_FabArray.H Src/Base/AMReX_FabArrayBase.H Src/Base/AMReX_FabArrayBase.cpp Src/Base/AMReX_FabArrayCommI.H Src/Base/AMReX_FabArrayUtility.H Src/Base/AMReX_FabConv.cpp Src/Base/AMReX_FileSystem.cpp Src/Base/AMReX_Geometry.H Src/Base/AMReX_Geometry.cpp Src/Base/AMReX_GpuAllocators.H Src/Base/AMReX_GpuComplex.H Src/Base/AMReX_GpuContainers.H Src/Base/AMReX_GpuDevice.cpp Src/Base/AMReX_GpuLaunch.H Src/Base/AMReX_GpuMemory.H Src/Base/AMReX_GpuUtility.H Src/Base/AMReX_IArrayBox.H Src/Base/AMReX_IArrayBox.cpp Src/Base/AMReX_INT.H Src/Base/AMReX_IntConv.H Src/Base/AMReX_IntConv.cpp Src/Base/AMReX_IntVect.H Src/Base/AMReX_Lazy.H Src/Base/AMReX_Lazy.cpp Src/Base/AMReX_MFCopyDescriptor.cpp Src/Base/AMReX_MFIter.H Src/Base/AMReX_MFIter.cpp Src/Base/AMReX_Machine.cpp Src/Base/AMReX_MemPool.cpp Src/Base/AMReX_MemProfiler.H Src/Base/AMReX_MemProfiler.cpp Src/Base/AMReX_MultiFab.H Src/Base/AMReX_MultiFab.cpp Src/Base/AMReX_MultiFabUtil.cpp Src/Base/AMReX_MultiFabUtil_1D_C.H Src/Base/AMReX_NFiles.H Src/Base/AMReX_NFiles.cpp Src/Base/AMReX_NonLocalBC.H Src/Base/AMReX_NonLocalBC.cpp Src/Base/AMReX_NonLocalBCImpl.H Src/Base/AMReX_PODVector.H Src/Base/AMReX_ParallelContext.cpp Src/Base/AMReX_ParallelDescriptor.H Src/Base/AMReX_ParallelDescriptor.cpp Src/Base/AMReX_ParallelDescriptor_F.F90 Src/Base/AMReX_ParmParse.H Src/Base/AMReX_ParmParse.cpp Src/Base/AMReX_Partition.H Src/Base/AMReX_Periodicity.H Src/Base/AMReX_Periodicity.cpp Src/Base/AMReX_PlotFileUtil.H Src/Base/AMReX_PlotFileUtil.cpp Src/Base/AMReX_Print.H Src/Base/AMReX_REAL.H Src/Base/AMReX_RealBox.cpp Src/Base/AMReX_RealVect.H Src/Base/AMReX_Reduce.H Src/Base/AMReX_SPACE.H Src/Base/AMReX_Scan.H Src/Base/AMReX_Slopes_K.H Src/Base/AMReX_ThirdPartyProfiling.H Src/Base/AMReX_TinyProfiler.H Src/Base/AMReX_TinyProfiler.cpp Src/Base/AMReX_TypeTraits.H Src/Base/AMReX_Utility.H Src/Base/AMReX_Utility.cpp Src/Base/AMReX_Vector.H Src/Base/AMReX_VectorIO.H Src/Base/AMReX_VectorIO.cpp Src/Base/AMReX_VisMF.H Src/Base/AMReX_VisMF.cpp Src/Base/AMReX_bc_types_mod.F90 Src/Base/AMReX_ccse-mpi.H Src/Base/AMReX_error_fi.cpp Src/Base/AMReX_error_mod.F90 Src/Base/AMReX_filcc_mod.F90 Src/Base/AMReX_iMultiFab.H Src/Base/AMReX_iMultiFab.cpp Src/Base/AMReX_mempool_mod.F90 Src/Base/AMReX_parmparse_fi.cpp Src/Base/AMReX_parmparse_mod.F90 Src/Boundary/AMReX_BndryData.H Src/Boundary/AMReX_BndryData.cpp Src/Boundary/AMReX_BndryRegister.H Src/Boundary/AMReX_BndryRegister.cpp Src/Boundary/AMReX_FabSet.H Src/Boundary/AMReX_FabSet.cpp Src/Boundary/AMReX_InterpBndryData.H Src/Boundary/AMReX_InterpBndryData.cpp Src/Boundary/AMReX_MacBndry.H Src/Boundary/AMReX_MacBndry.cpp Src/Boundary/AMReX_Mask.H Src/Boundary/AMReX_MultiMask.H Src/Boundary/AMReX_MultiMask.cpp Src/Boundary/AMReX_YAFluxRegister.cpp Src/EB/AMReX_EB2_3D_C.H Src/EB/AMReX_EB2_IF_Complement.H Src/EB/AMReX_EB2_IF_Lathe.H Src/EB/AMReX_EB2_IF_Sphere.H Src/EB/AMReX_EB2_Level.cpp Src/EB/AMReX_EBFluxRegister.cpp Src/EB/AMReX_EBInterpolater.cpp Src/EB/AMReX_EBMultiFabUtil.H Src/EB/AMReX_EBMultiFabUtil.cpp Src/EB/AMReX_EBMultiFabUtil_2D_C.H Src/EB/AMReX_EBMultiFabUtil_3D_C.H Src/EB/AMReX_EBToPVD.H Src/EB/AMReX_EBToPVD.cpp Src/EB/AMReX_EB_LeastSquares_2D_K.H Src/EB/AMReX_EB_LeastSquares_3D_K.H Src/EB/AMReX_EB_STL_utils.H Src/EB/AMReX_EB_STL_utils.cpp Src/EB/AMReX_EB_slopes_K.H Src/EB/AMReX_EB_triGeomOps_K.H Src/EB/AMReX_MultiCutFab.H Src/EB/AMReX_WriteEBSurface.H Src/EB/AMReX_WriteEBSurface.cpp Src/EB/AMReX_ebcellflag_mod.F90 Src/Extern/Conduit/AMReX_Conduit_Blueprint_ParticlesI.H Src/Extern/HYPRE/AMReX_Hypre.H Src/Extern/HYPRE/AMReX_Hypre.cpp Src/Extern/HYPRE/AMReX_HypreABecLap.H Src/Extern/HYPRE/AMReX_HypreABecLap.cpp Src/Extern/HYPRE/AMReX_HypreABecLap2.H Src/Extern/HYPRE/AMReX_HypreABecLap2.cpp Src/Extern/HYPRE/AMReX_HypreABecLap3.H Src/Extern/HYPRE/AMReX_HypreABecLap3.cpp Src/Extern/PETSc/AMReX_PETSc.H Src/Extern/PETSc/AMReX_PETSc.cpp Src/Extern/ProfParser/AMReX_AVGDOWN_1D.F Src/Extern/ProfParser/AMReX_AVGDOWN_2D.F Src/Extern/ProfParser/AMReX_AVGDOWN_3D.F Src/Extern/ProfParser/AMReX_AVGDOWN_F.H Src/Extern/ProfParser/AMReX_BLProfStats.H Src/Extern/ProfParser/AMReX_BLProfStats.cpp Src/Extern/ProfParser/AMReX_BLProfUtilities.H Src/Extern/ProfParser/AMReX_BLProfUtilities.cpp Src/Extern/ProfParser/AMReX_BLWritePlotFile.H Src/Extern/ProfParser/AMReX_BLWritePlotFile.cpp Src/Extern/ProfParser/AMReX_CommProfStats.H Src/Extern/ProfParser/AMReX_CommProfStats.cpp Src/Extern/ProfParser/AMReX_ProfParserBatch.cpp Src/Extern/ProfParser/AMReX_RegionsProfStats.H Src/Extern/ProfParser/AMReX_RegionsProfStats.cpp Src/Extern/SWFFT/AlignedAllocator.h Src/Extern/SWFFT/CheckDecomposition.c Src/Extern/SWFFT/Dfft.H Src/Extern/SWFFT/DfftC.cpp Src/Extern/SWFFT/Distribution.H Src/Extern/SWFFT/DistributionC.cpp Src/Extern/SWFFT/Error.h Src/Extern/SWFFT/TimingStats.h Src/Extern/SWFFT/complex-type.h Src/Extern/SWFFT/distribution.c Src/Extern/SWFFT/distribution_c.h Src/Extern/amrdata/AMReX_AmrData.H Src/Extern/amrdata/AMReX_AmrData.cpp Src/Extern/amrdata/AMReX_DataServices.H Src/Extern/amrdata/AMReX_DataServices.cpp Src/Extern/amrdata/AMReX_FABUTIL_1D.F Src/Extern/amrdata/AMReX_FABUTIL_2D.F Src/Extern/amrdata/AMReX_FABUTIL_3D.F Src/Extern/amrdata/AMReX_XYPlotDataList.H Src/Extern/amrdata/AMReX_XYPlotDataList.cpp Src/Extern/amrdata/CMakeLists.txt Src/Extern/hpgmg/BL_HPGMG.H Src/Extern/hpgmg/BL_HPGMG.cpp Src/F_Interfaces/AmrCore/AMReX_FAmrCore.H Src/F_Interfaces/AmrCore/AMReX_FAmrCore.cpp Src/F_Interfaces/AmrCore/AMReX_amrcore_fi.cpp Src/F_Interfaces/AmrCore/AMReX_amrcore_mod.F90 Src/F_Interfaces/AmrCore/AMReX_fillpatch_fi.cpp Src/F_Interfaces/AmrCore/AMReX_fillpatch_mod.F90 Src/F_Interfaces/AmrCore/AMReX_fluxregister_fi.cpp Src/F_Interfaces/AmrCore/AMReX_fluxregister_mod.F90 Src/F_Interfaces/AmrCore/AMReX_tagbox_fi.cpp Src/F_Interfaces/AmrCore/AMReX_tagbox_mod.F90 Src/F_Interfaces/Base/AMReX_FPhysBC.cpp Src/F_Interfaces/Base/AMReX_box_fi.cpp Src/F_Interfaces/Base/AMReX_box_mod.F90 Src/F_Interfaces/Base/AMReX_boxarray_fi.cpp Src/F_Interfaces/Base/AMReX_distromap_fi.cpp Src/F_Interfaces/Base/AMReX_distromap_mod.F90 Src/F_Interfaces/Base/AMReX_geometry_fi.cpp Src/F_Interfaces/Base/AMReX_geometry_mod.F90 Src/F_Interfaces/Base/AMReX_init_mod.F90 Src/F_Interfaces/Base/AMReX_multifab_fi.cpp Src/F_Interfaces/Base/AMReX_multifab_mod.F90 Src/F_Interfaces/Base/AMReX_multifabutil_fi.cpp Src/F_Interfaces/Base/AMReX_parallel_mod.F90 Src/F_Interfaces/Base/AMReX_physbc_fi.cpp Src/F_Interfaces/Base/AMReX_physbc_mod.F90 Src/F_Interfaces/Base/AMReX_plotfile_fi.cpp Src/F_Interfaces/Base/AMReX_vismf_fi.cpp Src/F_Interfaces/LinearSolvers/AMReX_linop_mod.F90 Src/F_Interfaces/LinearSolvers/AMReX_multigrid_fi.cpp Src/F_Interfaces/LinearSolvers/AMReX_multigrid_mod.F90 Src/F_Interfaces/LinearSolvers/AMReX_poisson_mod.F90 Src/F_Interfaces/Octree/AMReX_octree_fi.cpp Src/F_Interfaces/Particle/AMReX_particlecontainer_fi.cpp Src/LinearSolvers/MLMG/AMReX_MLABecLap_2D_K.H Src/LinearSolvers/MLMG/AMReX_MLABecLap_3D_K.H Src/LinearSolvers/MLMG/AMReX_MLABecLaplacian.cpp Src/LinearSolvers/MLMG/AMReX_MLALap_1D_K.H Src/LinearSolvers/MLMG/AMReX_MLALaplacian.cpp Src/LinearSolvers/MLMG/AMReX_MLCGSolver.H Src/LinearSolvers/MLMG/AMReX_MLCGSolver.cpp Src/LinearSolvers/MLMG/AMReX_MLCellABecLap.cpp Src/LinearSolvers/MLMG/AMReX_MLCellLinOp.cpp Src/LinearSolvers/MLMG/AMReX_MLEBABecLap.H Src/LinearSolvers/MLMG/AMReX_MLEBABecLap.cpp Src/LinearSolvers/MLMG/AMReX_MLEBABecLap_2D_K.H Src/LinearSolvers/MLMG/AMReX_MLEBABecLap_3D_K.H Src/LinearSolvers/MLMG/AMReX_MLEBTensorOp.cpp Src/LinearSolvers/MLMG/AMReX_MLEBTensor_2D_K.H Src/LinearSolvers/MLMG/AMReX_MLEBTensor_3D_K.H Src/LinearSolvers/MLMG/AMReX_MLLinOp.H Src/LinearSolvers/MLMG/AMReX_MLLinOp.cpp Src/LinearSolvers/MLMG/AMReX_MLLinOp_nd.F90 Src/LinearSolvers/MLMG/AMReX_MLMG.cpp Src/LinearSolvers/MLMG/AMReX_MLMGBndry.cpp Src/LinearSolvers/MLMG/AMReX_MLNodeLap_2D_K.H Src/LinearSolvers/MLMG/AMReX_MLNodeLap_3D_K.H Src/LinearSolvers/MLMG/AMReX_MLNodeLaplacian.cpp Src/LinearSolvers/MLMG/AMReX_MLPoisson.cpp Src/LinearSolvers/MLMG/AMReX_MLPoisson_1D_K.H Src/LinearSolvers/MLMG/AMReX_MLTensorOp.H Src/LinearSolvers/MLMG/AMReX_MLTensorOp.cpp Src/LinearSolvers/MLMG/AMReX_MLTensor_3D_K.H Src/LinearSolvers/Projections/AMReX_MacProjector.H Src/LinearSolvers/Projections/AMReX_NodalProjector.cpp Src/Particle/AMReX_ArrayOfStructs.H Src/Particle/AMReX_BinIterator.H Src/Particle/AMReX_NeighborParticlesGPUImpl.H Src/Particle/AMReX_ParGDB.H Src/Particle/AMReX_Particle.H Src/Particle/AMReX_ParticleCommunication.cpp Src/Particle/AMReX_ParticleContainerBase.H Src/Particle/AMReX_ParticleContainerI.H Src/Particle/AMReX_ParticleHDF5.H Src/Particle/AMReX_ParticleIO.H Src/Particle/AMReX_ParticleInit.H Src/Particle/AMReX_ParticleMPIUtil.H Src/Particle/AMReX_ParticleMesh.H Src/Particle/AMReX_ParticleReduce.H Src/Particle/AMReX_ParticleTransformation.H Src/Particle/AMReX_ParticleUtil.H Src/Particle/AMReX_Particle_mod_K.H Src/Particle/AMReX_Particles.H Src/Particle/AMReX_SparseBins.H Src/Particle/AMReX_TracerParticle_mod_K.H Src/Particle/AMReX_TracerParticles.H Src/Particle/AMReX_TracerParticles.cpp Src/SDC/AMReX_SDCquadrature.F90 Src/SDC/AMReX_SDCstruct.H Src/SDC/AMReX_SDCstruct.cpp Tests/AsyncOut/multifab/main.cpp Tests/DivFreePatch/main.cpp Tests/FillBoundaryComparison/main.cpp Tests/GPU/AnyOf/main.cpp Tests/GPU/Fuse/main.cpp Tests/GPU/Vector/main.cpp Tests/HDF5Benchmark/main.cpp Tests/LinearSolvers/ABecLaplacian_C/MyTest.H Tests/LinearSolvers/ABecLaplacian_C/MyTest.cpp Tests/LinearSolvers/ABecLaplacian_C/initProb.cpp Tests/LinearSolvers/ABecLaplacian_C/initProb_K.H Tests/LinearSolvers/ABecLaplacian_C/scalingtest/run-16.sh Tests/LinearSolvers/ABecLaplacian_C/scalingtest/run-32.sh Tests/LinearSolvers/ABecLaplacian_C/scalingtest/run-8.sh Tests/LinearSolvers/ABecLaplacian_C/threadmultiple_test/knl-mpi-1.sh Tests/LinearSolvers/ABecLaplacian_C/threadmultiple_test/knl-mpi-16.sh Tests/LinearSolvers/ABecLaplacian_C/threadmultiple_test/knl-mpi-2.sh Tests/LinearSolvers/ABecLaplacian_C/threadmultiple_test/knl-mpi-32.sh Tests/LinearSolvers/ABecLaplacian_C/threadmultiple_test/knl-mpi-4.sh Tests/LinearSolvers/ABecLaplacian_C/threadmultiple_test/knl-mpi-64.sh Tests/LinearSolvers/ABecLaplacian_C/threadmultiple_test/knl-mpi-8.sh Tests/LinearSolvers/ABecLaplacian_C/threadmultiple_test/knl-omp-1.sh Tests/LinearSolvers/ABecLaplacian_C/threadmultiple_test/knl-omp-16.sh Tests/LinearSolvers/ABecLaplacian_C/threadmultiple_test/knl-omp-2.sh Tests/LinearSolvers/ABecLaplacian_C/threadmultiple_test/knl-omp-32.sh Tests/LinearSolvers/ABecLaplacian_C/threadmultiple_test/knl-omp-4.sh Tests/LinearSolvers/ABecLaplacian_C/threadmultiple_test/knl-omp-64.sh Tests/LinearSolvers/ABecLaplacian_C/threadmultiple_test/knl-omp-8.sh Tests/LinearSolvers/ABecLaplacian_F/init_prob.F90 Tests/LinearSolvers/ABecLaplacian_F/mytest.F90 Tests/LinearSolvers/CellEB/MyTest.H Tests/LinearSolvers/CellEB/MyTest.cpp Tests/LinearSolvers/CellEB/initEB.cpp Tests/LinearSolvers/CellEB2/MyTest.H Tests/LinearSolvers/CellEB2/MyTest.cpp Tests/LinearSolvers/CellEB2/initEB.cpp Tests/LinearSolvers/EBConvergenceTest/BC_2D.F90 Tests/LinearSolvers/EBConvergenceTest/BC_3D.F90 Tests/LinearSolvers/EBConvergenceTest/Convergence_Data_Gen.sh Tests/LinearSolvers/EBConvergenceTest/MyTest.H Tests/LinearSolvers/EBConvergenceTest/MyTest.cpp Tests/LinearSolvers/EBConvergenceTest/MyTest_F.H Tests/LinearSolvers/EBConvergenceTest/RHS.F90 Tests/LinearSolvers/EBConvergenceTest/acoef.F90 Tests/LinearSolvers/EBConvergenceTest/bcoef.F90 Tests/LinearSolvers/EBTensor/MyTest.cpp Tests/LinearSolvers/EBTensor/MyTestPlotfile.cpp Tests/LinearSolvers/EBTensor/MyTest_2D_K.H Tests/LinearSolvers/EBflux_grad/MyTest.H Tests/LinearSolvers/EBflux_grad/MyTest.cpp Tests/LinearSolvers/EBflux_grad/initEB.cpp Tests/LinearSolvers/LeastSquares/MyTest.cpp Tests/LinearSolvers/LeastSquares/README.md Tests/LinearSolvers/LeastSquares/initData.cpp Tests/LinearSolvers/LeastSquares/initEB.cpp Tests/LinearSolvers/LeastSquares/initPoiseuilleData.cpp Tests/LinearSolvers/MAC_Projection_EB/main.cpp Tests/LinearSolvers/MLMG/fort_3d.F90 Tests/LinearSolvers/MLMG/init_prob.cpp Tests/LinearSolvers/MLMG/main.cpp Tests/LinearSolvers/MultiComponent/MCNodalLinOp.H Tests/LinearSolvers/MultiComponent/MCNodalLinOp.cpp Tests/LinearSolvers/MultiComponent/main.cpp Tests/LinearSolvers/Nodal_Projection_EB/main.cpp Tests/LinearSolvers/NodeEB/MyTest.cpp Tests/MultiBlock/Advection/main.cpp Tests/MultiBlock/IndexType/main.cpp Tests/Particles/AssignDensity/main.cpp Tests/Particles/AssignMultiLevelDensity/main.cpp Tests/Particles/AsyncIO/main.cpp Tests/Particles/GhostsAndVirtuals/main.cpp Tests/Particles/InitFromAscii/main.cpp Tests/Particles/Intersection/main.cpp Tests/Particles/NeighborParticles/CheckPair.H Tests/Particles/NeighborParticles/Constants.H Tests/Particles/NeighborParticles/MDParticleContainer.cpp Tests/Particles/NeighborParticles/README.md Tests/Particles/ParticleIterator/main.cpp Tests/Particles/ParticleMesh/main.cpp Tests/Particles/ParticleTransformations/main.cpp Tests/Particles/SparseBins/main.cpp Tests/Particles/TypeDescriptor/main.cpp Tools/AMRProfParser/TestCodes/AMRProfTest0.cpp Tools/AMRProfParser/TestCodes/ProfWaitTest.cpp Tools/AMRProfParser/TestCodes/SendTest0.cpp Tools/AMRProfParser/TestCodes/TokenizeTest.cpp Tools/C_util/AmrDeriveTecplot/AmrDeriveTecplot.cpp Tools/C_util/AppendToPlotFile.cpp Tools/C_util/AugmentPlotfile/AugmentPlotfile.cpp Tools/C_util/AugmentPlotfile/AugmentPlotfile_F.H Tools/C_util/Convergence/Add.cpp Tools/C_util/Convergence/ComparePlotfiles.cpp Tools/C_util/Convergence/ComputeAmrDataNorms.H Tools/C_util/Convergence/ComputeAmrDataNorms.cpp Tools/C_util/Convergence/DebugDump.H Tools/C_util/Convergence/DebugOut.H Tools/C_util/Convergence/DebugOut.cpp Tools/C_util/Convergence/DiffFab.cpp Tools/C_util/Convergence/DiffSameDomainRefined.cpp Tools/C_util/Convergence/DiffSameDomainRefinedComposite.cpp Tools/C_util/Convergence/DiffSameDomainRefinedFD.cpp Tools/C_util/Convergence/DiffSameDomainRefinedStag.cpp Tools/C_util/Convergence/DiffSameGrid.cpp Tools/C_util/Convergence/DiffSameGrid2.cpp Tools/C_util/Convergence/DiffSameGridRefined.cpp Tools/C_util/Convergence/DiffUniform.cpp Tools/C_util/Convergence/PltFileNorm.cpp Tools/C_util/Convergence/PltFileNormB.cpp Tools/C_util/Convergence/PltFileScalConvRate.cpp Tools/C_util/Convergence/RichardsonConvergenceTest.cpp Tools/C_util/DiffMultiFab/diffmultifab.cpp Tools/C_util/Statistics/AVGDOWN_2D.F Tools/C_util/Statistics/AVGDOWN_3D.F Tools/C_util/Statistics/AVGDOWN_F.H Tools/C_util/Statistics/ComputeAmrDataStat.H Tools/C_util/Statistics/ComputeAmrDataStat.cpp Tools/C_util/Statistics/PltFileFluxAve.H Tools/C_util/Statistics/PltFileFluxAve.cpp Tools/C_util/Statistics/PltFileList.cpp Tools/C_util/Statistics/PltFileStat.cpp Tools/C_util/Statistics/PltFileXAve.H Tools/C_util/Statistics/PltFileXAve.cpp Tools/C_util/TV_TempWrite.H Tools/C_util/ViewMF/MFNorm.cpp Tools/C_util/ViewMF/checkMFghostcells.cpp Tools/C_util/ViewMF/main.cpp Tools/C_util/ViewMF/mfMinMax.cpp Tools/C_util/ViewMF/viewMF.cpp Tools/C_util/ViewMF/viewMFcol.cpp Tools/C_util/ViewMF/viewMFdiff.cpp Tools/C_util/ViewMF/viewMFdiffHardWire.cpp Tools/C_util/WritePlotFile.H Tools/C_util/WritePlotFile.cpp Tools/C_util/dbgTools/crsGrids.cpp Tools/C_util/dbgTools/intersectGrids.cpp Tools/CompileTesting/compiletesting.py Tools/EBSurfaceTools/ConvertEBSurface.cpp Tools/EBSurfaceTools/isoToVTK.py Tools/F_scripts/fcheck.py Tools/GNUMake/README.md Tools/Plotfile/AMReX_PPMUtil.cpp Tools/Plotfile/fcompare.cpp Tools/Postprocessing/C_Src/HorizontalAvg.cpp Tools/Postprocessing/C_Src/IntegrateComp.cpp Tools/Postprocessing/C_Src/MultiFabToMatLab.cpp Tools/Postprocessing/C_Src/PlotfileToMatLab.cpp Tools/Postprocessing/C_Src/PlotfileToTurb.cpp Tools/Postprocessing/C_Src/PtwisePltTransform.cpp Tools/Postprocessing/C_Src/PtwisePltTransform_nd.f90 Tools/Postprocessing/C_Src/WritePlotfileToASCII.cpp Tools/Postprocessing/C_Src/cube_extract.cpp Tools/Postprocessing/C_Src/particle_compare.cpp Tools/Postprocessing/python/column_depth.py Tools/Postprocessing/python/conv_slopes.py Tools/Postprocessing/python/dumpparthistory.py Tools/Postprocessing/python/eos_data.txt Tools/Postprocessing/python/helmeos.py Tools/Postprocessing/python/parseparticles.py Tools/Postprocessing/python/test_helmeos.py Tools/Postprocessing/python/test_parseparticles.py Tools/Py_util/amrex_particles_to_vtp/amrex_binary_particles_to_vtp.py Tools/Py_util/amrex_particles_to_vtp/write_pview_file.py Tools/Py_util/plotsinglevar.py Tools/Release/ppCleanup.py Tools/Release/ppCleanupDir.py Tools/Release/release.py Tools/typechecker/typechecker.py Tutorials/Amr/Advection_AmrCore/Exec/Prob.H Tutorials/Amr/Advection_AmrCore/Exec/paraview_amr101.py Tutorials/Amr/Advection_AmrCore/README.md Tutorials/Amr/Advection_AmrCore/Source/AdvancePhiAllLevels.cpp Tutorials/Amr/Advection_AmrCore/Source/AdvancePhiAtLevel.cpp Tutorials/Amr/Advection_AmrCore/Source/AmrCoreAdv.H Tutorials/Amr/Advection_AmrCore/Source/AmrCoreAdv.cpp Tutorials/Amr/Advection_AmrCore/Source/DefineVelocity.cpp Tutorials/Amr/Advection_AmrCore/Source/Src_K/Adv_K.H Tutorials/Amr/Advection_AmrCore/Source/Src_K/compute_flux_3D_K.H Tutorials/Amr/Advection_AmrCore/Source/face_velocity.H Tutorials/Amr/Advection_AmrCore/Source/main.cpp Tutorials/Amr/Advection_AmrLevel/Exec/SingleVortex/Prob.f90 Tutorials/Amr/Advection_AmrLevel/Exec/SingleVortex/face_velocity_2d.f90 Tutorials/Amr/Advection_AmrLevel/Exec/SingleVortex/face_velocity_3d.f90 Tutorials/Amr/Advection_AmrLevel/Exec/UniformVelocity/Prob.f90 Tutorials/Amr/Advection_AmrLevel/Source/Adv_F.H Tutorials/Amr/Advection_AmrLevel/Source/AmrLevelAdv.H Tutorials/Amr/Advection_AmrLevel/Source/AmrLevelAdv.cpp Tutorials/Amr/Advection_AmrLevel/Source/LevelBldAdv.cpp Tutorials/Amr/Advection_AmrLevel/Source/Src_2d/Adv_2d.f90 Tutorials/Amr/Advection_AmrLevel/Source/Src_2d/compute_flux_2d.f90 Tutorials/Amr/Advection_AmrLevel/Source/Src_2d/slope_2d.f90 Tutorials/Amr/Advection_AmrLevel/Source/Src_3d/Adv_3d.f90 Tutorials/Amr/Advection_AmrLevel/Source/Src_3d/compute_flux_3d.f90 Tutorials/Amr/Advection_AmrLevel/Source/Src_3d/slope_3d.f90 Tutorials/Amr/Advection_AmrLevel/Source/Src_nd/Tagging_nd.f90 Tutorials/Amr/Advection_AmrLevel/Source/Src_nd/tagging_params.f90 Tutorials/Amr/Advection_AmrLevel/Source/main.cpp Tutorials/Basic/Build_with_libamrex/main.cpp Tutorials/Basic/Build_with_libamrex/test_parameters.cpp Tutorials/Basic/HeatEquation_EX1_C/Source/main.cpp Tutorials/Basic/HeatEquation_EX1_C/Source/myfunc.H Tutorials/Basic/HeatEquation_EX1_C/Source/myfunc.cpp Tutorials/Basic/HeatEquation_EX1_CF/Source/advance.cpp Tutorials/Basic/HeatEquation_EX1_CF/Source/advance_3d.f90 Tutorials/Basic/HeatEquation_EX1_CF/Source/init_phi_2d.f90 Tutorials/Basic/HeatEquation_EX1_CF/Source/init_phi_3d.f90 Tutorials/Basic/HeatEquation_EX1_CF/Source/main.cpp Tutorials/Basic/HeatEquation_EX1_CF/Source/myfunc.H Tutorials/Basic/HeatEquation_EX1_CF/Source/myfunc_F.H Tutorials/Basic/HeatEquation_EX1_F/advance.f90 Tutorials/Basic/HeatEquation_EX1_F/fmain.F90 Tutorials/Basic/HeatEquation_EX1_F/init_phi.f90 Tutorials/Basic/HeatEquation_EX2_C/Source/main.cpp Tutorials/Basic/HeatEquation_EX2_C/Source/myfunc.H Tutorials/Basic/HeatEquation_EX2_C/Source/myfunc.cpp Tutorials/Basic/HeatEquation_EX2_C/Source/mykernel.H Tutorials/Basic/HeatEquation_EX2_CF/Source/advance.cpp Tutorials/Basic/HeatEquation_EX2_CF/Source/advance_3d.f90 Tutorials/Basic/HeatEquation_EX2_CF/Source/init_phi_2d.f90 Tutorials/Basic/HeatEquation_EX2_CF/Source/init_phi_3d.f90 Tutorials/Basic/HeatEquation_EX2_CF/Source/main.cpp Tutorials/Basic/HeatEquation_EX2_CF/Source/myfunc.H Tutorials/Basic/HeatEquation_EX2_CF/Source/myfunc_F.H Tutorials/Basic/HeatEquation_EX3_C/Source/advance.cpp Tutorials/Basic/HeatEquation_EX3_C/Source/init_phi_2d.f90 Tutorials/Basic/HeatEquation_EX3_C/Source/init_phi_3d.f90 Tutorials/Basic/HeatEquation_EX3_C/Source/main.cpp Tutorials/Basic/HeatEquation_EX3_C/Source/myfunc.H Tutorials/Basic/HeatEquation_EX3_C/Source/myfunc_F.H Tutorials/Basic/main_C/main.cpp Tutorials/Basic/main_F/main.F90 Tutorials/Blueprint/AssignMultiLevelDensity/main.cpp Tutorials/Blueprint/CellSortedParticles/CellSortedPC.H Tutorials/Blueprint/CellSortedParticles/CellSortedPC.cpp Tutorials/Blueprint/CellSortedParticles/cell_sorted_3d.F90 Tutorials/Blueprint/CellSortedParticles/cell_sorted_F.H Tutorials/Blueprint/CellSortedParticles/main.cpp Tutorials/Blueprint/HeatEquation_EX1_C/Source/advance.cpp Tutorials/Blueprint/HeatEquation_EX1_C/Source/advance_3d.f90 Tutorials/Blueprint/HeatEquation_EX1_C/Source/init_phi_2d.f90 Tutorials/Blueprint/HeatEquation_EX1_C/Source/init_phi_3d.f90 Tutorials/Blueprint/HeatEquation_EX1_C/Source/main.cpp Tutorials/Blueprint/HeatEquation_EX1_C/Source/myfunc.H Tutorials/Blueprint/HeatEquation_EX1_C/Source/myfunc_F.H Tutorials/Blueprint/README.txt Tutorials/EB/CNS/Exec/Combustor/bc_fill_nd.F90 Tutorials/EB/CNS/Exec/Pulse/cns_prob.F90 Tutorials/EB/CNS/Exec/ShockRef/cns_prob.F90 Tutorials/EB/CNS/Exec/Sod/cns_prob.F90 Tutorials/EB/CNS/Source/CNS.H Tutorials/EB/CNS/Source/CNS.cpp Tutorials/EB/CNS/Source/CNS_F.H Tutorials/EB/CNS/Source/CNS_advance.cpp Tutorials/EB/CNS/Source/CNS_init_eb2.cpp Tutorials/EB/CNS/Source/CNS_io.cpp Tutorials/EB/CNS/Source/CNS_setup.cpp Tutorials/EB/CNS/Source/diffusion/cns_diff_mod.F90 Tutorials/EB/CNS/Source/diffusion/cns_eb_diff_wall.F90 Tutorials/EB/CNS/Source/diffusion/diff_coef_mod.F90 Tutorials/EB/CNS/Source/fortran/CNS_divop.F90 Tutorials/EB/CNS/Source/fortran/CNS_dudt.F90 Tutorials/EB/CNS/Source/fortran/CNS_f.F90 Tutorials/EB/CNS/Source/fortran/CNS_nd.F90 Tutorials/EB/CNS/Source/fortran/bc_fill_nd.F90 Tutorials/EB/CNS/Source/hydro/Hyp_gamma_MOL.F90 Tutorials/EB/CNS/Source/hydro/Hyp_gamma_MOL_EB.F90 Tutorials/EB/CNS/Source/hydro/analriem3d.F90 Tutorials/EB/CNS/Source/hydro/cns_eb_hyp_wall.F90 Tutorials/EB/CNS/Source/hydro/slope_mol_3d_gamma.F90 Tutorials/EB/CNS/Source/hydro/slope_mol_3d_gamma_EB.F90 Tutorials/EB/CNS/Source/main.cpp Tutorials/EB/GeometryGeneration/main.cpp Tutorials/EB/MacProj/main.cpp Tutorials/EB/Poisson/main.cpp Tutorials/ForkJoin/MLMG/ff.f90 Tutorials/FortranInterface/Advection_F/Exec/SingleVortex/Prob_2d.f90 Tutorials/FortranInterface/Advection_F/Exec/SingleVortex/Prob_3d.f90 Tutorials/FortranInterface/Advection_F/Exec/SingleVortex/face_velocity_2d.F90 Tutorials/FortranInterface/Advection_F/Exec/SingleVortex/face_velocity_3d.F90 Tutorials/FortranInterface/Advection_F/Source/Src_2d/advect_2d_mod.F90 Tutorials/FortranInterface/Advection_F/Source/Src_2d/compute_flux_2d.f90 Tutorials/FortranInterface/Advection_F/Source/Src_2d/slope_2d.f90 Tutorials/FortranInterface/Advection_F/Source/Src_3d/advect_3d_mod.F90 Tutorials/FortranInterface/Advection_F/Source/Src_3d/compute_flux_3d.f90 Tutorials/FortranInterface/Advection_F/Source/Src_3d/slope_3d.f90 Tutorials/FortranInterface/Advection_F/Source/amr_data_mod.F90 Tutorials/FortranInterface/Advection_F/Source/averagedown_mod.F90 Tutorials/FortranInterface/Advection_F/Source/bc_mod.F90 Tutorials/FortranInterface/Advection_F/Source/compute_dt_mod.F90 Tutorials/FortranInterface/Advection_F/Source/evolve_mod.F90 Tutorials/FortranInterface/Advection_F/Source/fillpatch_mod.F90 Tutorials/FortranInterface/Advection_F/Source/fmain.F90 Tutorials/FortranInterface/Advection_F/Source/initdata.F90 Tutorials/FortranInterface/Advection_F/Source/my_amr_mod.F90 Tutorials/FortranInterface/Advection_F/Source/plotfile_mod.F90 Tutorials/FortranInterface/Advection_F/Source/tagging_mod.F90 Tutorials/FortranInterface/Advection_octree_F/Exec/SingleVortex/Prob.f90 Tutorials/FortranInterface/Advection_octree_F/Exec/SingleVortex/face_velocity_2d.F90 Tutorials/FortranInterface/Advection_octree_F/Source/Src_2d/advect_2d_mod.F90 Tutorials/FortranInterface/Advection_octree_F/Source/Src_2d/compute_flux_2d.f90 Tutorials/FortranInterface/Advection_octree_F/Source/Src_2d/slope_2d.f90 Tutorials/FortranInterface/Advection_octree_F/Source/amr_data_mod.F90 Tutorials/FortranInterface/Advection_octree_F/Source/averagedown_mod.F90 Tutorials/FortranInterface/Advection_octree_F/Source/bc_mod.F90 Tutorials/FortranInterface/Advection_octree_F/Source/compute_dt_mod.F90 Tutorials/FortranInterface/Advection_octree_F/Source/evolve_mod.F90 Tutorials/FortranInterface/Advection_octree_F/Source/fillpatch_mod.F90 Tutorials/FortranInterface/Advection_octree_F/Source/my_amr_mod.F90 Tutorials/FortranInterface/Advection_octree_F2/Exec/SingleVortex/Prob.f90 Tutorials/FortranInterface/Advection_octree_F2/Exec/SingleVortex/face_velocity_2d.F90 Tutorials/FortranInterface/Advection_octree_F2/Source/Src_2d/advect_2d_mod.F90 Tutorials/FortranInterface/Advection_octree_F2/Source/Src_2d/compute_flux_2d.f90 Tutorials/FortranInterface/Advection_octree_F2/Source/Src_2d/slope_2d.f90 Tutorials/FortranInterface/Advection_octree_F2/Source/amr_data_mod.F90 Tutorials/FortranInterface/Advection_octree_F2/Source/averagedown_mod.F90 Tutorials/FortranInterface/Advection_octree_F2/Source/bc_mod.F90 Tutorials/FortranInterface/Advection_octree_F2/Source/compute_dt_mod.F90 Tutorials/FortranInterface/Advection_octree_F2/Source/evolve_mod.F90 Tutorials/FortranInterface/Advection_octree_F2/Source/fillpatch_mod.F90 Tutorials/FortranInterface/Advection_octree_F2/Source/my_amr_mod.F90 Tutorials/GPU/CNS/Source/CNS.H Tutorials/GPU/CNS/Source/CNS.cpp Tutorials/GPU/CNS/Source/CNS_K.H Tutorials/GPU/CNS/Source/CNS_advance.cpp Tutorials/GPU/CNS/Source/CNS_io.cpp Tutorials/GPU/CNS/Source/CNS_setup.cpp Tutorials/GPU/CNS/Source/hydro/CNS_hydro_K.H Tutorials/GPU/CNS/Source/main.cpp Tutorials/GPU/EBCNS/Source/CNS.H Tutorials/GPU/EBCNS/Source/CNS.cpp Tutorials/GPU/EBCNS/Source/CNS_advance.cpp Tutorials/GPU/EBCNS/Source/CNS_init_eb2.cpp Tutorials/GPU/EBCNS/Source/CNS_io.cpp Tutorials/GPU/EBCNS/Source/CNS_setup.cpp Tutorials/GPU/EBCNS/Source/hydro/CNS_hydro_K.H Tutorials/GPU/EBCNS/Source/main.cpp Tutorials/GPU/Launch/MyKernel_F.F90 Tutorials/GPU/ParallelScan/main.cpp Tutorials/MUI/Source_01/init_phi_3d.f90 Tutorials/MUI/Source_01/main_01.cpp Tutorials/MUI/Source_02/main_02.cpp Tutorials/MUI/doc/MUIcouplingNotes.tex Tutorials/Particles/CellSortedParticles/CellSortedPC.H Tutorials/Particles/CellSortedParticles/CellSortedPC.cpp Tutorials/Particles/CellSortedParticles/cell_sorted_3d.F90 Tutorials/Particles/CellSortedParticles/cell_sorted_F.H Tutorials/Particles/CellSortedParticles/main.cpp Tutorials/Particles/ElectromagneticPIC/Exec/CUDA/EMParticleContainer.cpp Tutorials/Particles/ElectromagneticPIC/Exec/CUDA/em_pic_K.H Tutorials/Particles/ElectromagneticPIC/Exec/OpenACC/EMParticleContainer.cpp Tutorials/Particles/ElectromagneticPIC/Exec/OpenACC/em_pic_3d.F90 Tutorials/Particles/ElectromagneticPIC/Exec/OpenMP/EMParticleContainer.cpp Tutorials/Particles/ElectromagneticPIC/Source/EMParticleContainer.H Tutorials/Particles/ElectromagneticPIC/Source/EMParticleContainerInit.cpp Tutorials/Particles/ElectromagneticPIC/Source/main.cpp Tutorials/Particles/ElectrostaticPIC/ElectrostaticParticleContainer.H Tutorials/Particles/ElectrostaticPIC/ElectrostaticParticleContainer.cpp Tutorials/Particles/ElectrostaticPIC/electrostatic_pic_2d.f90 Tutorials/Particles/ElectrostaticPIC/electrostatic_pic_3d.f90 Tutorials/Particles/ElectrostaticPIC/electrostatic_pic_F.H Tutorials/Particles/ElectrostaticPIC/main.cpp Tutorials/Particles/NeighborList/CheckPair.H Tutorials/Particles/NeighborList/Constants.H Tutorials/Particles/NeighborList/MDParticleContainer.H Tutorials/Particles/NeighborList/MDParticleContainer.cpp Tutorials/Particles/NeighborList/README.md Tutorials/Particles/NeighborList/main.cpp Tutorials/SDC/MISDC_ADR_2d/Source/SDC_sweeper.cpp Tutorials/SDC/MISDC_ADR_2d/Source/functions_2d.f90 Tutorials/SDC/MISDC_ADR_2d/Source/init_phi_2d.f90 Tutorials/SDC/MISDC_ADR_2d/Source/main.cpp Tutorials/SDC/MISDC_ADR_2d/Source/myfunc.H Tutorials/SDC/MISDC_ADR_2d/Source/myfunc_F.H Tutorials/SWFFT/SWFFT_poisson/SWFFT_Test.H Tutorials/SWFFT/SWFFT_poisson/SWFFT_Test.cpp Tutorials/SWFFT/SWFFT_poisson/SWFFT_Test_F.F90 Tutorials/SWFFT/SWFFT_poisson/SWFFT_Test_F.H Tutorials/SWFFT/SWFFT_poisson/main.cpp Tutorials/SWFFT/SWFFT_poisson/swfft_solver.cpp Tutorials/SWFFT/SWFFT_simple/SWFFT_Test.H Tutorials/SWFFT/SWFFT_simple/SWFFT_Test.cpp Tutorials/SWFFT/SWFFT_simple/SWFFT_Test_F.H Tutorials/SWFFT/SWFFT_simple/main.cpp Tutorials/SWFFT/SWFFT_simple/swfft_compute.cpp commit d84f787548d8e5b89466cd44fbad9d1656e7d08a Author: Weiqun Zhang Date: Sun Mar 14 06:15:13 2021 -0700 Free unused memory (#1861) If Arena is asked to allocate more than what's available in the system, try to free unused memory first. Src/Base/AMReX_Arena.H Src/Base/AMReX_Arena.cpp Src/Base/AMReX_CArena.H Src/Base/AMReX_CArena.cpp commit 85609ec18f8dd3b86eed3bf805e48515e5ce6e7d Author: kngott <30483578+kngott@users.noreply.github.com> Date: Sat Mar 13 17:52:38 2021 -0800 Fortran Interfaces for new FillPatch (#1793) Fortran Interfaces for new FillPatch. Compiles, and I believe the indexing is correct, but didn't test a Fortran implementation. Src/F_Interfaces/AmrCore/AMReX_fillpatch_fi.cpp Src/F_Interfaces/AmrCore/AMReX_fillpatch_mod.F90 Src/F_Interfaces/AmrCore/AMReX_interpolater_mod.F90 commit 9b17f925887a9058a3efb91af26a5ac082d5f2c6 Author: Weiqun Zhang Date: Fri Mar 12 12:14:08 2021 -0800 The_Device_Arena -> The_Arena (#1858) Replace The_Device_Arena with The_Arena in various places. Using The_Arena should help reduce the memory footprint. We used The_Device_Arena in the early days to avoid the penalty of page fault caused by touching managed memory on CPU. We are much better in that regard now. In fact, AMReX itself no longer depends on managed memory. Src/Base/AMReX_BlockMutex.cpp Src/Base/AMReX_CudaGraph.H Src/Base/AMReX_GpuAllocators.H Src/Base/AMReX_GpuFuse.cpp Src/Base/AMReX_GpuMemory.H Src/Base/AMReX_MFIter.H Src/Base/AMReX_Reduce.H Src/Base/AMReX_Scan.H commit 7e31cfd67ad9739b7779ee47d1a2750acf9c2cc1 Author: Weiqun Zhang Date: Fri Mar 12 12:13:42 2021 -0800 Arena Release Threshold (#1857) Release memory from Arena back to the system if it's above a threshold. For the pinned arena, the release threshold is set to the size of the global memory. For others, it's set to a huge number. These can be controlled with ParmParse parameters, `amrex.the_arena_release_threshold` and `the_[device|pinned|managed|async]_release_threshold`. The reason for setting the pinned arena's release threshold to the size of the global memory is that we received a report in the past that the pinned arena used by the communication functions wasted a huge amount of memory in the case that more and more grids were added over the run time. Src/Base/AMReX_Arena.H Src/Base/AMReX_Arena.cpp Src/Base/AMReX_CArena.H Src/Base/AMReX_CArena.cpp Src/Base/AMReX_DArena.H Src/Base/AMReX_DArena.cpp Src/Base/AMReX_EArena.H Src/Base/AMReX_EArena.cpp Src/Base/AMReX_PArena.cpp commit fafaccd7d4a7d0fbe90ec3507402e1988cfd00a1 Author: Maikel Nadolski Date: Fri Mar 12 17:07:57 2021 +0100 Replace is_pod with is_standard_layout && is_trivial (#1856) [`std::is_pod`](https://en.cppreference.com/w/cpp/types/is_pod) is deprecated in C++20 and should be replaced by `std::is_standard_layout && std::is_trivial`. This is the only occurrence of `std::is_pod` that I found in AMReX. If unchanged, this will trigger warnings in future compilers. This triggered a warning in our internal CI when using gcc 10.2 and the `-std=c++20` option. Src/Base/AMReX_GpuAsyncArray.H commit 7d594d2affc9188963ceb55a9ebca597d02462da Author: Maikel Nadolski Date: Thu Mar 11 17:49:00 2021 +0100 Feature/non local parallel copy (#1842) This PR introduces "Multiblock" capabilities to AMReX. Multiblock refers to having multiple distinct computational domains, known as blocks, that have some relation to each other. This PR proposes a generic NonLocalBC::ParallelCopy that performs copies between two FabArrays that might live on different blocks. The ParallelCopy respects - An index mapping such as already present in NonLocalBC (DTOS) - A projection function that will be called on either sender or receiver side to perform actions like e.g. swapping two components or changing the sign of velocity components The proposed usage is ``` NonLocalBC::ParallelCopy(dest, destbox, src, scomp, dcomp, ncomp, ngrow, index_mapping, fab_projection); ``` Notes - In the current implementation, `DTOS` needs to be a map between index spaces where inverse images of boxes are boxes again. - The name `NonLocalBC::ParallelCopy` is confusing if it does something else than `amrex::ParallelCopy`. Alternatives? - Parts of the implementation might be shared with other places in AMReX, such as in communication procedures of FabArray. But this is out of scope. - In addition to `SwapComponents`, a more generic `PermuteComponents` is useful. - The test is not commented yet. Src/Base/AMReX_Extension.H Src/Base/AMReX_NonLocalBC.H Src/Base/AMReX_NonLocalBC.cpp Src/Base/AMReX_NonLocalBCImpl.H Src/Base/AMReX_TypeTraits.H Src/Base/CMakeLists.txt Src/Base/Make.package Tests/CMakeLists.txt Tests/MultiBlock/Advection/CMakeLists.txt Tests/MultiBlock/Advection/GNUmakefile Tests/MultiBlock/Advection/Make.package Tests/MultiBlock/Advection/main.cpp Tests/MultiBlock/IndexType/CMakeLists.txt Tests/MultiBlock/IndexType/GNUmakefile Tests/MultiBlock/IndexType/Make.package Tests/MultiBlock/IndexType/main.cpp commit 4893049d2588261f235c1e05ef7d0121a9ad6b6c Author: Pankaj Jha Date: Wed Mar 10 14:35:57 2021 -0800 Include missing library 'pthread' for linking to amrex library (#1854) - [HOTFIX ] Include missing library 'pthread' for linking to amrex library in the tutorial 'Basic/Build_with_libamrex' Tutorials/Basic/Build_with_libamrex/GNUmakefile commit 6ec00666843dcc0ae794358c872dd92e30a3cd34 Author: Andrew Myers Date: Mon Mar 8 17:05:52 2021 -0500 Enable expanding MPI / GPU macros for doxygen. (#1853) Docs/Doxygen/doxygen.conf commit 9001a7486ffcdd68cec9f2c1212559c4982865d8 Author: Weiqun Zhang Date: Mon Mar 8 08:15:45 2021 -0800 Revert the use of codeplay_host_task (#1851) It results int NaNs. Src/Base/AMReX_GpuAsyncArray.H Src/Base/AMReX_GpuElixir.cpp commit fff2224879807f73c5a4acac19609f5630f77a8e Author: asalmgren Date: Sun Mar 7 10:01:11 2021 -0800 Add harmonic averaging (#1850) Mostly adding functionality, but also fixed a bug for 2d multicomponent cc->fc averaging Src/Base/AMReX_MultiFabUtil.H Src/Base/AMReX_MultiFabUtil.cpp Src/Base/AMReX_MultiFabUtil_1D_C.H Src/Base/AMReX_MultiFabUtil_2D_C.H Src/Base/AMReX_MultiFabUtil_3D_C.H commit 146f3f2b7596cbee73d2b2eb8d48d4576ecc3f7c Author: Candace Gilet Date: Sat Mar 6 21:36:28 2021 -0500 Fix typo in GNUMake/Make.defs introduced in PR#1846 (#1849) Tools/GNUMake/Make.defs commit 0fc630c5bbc08b65feda01cae000b2314772d240 Author: Weiqun Zhang Date: Fri Mar 5 14:53:57 2021 -0800 AMREX_NO_PROBINIT (#1846) For Amr/AmrLevel based codes, there is now a compile time option AMREX_NO_PROBINIT=[TRUE|FALSE] for the support of the probin file and amrex_probinit function. The default behavior of requiring a user defined amrex_probinit function unless amr.probin_file is empty does not change unless amrex is built into a shared library on non-GNU/Linux system. The reason for the inconsistency is there is no portable way of supporting a weak symbol (.e.g, undefined amrex_probinit in amrex itself) in a shared library. This will partially solve the shared library build issue on Mac and Windows. Co-authored-by: Michele Rosso Docs/sphinx_documentation/source/BuildingAMReX.rst Src/Amr/AMReX_Amr.H Src/Amr/AMReX_Amr.cpp Tools/CMake/AMReXGenerateConfigHeader.cmake Tools/CMake/AMReXOptions.cmake Tools/CMake/AMReXSetDefines.cmake Tools/CMake/AMReX_Config.H.in Tools/GNUMake/Make.defs Tools/libamrex/configure.py commit e7a264ddc0b008310d80c98ecf578e292f3cf13d Author: Maikel Nadolski Date: Fri Mar 5 19:53:28 2021 +0100 Use unique_ptr instead of naked pointer (#1839) This commit replaces naked pointer to char with `std::unique_ptr` in `NonLocalBC::CommHandler`. This will lead to proper cleanup in case of any exceptions. Note, that `Comm_nowait` and `Comm_finish` are *not* marked noexcept. Furthermore, future PRs will add interpolation and some kind of projection for non-local data transactions. Error handling for those is still unclear. Maybe, we want to mark everything noexcept. This is merely an implementation change without nearly any noticable difference in user code. Src/Base/AMReX_Extension.H Src/Base/AMReX_FabArray.H Src/Base/AMReX_FabArrayCommI.H Src/Base/AMReX_NonLocalBCImpl.H commit ef3a757eae988bff7948a47c346a6fb47ffd9ac0 Author: kngott <30483578+kngott@users.noreply.github.com> Date: Fri Mar 5 09:43:54 2021 -0800 Fix early returns in ParallelCopy. (#1845) Fixes a new segfault bug due to pc_src.size() being called when pc_src isn't set. Instead, tracks all ParallelCopy_nowait early returns and checks once at the top of the ParallelCopy_finish. Src/Base/AMReX_FabArray.H Src/Base/AMReX_FabArrayCommI.H commit 6c1296b2466263a5aba46b5df16aceda47457b11 Author: Candace Gilet Date: Fri Mar 5 11:49:42 2021 -0500 Revert "Change default for Amr::probin_file -> "" (#1837)" (#1844) This reverts commit 2519768caedcf46a05b0cfd2b552dff2148465f7. Src/Amr/AMReX_Amr.cpp commit 2fb82bf64838aba949f7a641988a0c58774e5a88 Author: mic84 Date: Thu Mar 4 13:25:13 2021 -0800 CMake: move AMReX_INSTALL option to AMReXOptions.cmake (#1840) * CMake: add missing AMReX_INSTALL option * CMake: remove AMReX_INSTALL from root CMakeLists.txt CMakeLists.txt Tools/CMake/AMReXOptions.cmake commit e01e1c73e670defda422b0f6a7b0a5fc45fb17af Author: Weiqun Zhang Date: Thu Mar 4 12:23:03 2021 -0800 Add EditorConfig (#1841) .editorconfig commit 076be729a1cc8fabc5c18f6836e134f85fa8b0c7 Author: asalmgren Date: Thu Mar 4 08:00:53 2021 -0800 Migrate more of Tutorials/LinearSolvers to Tests/LinearSolvers (#1835) Tests/LinearSolvers/ABecLaplacian_C/CMakeLists.txt Tests/LinearSolvers/ABecLaplacian_C/GNUmakefile Tests/LinearSolvers/ABecLaplacian_C/Make.package Tests/LinearSolvers/ABecLaplacian_C/MyTest.H Tests/LinearSolvers/ABecLaplacian_C/MyTest.cpp Tests/LinearSolvers/ABecLaplacian_C/MyTestPlotfile.cpp Tests/LinearSolvers/ABecLaplacian_C/initProb.cpp Tests/LinearSolvers/ABecLaplacian_C/initProb_K.H Tests/LinearSolvers/ABecLaplacian_C/inputs Tests/LinearSolvers/ABecLaplacian_C/inputs-inhomNeumann Tests/LinearSolvers/ABecLaplacian_C/inputs-rt-abeclap-com Tests/LinearSolvers/ABecLaplacian_C/inputs-rt-poisson-lev Tests/LinearSolvers/ABecLaplacian_C/inputs.hypre Tests/LinearSolvers/ABecLaplacian_C/inputs.petsc Tests/LinearSolvers/ABecLaplacian_C/main.cpp Tests/LinearSolvers/ABecLaplacian_C/scalingtest/inputs.test Tests/LinearSolvers/ABecLaplacian_C/scalingtest/main.diff Tests/LinearSolvers/ABecLaplacian_C/scalingtest/results.org Tests/LinearSolvers/ABecLaplacian_C/scalingtest/run-1.sh Tests/LinearSolvers/ABecLaplacian_C/scalingtest/run-1024.sh Tests/LinearSolvers/ABecLaplacian_C/scalingtest/run-128.sh Tests/LinearSolvers/ABecLaplacian_C/scalingtest/run-16.sh Tests/LinearSolvers/ABecLaplacian_C/scalingtest/run-2.sh Tests/LinearSolvers/ABecLaplacian_C/scalingtest/run-2048.sh Tests/LinearSolvers/ABecLaplacian_C/scalingtest/run-256.sh Tests/LinearSolvers/ABecLaplacian_C/scalingtest/run-32.sh Tests/LinearSolvers/ABecLaplacian_C/scalingtest/run-4.sh Tests/LinearSolvers/ABecLaplacian_C/scalingtest/run-512.sh Tests/LinearSolvers/ABecLaplacian_C/scalingtest/run-64.sh Tests/LinearSolvers/ABecLaplacian_C/scalingtest/run-8.sh Tests/LinearSolvers/ABecLaplacian_C/threadmultiple_test/inputs.test Tests/LinearSolvers/ABecLaplacian_C/threadmultiple_test/knl-mpi-1.sh Tests/LinearSolvers/ABecLaplacian_C/threadmultiple_test/knl-mpi-16.sh Tests/LinearSolvers/ABecLaplacian_C/threadmultiple_test/knl-mpi-2.sh Tests/LinearSolvers/ABecLaplacian_C/threadmultiple_test/knl-mpi-32.sh Tests/LinearSolvers/ABecLaplacian_C/threadmultiple_test/knl-mpi-4.sh Tests/LinearSolvers/ABecLaplacian_C/threadmultiple_test/knl-mpi-64.sh Tests/LinearSolvers/ABecLaplacian_C/threadmultiple_test/knl-mpi-8.sh Tests/LinearSolvers/ABecLaplacian_C/threadmultiple_test/knl-omp-1.sh Tests/LinearSolvers/ABecLaplacian_C/threadmultiple_test/knl-omp-16.sh Tests/LinearSolvers/ABecLaplacian_C/threadmultiple_test/knl-omp-2.sh Tests/LinearSolvers/ABecLaplacian_C/threadmultiple_test/knl-omp-32.sh Tests/LinearSolvers/ABecLaplacian_C/threadmultiple_test/knl-omp-4.sh Tests/LinearSolvers/ABecLaplacian_C/threadmultiple_test/knl-omp-64.sh Tests/LinearSolvers/ABecLaplacian_C/threadmultiple_test/knl-omp-8.sh Tests/LinearSolvers/ABecLaplacian_C/threadmultiple_test/main.diff Tests/LinearSolvers/ABecLaplacian_C/threadmultiple_test/results.org Tests/LinearSolvers/ABecLaplacian_F/CMakeLists.txt Tests/LinearSolvers/ABecLaplacian_F/GNUmakefile Tests/LinearSolvers/ABecLaplacian_F/Make.package Tests/LinearSolvers/ABecLaplacian_F/README Tests/LinearSolvers/ABecLaplacian_F/init_prob.F90 Tests/LinearSolvers/ABecLaplacian_F/inputs Tests/LinearSolvers/ABecLaplacian_F/inputs-rt-abeclap-lev Tests/LinearSolvers/ABecLaplacian_F/inputs-rt-poisson-com Tests/LinearSolvers/ABecLaplacian_F/main.F90 Tests/LinearSolvers/ABecLaplacian_F/mytest.F90 Tests/LinearSolvers/MultiComponent/GNUmakefile Tests/LinearSolvers/MultiComponent/MCNodalLinOp.H Tests/LinearSolvers/MultiComponent/MCNodalLinOp.cpp Tests/LinearSolvers/MultiComponent/Make.package Tests/LinearSolvers/MultiComponent/inputs Tests/LinearSolvers/MultiComponent/main.cpp Tests/LinearSolvers/NodalPoisson/CMakeLists.txt Tests/LinearSolvers/NodalPoisson/GNUmakefile Tests/LinearSolvers/NodalPoisson/Make.package Tests/LinearSolvers/NodalPoisson/MyTest.H Tests/LinearSolvers/NodalPoisson/MyTest.cpp Tests/LinearSolvers/NodalPoisson/MyTestPlotfile.cpp Tests/LinearSolvers/NodalPoisson/inputs-rt Tests/LinearSolvers/NodalPoisson/main.cpp Tests/LinearSolvers/Nodal_Projection_EB/CMakeLists.txt Tests/LinearSolvers/Nodal_Projection_EB/GNUmakefile Tests/LinearSolvers/Nodal_Projection_EB/Make.package Tests/LinearSolvers/Nodal_Projection_EB/README Tests/LinearSolvers/Nodal_Projection_EB/inputs_3d Tests/LinearSolvers/Nodal_Projection_EB/main.cpp Tests/LinearSolvers/NodeTensorLap/CMakeLists.txt Tests/LinearSolvers/NodeTensorLap/GNUmakefile Tests/LinearSolvers/NodeTensorLap/Make.package Tests/LinearSolvers/NodeTensorLap/MyTest.H Tests/LinearSolvers/NodeTensorLap/MyTest.cpp Tests/LinearSolvers/NodeTensorLap/MyTestPlotfile.cpp Tests/LinearSolvers/NodeTensorLap/main.cpp Tools/RegressionTesting/AMReX-cuda-tests.ini Tools/RegressionTesting/AMReX-dpcpp-tests.ini Tools/RegressionTesting/AMReX-hip-tests.ini Tools/RegressionTesting/AMReX-tests.ini commit 2519768caedcf46a05b0cfd2b552dff2148465f7 Author: Candace Gilet Date: Thu Mar 4 11:00:18 2021 -0500 Change default for Amr::probin_file -> "" (#1837) Src/Amr/AMReX_Amr.cpp commit 86c58645a181907962bd9a72d7c12089d68c1a3a Author: Weiqun Zhang Date: Wed Mar 3 21:47:44 2021 -0800 Fix BoxArray in doxygen (#1838) Src/Base/AMReX_BoxArray.H commit 0ad9fd9c31e9ddfbbdbb57e659d7864b2f2a6ed8 Author: asalmgren Date: Wed Mar 3 14:35:12 2021 -0800 Move Tutorials/LinearSolvers/MAC_Projection_EB to Tests/LinearSolvers/MAC_Projection_EB (#1834) Tests/LinearSolvers/MAC_Projection_EB/CMakeLists.txt Tests/LinearSolvers/MAC_Projection_EB/GNUmakefile Tests/LinearSolvers/MAC_Projection_EB/Make.package Tests/LinearSolvers/MAC_Projection_EB/README Tests/LinearSolvers/MAC_Projection_EB/inputs_3d Tests/LinearSolvers/MAC_Projection_EB/main.cpp Tools/RegressionTesting/AMReX-cuda-tests.ini Tools/RegressionTesting/AMReX-hip-tests.ini Tools/RegressionTesting/AMReX-tests.ini commit 42c927d1cae12f90a841c07ccd88275d8a66d5cc Author: Axel Huebl Date: Wed Mar 3 09:16:46 2021 -0800 AMReX Install: Config Header (#1833) Found another header that still installed itself :) Tools/CMake/AMReXGenerateConfigHeader.cmake commit dd2ec76a5cc5db01caa17bdf51efb603c019346d Author: Weiqun Zhang Date: Tue Mar 2 10:30:48 2021 -0800 Fix ncomp bug in linear solver (#1832) Need to give m_ncomp a default value, otherwise default constructor followed by define(...) will not work. Src/LinearSolvers/MLMG/AMReX_MLABecLaplacian.H Src/LinearSolvers/MLMG/AMReX_MLEBABecLap.H commit 28fd2ee530e4c0229f04e33e041989f8535c9d33 Author: Axel Huebl Date: Tue Mar 2 09:41:39 2021 -0800 CMake: AMReX_INSTALL (#1831) Add an `AMReX_INSTALL` option that controls `install()` generation. Such options are typically used in superbuilds by dependent projects to supress the install of AMReX headers, helpers and libs, e.g. if AMReX is fully consumed as private dependency. CMakeLists.txt Docs/sphinx_documentation/source/BuildingAMReX.rst commit b7ddf2d2677fce63a567612978e01ced288dbda2 Author: Weiqun Zhang Date: Mon Mar 1 11:18:19 2021 -0800 Update CHANGES for 21.03 (#1830) CHANGES commit 2794f2f8f89d7461e4d99f19edc805c0feb8d7fe Author: Weiqun Zhang Date: Mon Mar 1 08:24:18 2021 -0800 Fix a sign compare warning (#1827) Src/Base/AMReX_GpuLaunchFunctsG.H commit 0518bb82022da8e6d6250d65b11f7cdaa8b31b24 Author: asalmgren Date: Sun Feb 28 12:58:34 2021 -0800 Ncomp for average cc to fc (#1826) Src/Base/AMReX_MultiFabUtil.H Src/Base/AMReX_MultiFabUtil.cpp Src/Base/AMReX_MultiFabUtil_1D_C.H Src/Base/AMReX_MultiFabUtil_2D_C.H Src/Base/AMReX_MultiFabUtil_3D_C.H commit 90efb8b28714a496a3cd9188bdba4794579c59a4 Author: asalmgren Date: Sun Feb 28 09:01:48 2021 -0800 Add n_comp as an optional argument to the constructor of AMReX_MLABecLaplacian (#1825) Src/LinearSolvers/MLMG/AMReX_MLABecLaplacian.H Src/LinearSolvers/MLMG/AMReX_MLABecLaplacian.cpp commit 662231a896991dd4fc110aa2fae6740a1f511c59 Author: Weiqun Zhang Date: Sat Feb 27 08:37:59 2021 -0800 Arena Query Functions and Bug Fix in FArrayBox & IArrayBox (#1823) Add Arena::isDeviceAccessible, isHostAccessible, isManaged, isDevice, and isPinned. Use them in various places. Fix bugs in FArrayBox & IArrayBox that touch device memory allocated by The_Async_Arena() on the host. Src/Base/AMReX_Arena.H Src/Base/AMReX_Arena.cpp Src/Base/AMReX_BArena.H Src/Base/AMReX_BArena.cpp Src/Base/AMReX_BaseFab.H Src/Base/AMReX_FArrayBox.cpp Src/Base/AMReX_FabArrayUtility.H Src/Base/AMReX_GpuControl.H Src/Base/AMReX_GpuDevice.H Src/Base/AMReX_GpuDevice.cpp Src/Base/AMReX_IArrayBox.cpp Src/Base/AMReX_PArena.H Src/Base/AMReX_PArena.cpp Src/Base/AMReX_VisMF.cpp commit 272e9fc5e93b54f125e907b29e3b9db149c4ae63 Author: Weiqun Zhang Date: Sat Feb 27 08:09:28 2021 -0800 DPCPP: Fix bug in computing the number of active threads (#1822) Thank Intel for the help on debugging! Src/Base/AMReX_GpuLaunchFunctsG.H commit da6eca54acd9fc291fec189a9ee75b54d9727cab Author: mic84 Date: Fri Feb 26 09:51:54 2021 -0800 CMake: add -Wno-pass-failed to Clang-based compilers (#1815) * CMake: add -Wno-pass-failed to Clang-based compilers * CMake: forgot to add flag in one place * CMake: need that flag for Release too Tools/CMake/AMReXFlagsTargets.cmake Tools/CMake/AMReXSYCL.cmake commit f208bc92bc0c9c0b744423cf3cc847db23e5a0ba Author: drangara <69211175+drangara@users.noreply.github.com> Date: Fri Feb 26 11:55:37 2021 -0500 Cover multiple cuts (#1810) ## Summary When building the edge types, cover the cut edges that touch opposite regular corners in order to prevent the `too many cuts in a face error`. Logic is only enabled when flag is set to true in inputs file (default is false). ## Additional background When trying to resolve complex geometries with sharp edges one can run into cases where there are multiple EB cuts on a face. In this change, when the flag `cover_multiple_cuts` is set, we cover appropriate edges in order to avoid this error. Manually tested geometry generation on some complex geometries that were previously failing to resolve. Src/EB/AMReX_EB2_3D_C.H Src/EB/AMReX_EB2_Level.H Src/EB/AMReX_EB2_MultiGFab.H Src/EB/AMReX_EB2_MultiGFab.cpp commit 35d27f7b8ba4ac3b77faadf12af5132ecd1f24cc Author: Weiqun Zhang Date: Fri Feb 26 07:33:30 2021 -0800 Fix sign-compare warnings (#1817) Src/Base/AMReX_DistributionMapping.cpp Src/Base/AMReX_Machine.cpp Src/Base/AMReX_MultiFabUtil.cpp Src/Base/AMReX_ParmParse.cpp Src/Base/AMReX_VisMF.cpp Src/EB/AMReX_EBToPVD.cpp Src/EB/AMReX_distFcnElement.cpp Src/LinearSolvers/MLMG/AMReX_MLLinOp.cpp Src/Particle/AMReX_ParticleCommunication.cpp Src/Particle/AMReX_StructOfArrays.H Tools/GNUMake/comps/gnu.mak commit 8cdf250b04aa1c06ae745133100be2988e9a35aa Author: Erik Schnetter Date: Thu Feb 25 22:52:50 2021 -0500 AMReX_GpuFuse.H: Avoid compiler warning (#1816) Src/Base/AMReX_GpuFuse.H commit a2c92323ee40c46a1848438230ee2b6ac1cfefd8 Author: Erik Schnetter Date: Thu Feb 25 17:03:58 2021 -0500 Avoid compiler warning in FabArray::defined (#1814) Avoid compiler warning in `FabArray::defined`: Convert `size_t` to `int`. Src/Base/AMReX_FabArray.H commit 327c62b150bb23ce01633a4d7ae060291156c62d Author: Weiqun Zhang Date: Wed Feb 24 13:42:23 2021 -0800 Add regression testing script for HIP (#1812) Tools/RegressionTesting/AMReX-hip-tests.ini commit f2e46722e8a6d2a8fd383d268b14579c4bae2929 Author: Weiqun Zhang Date: Wed Feb 24 09:16:25 2021 -0800 Add Axel to the list of core developers (#1811) .zenodo.json CONTRIBUTING.md commit 47389357d5de37d84be69d9b06a294e64cb2a96b Author: Weiqun Zhang Date: Mon Feb 22 09:30:43 2021 -0800 Update docs for GitHub Discussions (#1808) CONTRIBUTING.md Docs/sphinx_documentation/source/index.rst commit 13aa70f0ae5e156b9afdec8ed9f8ef13ba86a0fb Author: Weiqun Zhang Date: Mon Feb 22 08:40:22 2021 -0800 The_Async_Arena and Elixir::append (#1804) Add The_Async_Arena that uses CUDA steam-ordered memory allocator, if available. If the feature is not available, use The_Arena to allocate and free memory and the free is done with a host callback function. The purpose of this Arena is to be used by a temporary FArrayBox inside an MFIter loop. Add Elixir::append that appends an rvalue Elixir to an existing one to reduce the number of host callback calls. Docs/sphinx_documentation/source/GPU.rst Src/Base/AMReX_Arena.H Src/Base/AMReX_Arena.cpp Src/Base/AMReX_BaseFab.H Src/Base/AMReX_FArrayBox.H Src/Base/AMReX_FArrayBox.cpp Src/Base/AMReX_GpuDevice.H Src/Base/AMReX_GpuDevice.cpp Src/Base/AMReX_GpuElixir.H Src/Base/AMReX_GpuElixir.cpp Src/Base/AMReX_IArrayBox.H Src/Base/AMReX_IArrayBox.cpp Src/Base/AMReX_PArena.H Src/Base/AMReX_PArena.cpp Src/Base/CMakeLists.txt Src/Base/Make.package commit 110422d41aae2eee79772d890a52de4ee54f1d2c Author: kngott <30483578+kngott@users.noreply.github.com> Date: Sun Feb 21 20:13:08 2021 -0800 Add function name output option to TinyProfiler. (#1803) Use tiny_profiler.verbose=1 to output "Entering " and "Leaving " output. Primarily for hunting down hang issues. Src/Base/AMReX_TinyProfiler.H Src/Base/AMReX_TinyProfiler.cpp commit 85cad7492dfa14851cc47ed4c55e4d2a56d6d56c Author: Axel Huebl Date: Fri Feb 19 15:40:58 2021 -0800 CMake: More Particle Tests (#1801) Tests/Particles/AssignDensity/CMakeLists.txt Tests/Particles/AssignMultiLevelDensity/CMakeLists.txt Tests/Particles/AsyncIO/CMakeLists.txt Tests/Particles/AsyncIO/inputs Tests/Particles/GhostsAndVirtuals/CMakeLists.txt Tests/Particles/GhostsAndVirtuals/main.cpp Tests/Particles/InitFromAscii/CMakeLists.txt Tests/Particles/InitFromAscii/main.cpp Tests/Particles/Intersection/CMakeLists.txt Tests/Particles/Intersection/main.cpp Tests/Particles/ParallelContext/CMakeLists.txt Tests/Particles/ParticleArray/CMakeLists.txt Tests/Particles/ParticleIterator/CMakeLists.txt Tests/Particles/SparseBins/CMakeLists.txt Tests/Particles/TypeDescriptor/CMakeLists.txt commit 093275a3bbf26e45faa43971418bac597b8a7b85 Author: mic84 Date: Wed Feb 17 13:36:06 2021 -0800 Extrapolater: make assertion less restrictive (#1799) Src/Amr/AMReX_Extrapolater.cpp commit 86e26ca7bdcb64d5e694f325973541f721c568b9 Author: Weiqun Zhang Date: Wed Feb 17 10:57:36 2021 -0800 DPCPP: Use codeplay_host_task (#1797) In AsyncArray and Elixir, replace streamSynchronize with SYCL 2020's host_task. Note that the function name in the current oneAPI release is codeplay_host_task, because it started as a Codeplay extension. This will probably change in the future, since the extension has been accepted into the standard. Src/Base/AMReX_GpuAsyncArray.H Src/Base/AMReX_GpuElixir.cpp commit d29a5709545b0d44051e82a895be37f5f821cbc6 Author: Weiqun Zhang Date: Wed Feb 17 10:47:06 2021 -0800 Remove functions about the number of host callbacks (#1798) We used to use the number of host callbacks to limit the number of GPU streams. But we no longer do that anymore, and the default number of streams has been reduced to 4. So the functions keeping track of the number of host callbacks can now be removed. Src/Base/AMReX_GpuAsyncArray.H Src/Base/AMReX_GpuDevice.H Src/Base/AMReX_GpuDevice.cpp Src/Base/AMReX_GpuElixir.cpp Src/Base/AMReX_MFIter.cpp commit 23c2fa054f21263220d1f1d384889af9593fc330 Author: Andrew Myers Date: Wed Feb 17 10:10:33 2021 -0800 Add ParticleArray classes (#1796) Add a class for storing arrays of particle data that can be toggled between AoS and SoA. Src/Base/AMReX_PODVector.H Src/Particle/AMReX_ParticleArray.H Src/Particle/CMakeLists.txt Src/Particle/Make.package Tests/Particles/ParticleArray/GNUmakefile Tests/Particles/ParticleArray/Make.package Tests/Particles/ParticleArray/main.cpp commit e611121acca7696320f3757982de857cbab74926 Author: kngott <30483578+kngott@users.noreply.github.com> Date: Tue Feb 16 16:43:21 2021 -0800 ParallelCopy_nowait & ParallelCopy_finish (#1765) Most direct implementation of `ParallelCopy_nowait` and `ParallelCopy_finish`, with minimal changes. Previous version had a max component limit per message, set by the input parameter `fabarray.maxcomp`, 25 by default. This is preserved. If `ncomp > fabarray.maxcomp`, `ParallelCopy_nowait` calls `ParallelCopy_finish` internally for all iterations of the loop except the final iteration. For most apps using GPUs that haven't been manually setting `fabarray.maxcomp`, this will almost certainly mean behavior identical to `FillBoundary_nowait`, as `ncomp > 25` would require very small `MultiFabs` to fit on current GPUs. Keeps `ParallelCopy`, which simply calls `ParallelCopy_nowait` followed by `ParallelCopy_finish`. No doubt, there are potential optimizations available: e.g. if `ncomp>fabarray.maxcomp` and `ncomp%fabarray.maxcomp != 0` returning from `nowait` with a message with a full `fabarray.maxcomp` components may allow the largest possible overlap. But, this version gets the feature running while we discuss additional improvements such as this. Plan to follow up with a scaling study on the effect of message size on `ParallelCopy` timing on Summit by exploiting the `fabarray.maxcomp` feature to see if rewriting with more controlled buffer sizes yields enough benefit to justify a rewrite. Src/Base/AMReX_FabArray.H Src/Base/AMReX_FabArrayCommI.H commit 305770490a946deede9eee31ddd3814fe077ff67 Author: Weiqun Zhang Date: Tue Feb 16 14:40:47 2021 -0800 Fix a new bug in GNU Make (#1795) Must set CXXSTD for nvcc because we need to use it later. Tools/GNUMake/comps/nvcc.mak commit 446fb1e9ee4647604a56a68063f64cb7d94cfd50 Author: Weiqun Zhang Date: Tue Feb 16 10:50:25 2021 -0800 Add FabArray::tags() to return the tags. (#1794) Src/Base/AMReX_FabArray.H commit 4a332b5c3875a5eba705ad0d33f76d7f592bef80 Author: Andrew Myers Date: Tue Feb 16 09:30:16 2021 -0800 Bump minimum C++ standard from 11 to 14. (#1787) This is well-supported now and will allow a lot of the code in amrex to be simplified. I have also bumped the version on the Cuda 9 CI test to Cuda 9.2 and GCC 6.5.0, since the old combination did not work with C++14. .github/workflows/dependencies/dependencies_nvcc9.sh .github/workflows/linux.yml Docs/sphinx_documentation/source/BuildingAMReX.rst Docs/sphinx_documentation/source/BuildingAMReX_Chapter.rst INSTALL Tools/CMake/AMReX_Config.cmake Tools/GNUMake/comps/cray.mak Tools/GNUMake/comps/gnu.mak Tools/GNUMake/comps/intel.mak Tools/GNUMake/comps/llvm-flang.mak Tools/GNUMake/comps/nag.mak Tools/GNUMake/comps/nvcc.mak Tools/GNUMake/comps/nvhpc.mak Tools/GNUMake/comps/pgi.mak Tutorials/Basic/Build_with_libamrex/GNUmakefile commit 3ed36ff23c1d408a855573eadcce2686b51160d8 Author: asalmgren Date: Mon Feb 15 15:07:55 2021 -0800 fix table formatting in GPU.rst (#1790) Docs/sphinx_documentation/source/GPU.rst Docs/sphinx_documentation/source/Post_Processing.rst Docs/sphinx_documentation/source/Post_Processing_Chapter.rst commit e8b4d0739e4f21b44169a195a7aa40eaf0092c1b Author: asalmgren Date: Mon Feb 15 15:03:15 2021 -0800 Add an interface to the flux redistribution that takes the arguments (#1789) as Array4s rather than FABs or MultiFabs. Docs/sphinx_documentation/source/Post_Processing.rst Src/EB/AMReX_EB_utils.H Src/EB/AMReX_EB_utils.cpp commit 15a84595c5b88f4e289be505869786400bac49e3 Author: Max Katz Date: Mon Feb 15 15:00:58 2021 -0800 Update the preferred short name of NVIDIA HPC SDK to nvhpc (#1788) `nvhpc` is the preferred short name of the NVIDIA HPC SDK (for example when used in modules). Tools/GNUMake/Make.defs Tools/GNUMake/comps/nvhpc.mak commit 492c5524375858d46f4dd6ea97589cffef749c42 Author: Weiqun Zhang Date: Mon Feb 15 14:58:59 2021 -0800 Fix sphinx config for sphinx 3.5.0 (#1792) https://github.com/sphinx-doc/sphinx/issues/8885 Docs/sphinx_documentation/source/conf.py Docs/sphinx_tutorials/source/conf.py commit df5bbe922563bf3ae9eb97a566014205e1a1b2f4 Author: Weiqun Zhang Date: Thu Feb 11 10:07:38 2021 -0800 BoxArray::simplified() & simplified_list() (#1786) Make them public so that they can be used by application codes to regenerate grids covering exactly the same region but with a different max grid size. However, it should be emphasized that they only work with regular BoxArrays without anything like coarsening. Src/Base/AMReX_BoxArray.H commit 7742e1c99f5e7a5ac7a4fa3530dba3a30d51daeb Author: mic84 Date: Wed Feb 10 15:21:10 2021 -0800 CMake: bump up minimum required HYPRE version (#1785) Tools/CMake/AMReXThirdPartyLibraries.cmake commit 932836ad422ad742a7c3c9b6375bad9cffec76d7 Author: asalmgren Date: Tue Feb 9 18:17:01 2021 -0800 fix bug in EB redistribution which was accidentally including (#1784) contributions from outside a non-periodic domain Src/EB/AMReX_EB_utils.cpp commit d2c88194fac2b9e35e469cf332034bedbf361cf9 Author: Axel Huebl Date: Tue Feb 9 00:02:41 2021 -0800 Windows: WINDOWS_EXPORT_ALL_SYMBOLS (#1782) I am seeing some linker issues downstream when building Python bindings with WarpX on Windows. This nice option makes sure we don't need to add dll export symbols all over the place when building a shared AMReX library (dll). Kudos to the ROOT team at CERN for contributing this feature years ago to CMake. Tools/CMake/AMReXBuildInfo.cmake Tools/CMake/AMReX_Config.cmake commit 6219fa1df9b1980327da3f40bbcf4457a93a6602 Author: Weiqun Zhang Date: Mon Feb 8 18:35:42 2021 -0800 Fix precision issue in ParmParse::add (#1783) When adding values to ParmParse, we need to call `std::setprecision` so as not to lose precision. Src/Base/AMReX_ParmParse.cpp commit ba0f1c0e77515debd83fe651abd3045d87567dab Author: Weiqun Zhang Date: Mon Feb 8 14:57:30 2021 -0800 Fix bug in #1774 (#1779) A bug was introduced in #1774 that resulted in incorrect weights when making nodal RHS from cell-centered RHS. Src/LinearSolvers/MLMG/AMReX_MLNodeLap_2D_K.H Src/LinearSolvers/MLMG/AMReX_MLNodeLap_3D_K.H commit d44e353bb9f7745973502e4d7b16bbde3d29ed09 Author: Andrew Myers Date: Mon Feb 8 11:45:59 2021 -0800 Fix up neighbor list for 2D (#1781) Src/Particle/AMReX_NeighborList.H commit 591f48db3c0539fa639e8d846cd4a036a7e63c53 Author: Weiqun Zhang Date: Sat Feb 6 17:38:19 2021 -0800 Remove managed memory from CNS (#1780) Tutorials/GPU/CNS/Exec/RT/cns_prob.cpp Tutorials/GPU/CNS/Exec/RT/cns_prob_parm.H Tutorials/GPU/CNS/Exec/Sod/cns_prob.cpp Tutorials/GPU/CNS/Exec/Sod/cns_prob_parm.H Tutorials/GPU/CNS/Source/CNS.H Tutorials/GPU/CNS/Source/CNS.cpp Tutorials/GPU/CNS/Source/CNS_advance.cpp Tutorials/GPU/CNS/Source/CNS_derive.cpp Tutorials/GPU/CNS/Source/CNS_parm.H Tutorials/GPU/CNS/Source/CNS_setup.cpp commit d31f9b4a36a1cb912d4dd52bb1807117fe9d79f6 Author: Weiqun Zhang Date: Fri Feb 5 19:17:43 2021 -0800 Overset Solver with Refinement Ratio of 4 (#1778) We need to define overset mask for on coarse MG levels of fine AMR levels. We didn't need to do it for refinement ratio of 2 because there are no coarse MG levels on fine AMR levels. Src/LinearSolvers/MLMG/AMReX_MLCellABecLap.cpp commit eb9b4242a07028a77bd95986f5a5686c81dce428 Author: Andrew Myers Date: Thu Feb 4 12:11:37 2021 -0800 Fix a circular include involving PODVector. (#1776) Src/Base/AMReX_PODVector.H commit 5e08d985d3a91ab8ec10d9aec8c318b38ff10849 Author: Andrew Myers Date: Thu Feb 4 12:09:15 2021 -0800 add missing template arguments to the particle HDF5 IO methods (#1777) Src/Particle/AMReX_ParticleHDF5.H commit f8808fcb56e172f9548954240468cf3d18223851 Author: Weiqun Zhang Date: Thu Feb 4 10:28:54 2021 -0800 Refinement Ratio of 4 Support in Nodal Solver (#1774) * Refinement Ratio of 4 Support in Nodal Solver This adds refinement ratio of 4 support in non-EB nodal linear solver. Roundoff errors are expected for multi-level solves with a refinement ratio of 2. * MacProjector refinement ratio of 4 bug Move setLevelBC to project and call it only if it has not been called. This fixes a refinement ratio of 4 bug, because setLevelBC should not be called before coarse/fine ratio is set. Src/Base/AMReX_Box.H Src/LinearSolvers/MLMG/AMReX_MLLinOp.cpp Src/LinearSolvers/MLMG/AMReX_MLMG.cpp Src/LinearSolvers/MLMG/AMReX_MLMG_1D_K.H Src/LinearSolvers/MLMG/AMReX_MLMG_2D_K.H Src/LinearSolvers/MLMG/AMReX_MLMG_3D_K.H Src/LinearSolvers/MLMG/AMReX_MLNodeLap_1D_K.H Src/LinearSolvers/MLMG/AMReX_MLNodeLap_2D_K.H Src/LinearSolvers/MLMG/AMReX_MLNodeLap_3D_K.H Src/LinearSolvers/MLMG/AMReX_MLNodeLap_K.H Src/LinearSolvers/MLMG/AMReX_MLNodeLaplacian.H Src/LinearSolvers/MLMG/AMReX_MLNodeLaplacian.cpp Src/LinearSolvers/MLMG/AMReX_MLNodeLinOp.H Src/LinearSolvers/MLMG/AMReX_MLNodeLinOp.cpp Src/LinearSolvers/Projections/AMReX_MacProjector.H Src/LinearSolvers/Projections/AMReX_MacProjector.cpp Tutorials/LinearSolvers/NodalPoisson/MyTest.H Tutorials/LinearSolvers/NodalPoisson/MyTest.cpp commit c686e458ffccfa0032f231753c65ea514df8f122 Author: Candace Gilet Date: Wed Feb 3 23:48:45 2021 -0500 For FillPatchTwoLevels, BCs should be set on the coarse box (#1772) since they are only used on the coarse level (when computing slopes). This corrects certain cases where a level of refinement is very close to, but not actually touching an ext_dir or hoextrap boundary. Src/Amr/AMReX_AmrLevel.cpp Src/AmrCore/AMReX_FillPatchUtil_I.H commit ebf64aa13ff7d57013473bdd4df0acf7be144193 Author: Michael Zingale Date: Wed Feb 3 23:34:09 2021 -0500 remove f90cat.py (#1775) it was only ever used with Castro as an experiment Tools/F_scripts/f90cat.py commit c9265f4c85057e94b39db780df42491f20137d19 Author: kngott <30483578+kngott@users.noreply.github.com> Date: Wed Feb 3 16:06:45 2021 -0800 Make.unknown MPI Detection Update (#1770) Simplify/robust-ify Make.unknown's MPI detection. Tools/GNUMake/sites/Make.unknown commit 39fe5870a75b7dfd9f3b0cdc531ab63f92a9f15b Author: Axel Huebl Date: Wed Feb 3 13:45:56 2021 -0800 MSVC: Proper __cplusplus macro (#1773) Modern MSVC versions finally set the __cplusplus macro correctly, but we need to request this with an additional flag. Tools/CMake/AMReX_Config.cmake commit b00475d69aec7304d18129d6b3ce9c2a2f4d835a Author: Michael Zingale Date: Tue Feb 2 16:03:12 2021 -0500 remove any trailing / from HIP_PATH via realpath (#1771) Tools/GNUMake/comps/hip.mak commit c68493447deeaf1675a6cdfef323d0adb62f4bbe Author: Andrew Myers Date: Tue Feb 2 08:55:09 2021 -0800 Use ParIter instead of MakeMFIter in WriteBinaryParticleDataAsync. (#1767) Src/Particle/AMReX_WriteBinaryParticleData.H commit d2270fab703098760806d9b9297a7502e8918733 Author: Axel Huebl Date: Tue Feb 2 08:35:34 2021 -0800 IntVect::toArray() (#1768) ## Summary Add a ::toArray() conversion member to IntVect that can be used to convert back to an array type and cast the result on the way, e.g. to unsigned. ## Additional background Functions in https://github.com/ECP-WarpX/WarpX/pull/1660 Src/Base/AMReX_IntVect.H commit 4a425335e40fbae69261987e8a97bc094712fb9f Author: Weiqun Zhang Date: Mon Feb 1 08:37:34 2021 -0800 Update CHANGES for 21.02 (#1766) CHANGES commit 466e3071713ced99cdfce4426c283d75da21b2c5 Author: Andrew Myers Date: Sat Jan 30 13:21:11 2021 -0800 Add a Github action that tries to build tutorials with SP particles and DP mesh. (#1764) Currently this combination is only caught by the Nyx CI. .github/workflows/linux.yml commit 302fad36faaa8f5bb65c207c608ca332b1a07116 Author: Candace Gilet Date: Sat Jan 30 16:18:05 2021 -0500 Abort when MLMG is detected as failing regardless of verbosity setting. (#1762) Move Abort() outside if( verbose >0 ) block. Src/LinearSolvers/MLMG/AMReX_MLMG.cpp commit f649f04fef6922374f2d2b31474ac87eda35a5f6 Author: kngott <30483578+kngott@users.noreply.github.com> Date: Sat Jan 30 13:16:45 2021 -0800 Anyof (#1757) * DPC++ versions of AnyOf. Written to match CUDA/HIP version. Performance not explicitly tested. * Also, adjustment to AMREX_DEVICE_PRINTF to remove the need for AMREX_DEVICE_COMPILE. Src/Base/AMReX_GpuPrint.H Src/Base/AMReX_Reduce.H Tests/GPU/AnyOf/GNUmakefile Tests/GPU/AnyOf/Make.package Tests/GPU/AnyOf/inputs Tests/GPU/AnyOf/main.cpp commit 896916428ca28b012c7c4ed53812898f2c26f815 Author: jmsexton03 Date: Fri Jan 29 13:32:24 2021 -0800 Use ParticleReal for particle interpolation (#1763) Src/Particle/AMReX_TracerParticles.cpp commit adbb5695469183897014c05824bd44772252f398 Author: Andrew Myers Date: Fri Jan 29 11:59:32 2021 -0800 Fix new logic error with aggregation_type. (#1761) Src/Particle/AMReX_ParticleContainerBase.cpp commit ef3e68bbc1380065f9515ea5c0236fe2f18cda5f Author: Andrew Myers Date: Fri Jan 29 11:51:30 2021 -0800 Remove obsolete interpolation functions from AMReX (#1754) Src/Particle/AMReX_Particle.H Src/Particle/AMReX_TracerParticle_mod_K.H Src/Particle/AMReX_TracerParticles.cpp commit 5df10f661e060de45fca677026325fc278b5c70c Author: Andrew Myers Date: Fri Jan 29 11:06:53 2021 -0800 Move more PC methods that don't need the template parameters into base class. (#1760) Src/Particle/AMReX_ParticleContainerBase.H Src/Particle/AMReX_ParticleContainerBase.cpp Src/Particle/AMReX_ParticleContainerI.H Src/Particle/AMReX_Particles.H commit f45ede1d49940035f84dc9a8562fef0a2ad3b720 Author: Andrew Myers Date: Thu Jan 28 13:28:26 2021 -0800 make pc.Geom always refer to the particle one. (#1470) Src/Particle/AMReX_Particles.H commit c143555e74e4096ec25400369651bf884b8561b3 Author: Andrew Myers Date: Wed Jan 27 16:04:37 2021 -0800 Fix 'potential null dereference warning' in StructOfArrays (#1759) Src/Particle/AMReX_StructOfArrays.H commit 2cd12635b7d7c2d41ee5ebce07c8003ef7fbe1b0 Author: kngott <30483578+kngott@users.noreply.github.com> Date: Wed Jan 27 14:31:30 2021 -0800 Gpu Docs Update First Pass (#1758) First pass at updating the GPU docs. More detailed updates to follow. Docs/sphinx_documentation/source/GPU.rst Docs/sphinx_documentation/source/GPU_Chapter.rst commit 0b79c5f7d855ed0078b63bfe34cf982c0956f06e Author: Andrew Myers Date: Wed Jan 27 09:29:29 2021 -0800 Move DataPrefix and Version... (#1756) * DataPrefix and Version do not need the template arguments, so moving them to a base class. Src/Particle/AMReX_Particle.H Src/Particle/AMReX_ParticleContainerBase.H Src/Particle/AMReX_ParticleContainerBase.cpp Src/Particle/AMReX_ParticleHDF5.H Src/Particle/AMReX_ParticleIO.H Src/Particle/AMReX_ParticleInit.H Src/Particle/AMReX_Particles.H Src/Particle/AMReX_WriteBinaryParticleData.H Src/Particle/CMakeLists.txt Src/Particle/Make.package commit b95a0f30003d6eb97fb6e33a5fde5a902221495d Author: Andrew Myers Date: Wed Jan 27 09:19:47 2021 -0800 Remove unused move random methods. (#1755) These methods are not used, and if needed, they should be implemented at the application level. Src/Particle/AMReX_ParticleContainerI.H Src/Particle/AMReX_Particles.H commit 02c52cda1b70c77384807d898a7a5c1f705c96c4 Author: Andrew Myers Date: Tue Jan 26 08:50:02 2021 -0800 Add IntVect versions of numParticlesOutOfRange. (#1751) Src/Particle/AMReX_ParticleUtil.H commit 9896c7643f72bc091b119e63ab6f46e5d1a058d2 Author: Axel Huebl Date: Mon Jan 25 16:52:20 2021 -0800 Fix ParIter Ctor (#1752) With user-defined allocators, this lead to: ``` Src/Particle/AMReX_ParIter.H:114:84: error: type ‘amrex::ParIterBase’ is not a direct base of ‘amrex::ParIter<0, 0, 4, 0, amrex::PinnedArenaAllocator>’ 114 | : ParIterBase(pc,level) | ``` in non-const contexts. Src/Particle/AMReX_ParIter.H commit b560557c4f5876337c0eb47df82f0ead37cf52b6 Author: asalmgren Date: Mon Jan 25 13:40:42 2021 -0800 Improve the interpolation from cell centroids to face centroids: (#1750) 1) if the centroids are aligned tangentially to the face but not at equal distance to the face, use just those two cells for interpolation 2) in the case where the face centroid is at 0 but the cell centroids are not aligned, instead of testing on an area fraction tangential to the face to decide which way to go, in 2d, now test on the minimum volume for each option and choose the option with the larger minimum volume -- this should avoid using covered or very small cells. (3D already has logic to avoid using covered cells) Src/EB/AMReX_EBMultiFabUtil_2D_C.H Src/EB/AMReX_EBMultiFabUtil_3D_C.H commit 3069f1ba4dd899f40ac6092fff2916466af8d980 Author: asalmgren Date: Mon Jan 25 13:40:25 2021 -0800 Fix EB slopes in 3D when two of the domain boundaries with ext_dir bc's meet at an edge -- (#1749) this case was handled incorrectly before Src/EB/AMReX_EB_slopes_K.H commit e93c5705a6c013ffc602a16e877d22ea3afbf1fc Author: Don E. Willcox Date: Sun Jan 24 20:23:00 2021 -0800 In ParticleToMesh and MeshToParticle, use nGrowVect() to accommodate different a different number of grow cells in each dimension. (#1748) Src/Particle/AMReX_ParticleMesh.H commit 162bca40a4ecec77a5d8d1df68b1864bfd15ec6a Author: asalmgren Date: Sat Jan 23 12:50:54 2021 -0800 Add additional tests when interpolating from cell centroids to face centroids (#1746) when the face centroid is exactly at the center of the face we need an additional test to know which way we can look transversely and not hit covered cells. Src/EB/AMReX_EBMultiFabUtil_2D_C.H Src/EB/AMReX_EBMultiFabUtil_3D_C.H commit cbbd1c5bd623fc59b67a96e1d5cd5339f2b24cbb Author: Weiqun Zhang Date: Fri Jan 22 09:40:57 2021 -0800 Fix single precision build of CNS (#1745) Tutorials/GPU/CNS/Source/main.cpp commit e2b1108dde79e258f4d06e584e745e0ffd2bbd5f Author: Andrew Myers Date: Thu Jan 21 17:37:54 2021 -0800 Add missing template parameters in ParIter.H. (#1744) Src/Particle/AMReX_ParIter.H commit 1656d55a6e001f139f88c5ada93068a7b862b59c Author: Andrew Myers Date: Thu Jan 21 15:23:39 2021 -0800 Template copy/add particles based on src type. (#1741) Src/Particle/AMReX_ParticleContainerI.H Src/Particle/AMReX_ParticleTile.H Src/Particle/AMReX_Particles.H commit a63941a8f13fd972b860297de2132185a739418e Author: Axel Huebl Date: Thu Jan 21 15:00:09 2021 -0800 HIP: New Warp Size Macro (#1742) Use new the HIP macro for the warp size on various AMD devices. Src/Base/AMReX_GpuDevice.H commit c2beebd492f1efb202d09cef2f62e5283998e1b7 Author: Andrew Myers Date: Wed Jan 20 17:21:05 2021 -0800 Add allocator template parameter to ParticleContainer (#1416) Src/AmrCore/AMReX_AmrParticles.H Src/Base/AMReX_TypeTraits.H Src/Particle/AMReX_ParIter.H Src/Particle/AMReX_ParticleContainerI.H Src/Particle/AMReX_ParticleHDF5.H Src/Particle/AMReX_ParticleIO.H Src/Particle/AMReX_ParticleInit.H Src/Particle/AMReX_ParticleTile.H Src/Particle/AMReX_Particles.H commit bcf034a796031d362b85dba9c88fad24f30b36e6 Author: asalmgren Date: Wed Jan 20 15:21:05 2021 -0800 Reduce compiler warnings (#1740) Co-authored-by: Andrew Myers Src/EB/AMReX_EB_slopes_K.H Src/Particle/AMReX_NeighborList.H commit b2c1b6077a07a23201aa65156ef2a1033943482f Author: Andrew Myers Date: Wed Jan 20 11:44:24 2021 -0800 Fix some new warnings about 'pointless comparisons' with 0 for unsigned types. (#1737) Src/Particle/AMReX_StructOfArrays.H commit 1775ce65c769d0502f727aa1f1955366a172ff2c Author: drangara <69211175+drangara@users.noreply.github.com> Date: Wed Jan 20 14:16:39 2021 -0500 Add ability to calculate slopes on face centroids (#1707) And apply the explicit stencil when computing the laplacian These changes allow the computation of EB viscous laplacian operator when phi is defined on cell centroids by using the Least squares method for calculating slopes on face centroids. Currently only works when Dirichlet BC is set. Src/EB/AMReX_EB_LeastSquares_2D_K.H Src/EB/AMReX_EB_LeastSquares_3D_K.H Src/LinearSolvers/MLMG/AMReX_MLEBABecLap.cpp Src/LinearSolvers/MLMG/AMReX_MLEBABecLap_2D_K.H Src/LinearSolvers/MLMG/AMReX_MLEBABecLap_3D_K.H Tests/LinearSolvers/LeastSquares/GNUmakefile Tests/LinearSolvers/LeastSquares/Make.package Tests/LinearSolvers/LeastSquares/MyEB.H Tests/LinearSolvers/LeastSquares/MyTest.H Tests/LinearSolvers/LeastSquares/MyTest.cpp Tests/LinearSolvers/LeastSquares/README.md Tests/LinearSolvers/LeastSquares/initData.cpp Tests/LinearSolvers/LeastSquares/initEB.cpp Tests/LinearSolvers/LeastSquares/initPoiseuilleData.cpp Tests/LinearSolvers/LeastSquares/initPoiseuilleDataFor2D.cpp Tests/LinearSolvers/LeastSquares/initPoiseuilleDataFor3D.cpp Tests/LinearSolvers/LeastSquares/inputs.2d.askew-x Tests/LinearSolvers/LeastSquares/inputs.2d.askew-y Tests/LinearSolvers/LeastSquares/inputs.2d.base Tests/LinearSolvers/LeastSquares/inputs.2d.fullyrotated Tests/LinearSolvers/LeastSquares/inputs.2d.other Tests/LinearSolvers/LeastSquares/inputs.3d.poiseuille.aligned.xy-x Tests/LinearSolvers/LeastSquares/inputs.3d.poiseuille.aligned.xy-y Tests/LinearSolvers/LeastSquares/inputs.3d.poiseuille.aligned.xz-x Tests/LinearSolvers/LeastSquares/inputs.3d.poiseuille.aligned.xz-z Tests/LinearSolvers/LeastSquares/inputs.3d.poiseuille.aligned.yz-y Tests/LinearSolvers/LeastSquares/inputs.3d.poiseuille.aligned.yz-z Tests/LinearSolvers/LeastSquares/inputs.3d.poiseuille.askew-all Tests/LinearSolvers/LeastSquares/inputs.3d.poiseuille.askew-xy Tests/LinearSolvers/LeastSquares/inputs.3d.poiseuille.askew-xz Tests/LinearSolvers/LeastSquares/inputs.3d.poiseuille.askew-yz Tests/LinearSolvers/LeastSquares/main.cpp commit 9bda4f80dea74dcef0ccd4cc9dfa60c693613228 Author: Weiqun Zhang Date: Wed Jan 20 09:53:21 2021 -0800 HIP indirect function and kernel fusing (#1739) Add a GNU Make flag, HIP_INDIRECT_FUNCTION=[TRUE|FALSE], and modify the Gpu kernel fusing test. Currently HIP does not support indirect function calls via device function pointers, which our GPU fusing implementation depends on. Src/Base/AMReX_GpuFuse.H Src/Base/AMReX_GpuFuse.cpp Src/Base/AMReX_GpuLaunch.H Src/Base/AMReX_GpuLaunchFunctsG.H Src/Base/AMReX_GpuLaunchMacrosG.H Src/Base/AMReX_Reduce.H Tests/GPU/Fuse/GNUmakefile Tests/GPU/Fuse/main.cpp Tools/GNUMake/Make.defs commit 9437e36505e115da96cebebab973d5499f357bb2 Author: Robert Maynard Date: Tue Jan 19 13:39:04 2021 -0500 When using CMake 3.15+ allow for MSVC runtime library changes (#1736) Enabling of CMP0091 needs to go before the first `project` call as that computes if MSVC_RUNTIME_LIBRARY information is used CMakeLists.txt Tools/CMake/AMReX_Config.cmake commit 5791c90593389dbf23502d10c52f01b35e77bfa6 Author: Weiqun Zhang Date: Mon Jan 18 20:49:38 2021 -0800 RoundRobin: option to not sort processes (#1721) Add an optional parameter `sort` to RoundRobin constructors. This makes it consistent with other DistributionMapping strategies. And this fixes an incorrect behavior when RoundRobin is used by KnapSack that has been given `sort=false`. This should be able to fix the hanging issue observed in WarpX simulations with load balancing enabled and more MPI processes than the number of boxes. Src/Base/AMReX_DistributionMapping.H Src/Base/AMReX_DistributionMapping.cpp commit 5d3e31a08af987cf9e127ab07375483c74724743 Author: Weiqun Zhang Date: Sun Jan 17 12:02:00 2021 -0800 EB: edge centroid (#1730) Add edge centroid to the database and make it available via EBFArrayBoxFactory. Src/EB/AMReX_EB2_2D_C.H Src/EB/AMReX_EB2_2D_C.cpp Src/EB/AMReX_EB2_3D_C.H Src/EB/AMReX_EB2_3D_C.cpp Src/EB/AMReX_EB2_C.H Src/EB/AMReX_EB2_GeometryShop.H Src/EB/AMReX_EB2_IF_Rotation.H Src/EB/AMReX_EB2_Level.H Src/EB/AMReX_EB2_Level.cpp Src/EB/AMReX_EB2_ND_C.cpp Src/EB/AMReX_EBDataCollection.H Src/EB/AMReX_EBDataCollection.cpp Src/EB/AMReX_EBFabFactory.H Src/EB/AMReX_EB_utils.cpp Src/EB/CMakeLists.txt Src/EB/Make.package commit 8b123354e9929bc2515b032c2069e2d924e88e59 Author: Weiqun Zhang Date: Sun Jan 17 11:58:36 2021 -0800 No Fortran needed for building plotfile tools (#1711) Tools/Plotfile/GNUmakefile commit d2ff769fa0ad57243a3c9d0010eb68e2ab049005 Author: Emmanuel Motheau Date: Sun Jan 17 11:57:30 2021 -0800 fix RegridOnly to regrid even with single level grids (#1722) Src/Amr/AMReX_Amr.cpp commit 832ce5e8bcdd224c8946392dfaeb8915e0e02eed Author: Andy Nonaka Date: Sat Jan 16 11:54:48 2021 -0800 Consolidate agglomerate (#1733) a few lines in the docs about consolidation and agglomeration options in the linear solvers Docs/sphinx_documentation/source/LinearSolvers.rst commit eab09708761036232dd650c5ee724af7f28249f4 Author: Axel Huebl Date: Sat Jan 16 11:52:55 2021 -0800 Doc: Break into Debugger (#1732) Describe how to disable AMReX' signal and backtrace handling. Thanks to @sayerhs for pointing me to the right place (again :) ). Docs/sphinx_documentation/source/Basics.rst commit f6b0abc7c32368bafffe8e81a2de18c492c8971a Author: Max Katz Date: Sat Jan 16 11:50:44 2021 -0800 Only add Hypre interfaces to MLMG in 2D/3D (#1734) This is necessary for compiling with both Hypre and MLMG in 1D. Src/LinearSolvers/MLMG/AMReX_MLCellABecLap.H Src/LinearSolvers/MLMG/AMReX_MLCellABecLap.cpp Src/LinearSolvers/MLMG/AMReX_MLEBABecLap.H Src/LinearSolvers/MLMG/AMReX_MLEBABecLap.cpp Src/LinearSolvers/MLMG/AMReX_MLLinOp.H Src/LinearSolvers/MLMG/AMReX_MLMG.H Src/LinearSolvers/MLMG/AMReX_MLMG.cpp Src/LinearSolvers/MLMG/AMReX_MLNodeLaplacian.H Src/LinearSolvers/MLMG/AMReX_MLNodeLaplacian.cpp Src/LinearSolvers/MLMG/AMReX_MLNodeLinOp.H Src/LinearSolvers/MLMG/AMReX_MLNodeLinOp.cpp Src/LinearSolvers/MLMG/AMReX_MLNodeTensorLap_2D_K.H Src/LinearSolvers/MLMG/AMReX_MLNodeTensorLap_3D_K.H Src/LinearSolvers/MLMG/AMReX_MLNodeTensorLaplacian.H Src/LinearSolvers/MLMG/AMReX_MLNodeTensorLaplacian.cpp commit 889af2f8a8e1ec191627f1fb5b28ac9ed6d5aa9a Author: Max Katz Date: Fri Jan 15 18:13:37 2021 -0800 Don't include Hypre files in GNU make for 1D (#1731) These files cannot be compiled in 1D, so there is no point in adding them. CMake treats this as fatal, but for Castro we would like to use the AMReX Hypre GNU make package even without the actual interfaces. Src/Extern/HYPRE/Make.package commit 549bb3d5675f7cac9fd2d116f8cde1daaa477a09 Author: Andrew Myers Date: Fri Jan 15 17:05:38 2021 -0800 Always assert that tiling is false in RedistributeGPU (#1729) Src/Particle/AMReX_ParticleContainerI.H commit 55b62cb42fb6afcfd2e78c8d68fab80c39180a9a Author: Axel Huebl Date: Fri Jan 15 16:33:01 2021 -0800 Define: _OPENMP -> AMREX_USE_OMP (#1560) Replace the define `_OPENMP` with `AMREX_USE_OMP` for all parallel "backend" implementations and control of MFIter loops. This avoids accidentially enabling OpenMP for the parallel compute components, i.e. when a users explicitly uses OpenMP in auxiliary or implicit dependent functionality, but does not request to also OpenMP-parallelize those sections. Co-authored-by: Weiqun Zhang Docs/sphinx_documentation/source/AmrCore.rst Docs/sphinx_documentation/source/Basics.rst Docs/sphinx_documentation/source/GPU.rst Src/Amr/AMReX_Amr.cpp Src/Amr/AMReX_AmrLevel.cpp Src/Amr/AMReX_Extrapolater.cpp Src/Amr/AMReX_StateData.cpp Src/Amr/AMReX_StateDescriptor.cpp Src/AmrCore/AMReX_AmrCore.cpp Src/AmrCore/AMReX_ErrorList.cpp Src/AmrCore/AMReX_FillPatchUtil.H Src/AmrCore/AMReX_FillPatchUtil.cpp Src/AmrCore/AMReX_FillPatchUtil_I.H Src/AmrCore/AMReX_FluxRegister.cpp Src/AmrCore/AMReX_TagBox.cpp Src/Base/AMReX.cpp Src/Base/AMReX_BLBackTrace.H Src/Base/AMReX_BLBackTrace.cpp Src/Base/AMReX_BLProfiler.cpp Src/Base/AMReX_BaseFab.H Src/Base/AMReX_BaseFab.cpp Src/Base/AMReX_BaseUmap.H Src/Base/AMReX_BoxArray.H Src/Base/AMReX_BoxArray.cpp Src/Base/AMReX_BoxList.cpp Src/Base/AMReX_DistributionMapping.cpp Src/Base/AMReX_Extension.H Src/Base/AMReX_FBI.H Src/Base/AMReX_FabArray.H Src/Base/AMReX_FabArrayBase.H Src/Base/AMReX_FabArrayBase.cpp Src/Base/AMReX_FabArrayCommI.H Src/Base/AMReX_FabArrayUtility.H Src/Base/AMReX_Geometry.H Src/Base/AMReX_Geometry.cpp Src/Base/AMReX_GpuAtomic.H Src/Base/AMReX_GpuReduce.H Src/Base/AMReX_GpuUtility.cpp Src/Base/AMReX_MFIter.H Src/Base/AMReX_MFIter.cpp Src/Base/AMReX_MemPool.cpp Src/Base/AMReX_MultiFab.cpp Src/Base/AMReX_MultiFabUtil.H Src/Base/AMReX_MultiFabUtil.cpp Src/Base/AMReX_NonLocalBCImpl.H Src/Base/AMReX_OpenMP.H Src/Base/AMReX_PCI.H Src/Base/AMReX_ParallelDescriptor.H Src/Base/AMReX_ParallelDescriptor.cpp Src/Base/AMReX_PhysBCFunct.H Src/Base/AMReX_Print.H Src/Base/AMReX_Random.cpp Src/Base/AMReX_Reduce.H Src/Base/AMReX_TinyProfiler.cpp Src/Base/AMReX_Utility.cpp Src/Base/AMReX_iMultiFab.cpp Src/Base/AMReX_omp_mod.F90 Src/Boundary/AMReX_BndryRegister.cpp Src/Boundary/AMReX_FabSet.cpp Src/Boundary/AMReX_InterpBndryData.cpp Src/Boundary/AMReX_MultiMask.H Src/Boundary/AMReX_MultiMask.cpp Src/Boundary/AMReX_YAFluxRegister.cpp Src/EB/AMReX_EB2_Level.H Src/EB/AMReX_EB2_Level.cpp Src/EB/AMReX_EBAmrUtil.cpp Src/EB/AMReX_EBCellFlag.cpp Src/EB/AMReX_EBFluxRegister.cpp Src/EB/AMReX_EBMultiFabUtil.cpp Src/EB/AMReX_EB_utils.H Src/EB/AMReX_EB_utils.cpp Src/EB/AMReX_MultiCutFab.cpp Src/EB/AMReX_algoim.cpp Src/Extern/HYPRE/AMReX_HypreABecLap.cpp Src/Extern/HYPRE/AMReX_HypreABecLap2.cpp Src/Extern/HYPRE/AMReX_HypreABecLap3.cpp Src/Extern/HYPRE/AMReX_HypreNodeLap.cpp Src/Extern/PETSc/AMReX_PETSc.cpp Src/Extern/ProfParser/AMReX_BLProfStats.cpp Src/Extern/ProfParser/AMReX_CommProfStats.cpp Src/Extern/ProfParser/AMReX_RegionsProfStats.cpp Src/Extern/hpgmg/BL_HPGMG.cpp Src/F_Interfaces/Octree/AMReX_octree_fi.cpp Src/LinearSolvers/MLMG/AMReX_MLABecLaplacian.cpp Src/LinearSolvers/MLMG/AMReX_MLALaplacian.cpp Src/LinearSolvers/MLMG/AMReX_MLCGSolver.cpp Src/LinearSolvers/MLMG/AMReX_MLCellABecLap.cpp Src/LinearSolvers/MLMG/AMReX_MLCellLinOp.cpp Src/LinearSolvers/MLMG/AMReX_MLEBABecLap.cpp Src/LinearSolvers/MLMG/AMReX_MLEBTensorOp.cpp Src/LinearSolvers/MLMG/AMReX_MLMG.cpp Src/LinearSolvers/MLMG/AMReX_MLMGBndry.cpp Src/LinearSolvers/MLMG/AMReX_MLNodeLaplacian.cpp Src/LinearSolvers/MLMG/AMReX_MLNodeLinOp.cpp Src/LinearSolvers/MLMG/AMReX_MLNodeTensorLaplacian.cpp Src/LinearSolvers/MLMG/AMReX_MLPoisson.cpp Src/LinearSolvers/MLMG/AMReX_MLTensorOp.cpp Src/LinearSolvers/Projections/AMReX_NodalProjector.cpp Src/Particle/AMReX_NeighborParticlesCPUImpl.H Src/Particle/AMReX_NeighborParticlesI.H Src/Particle/AMReX_ParIter.H Src/Particle/AMReX_Particle.H Src/Particle/AMReX_ParticleContainerI.H Src/Particle/AMReX_ParticleMesh.H Src/Particle/AMReX_ParticleReduce.H Src/Particle/AMReX_ParticleUtil.H Src/Particle/AMReX_Particles.H Src/Particle/AMReX_TracerParticles.cpp Tests/FillBoundaryComparison/main.cpp Tests/LinearSolvers/CellEB2/MyTest.cpp Tests/LinearSolvers/CellOverset/MyTest.cpp Tests/LinearSolvers/MLMG/init_prob.cpp Tests/LinearSolvers/TensorOverset/MyTest.cpp Tests/Particles/ParticleIterator/main.cpp Tools/Postprocessing/C_Src/IntegrateComp.cpp Tutorials/Amr/Advection_AmrCore/Source/AdvancePhiAllLevels.cpp Tutorials/Amr/Advection_AmrCore/Source/AdvancePhiAtLevel.cpp Tutorials/Amr/Advection_AmrCore/Source/AmrCoreAdv.H Tutorials/Amr/Advection_AmrCore/Source/AmrCoreAdv.cpp Tutorials/Amr/Advection_AmrCore/Source/DefineVelocity.cpp Tutorials/Amr/Advection_AmrLevel/Source/AmrLevelAdv.H Tutorials/Amr/Advection_AmrLevel/Source/AmrLevelAdv.cpp Tutorials/Blueprint/CellSortedParticles/CellSortedPC.cpp Tutorials/EB/CNS/Source/CNS.cpp Tutorials/EB/CNS/Source/CNS_advance.cpp Tutorials/EB/Poisson/Poisson.cpp Tutorials/GPU/CNS/Source/CNS.cpp Tutorials/GPU/EBCNS/Source/CNS.cpp Tutorials/LinearSolvers/ABecLaplacian_C/initProb.cpp Tutorials/LinearSolvers/MultiComponent/MCNodalLinOp.cpp Tutorials/LinearSolvers/NodalPoisson/MyTest.cpp Tutorials/LinearSolvers/NodeTensorLap/MyTest.cpp Tutorials/Particles/CellSortedParticles/CellSortedPC.cpp commit 2536f1ae9f7b7e73bbeee71d5642aa22e02432fc Author: Robert Maynard Date: Fri Jan 15 14:12:12 2021 -0500 When using CMake 3.15 allow for MSVC runtime library changes (#1724) Tools/CMake/AMReX_Config.cmake commit 40dcbf1f69cb81fc875e4e1db49d33c3292bbe9f Author: Mark Meredith Date: Fri Jan 15 12:04:44 2021 -0500 fix -Werror=sign-compare (#1699) Src/Particle/AMReX_NeighborParticlesCPUImpl.H Src/Particle/AMReX_NeighborParticlesI.H Src/Particle/AMReX_StructOfArrays.H commit 633c971c3258d4a662f91d5620e44834bf8d4c92 Author: asalmgren Date: Thu Jan 14 17:25:58 2021 -0800 Quiet the compiler warnings coming from EB/AMReX_EB_slopes_K.H (#1728) Src/EB/AMReX_EB_slopes_K.H commit fd6f5ce3c255d449d5de3d664269d64d069cbdc9 Author: mic84 Date: Thu Jan 14 15:55:57 2021 -0800 CMake: link tests exes against alias library (#1727) Tests/CMakeLists.txt commit 873c4801ad9b1f262a8865f6b9fe198a4ead29ca Author: Jon Rood Date: Thu Jan 14 15:36:17 2021 -0700 Update Make.nrel to accomodate MPT MPI with CUDA enabled (#1726) For our main computer at NREL, Eagle, we mostly use HPE's MPT for MPI. Until these changes, CUDA did not work with MPT when using the GNU makefiles. Tools/GNUMake/Make.defs Tools/GNUMake/sites/Make.nrel commit bcbdb65b525644fcb1cd61dc57d4e77c19b8eac6 Author: Weiqun Zhang Date: Thu Jan 14 13:38:11 2021 -0800 Fix CNS tutorial for HIP (#1725) Use AMREX_GPU_HOST_DEVICE instead of AMREX_GPU_DEVICE because of we cannot detect the lambda type in HIP. Tutorials/GPU/CNS/Source/CNS.cpp commit 10f3a2348e8cb4d0f52e90d273f70426ef1fcbbc Author: Weiqun Zhang Date: Thu Jan 14 11:14:01 2021 -0800 Cell-centered Hypre and Petsc setup on GPU (#1709) Do cell-centered hypre and Petsc solver setup on GPU. Docs/sphinx_documentation/source/LinearSolvers.rst Src/Base/AMReX.cpp Src/Base/AMReX_BaseFabUtility.H Src/Extern/HYPRE/AMReX_Habec_2D_K.H Src/Extern/HYPRE/AMReX_Habec_3D_K.H Src/Extern/HYPRE/AMReX_Habec_K.H Src/Extern/HYPRE/AMReX_Hypre.H Src/Extern/HYPRE/AMReX_HypreABecLap.H Src/Extern/HYPRE/AMReX_HypreABecLap.cpp Src/Extern/HYPRE/AMReX_HypreABecLap2.H Src/Extern/HYPRE/AMReX_HypreABecLap2.cpp Src/Extern/HYPRE/AMReX_HypreABecLap3.H Src/Extern/HYPRE/AMReX_HypreABecLap3.cpp Src/Extern/HYPRE/AMReX_HypreNodeLap.cpp Src/Extern/PETSc/AMReX_PETSc.H Src/Extern/PETSc/AMReX_PETSc.cpp Src/LinearSolvers/MLMG/AMReX_MLMG.H Tools/GNUMake/packages/Make.hypre Tutorials/LinearSolvers/ABecLaplacian_C/initProb.cpp commit dd0a3ceb3c62f68ad096b0692352acc776c352ae Author: mic84 Date: Wed Jan 13 12:59:28 2021 -0800 Tests: replace D_DECL with AMREX_D_DECL (#1723) Tests/HDF5Benchmark/main.cpp Tests/Particles/AssignMultiLevelDensity/main.cpp Tests/Particles/AsyncIO/main.cpp Tests/Particles/GhostsAndVirtuals/main.cpp Tests/Particles/Intersection/main.cpp Tests/Particles/Redistribute/main.cpp Tests/Particles/SparseBins/main.cpp commit 07488476533e1bf4731f1bd7c17c1b14c0fd13d2 Author: Matt Larsen Date: Mon Jan 11 13:35:37 2021 -0800 fixing assert (#1720) Co-authored-by: Matt Larsen Src/Extern/Conduit/AMReX_Conduit_Blueprint.cpp commit 95661e553fbd92c4d41ffa88438272a3ca4f385f Author: Andrew Myers Date: Mon Jan 11 11:38:54 2021 -0800 Fix compiler warnings in CreateVirtualParticles (#1718) Src/Particle/AMReX_ParticleContainerI.H commit f35b856849f22909e2e4395ac73edb263b108434 Author: Andrew Myers Date: Mon Jan 11 11:17:50 2021 -0800 Fix compiler warnings in ParticleInit.H (#1719) Src/Particle/AMReX_ParticleInit.H commit cd56f543e5372230314f1f0acfb6566a754376ed Author: Weiqun Zhang Date: Sun Jan 10 12:46:38 2021 -0800 ParmParse::hasUnusedInputs & getUnusedInputs (#1716) Add functions to find if there are unused `ParmParse` parameters starting with `prefix.`. If `prefix` is empty, all unused `ParmParse` parameters are included. Src/Base/AMReX_ParmParse.H Src/Base/AMReX_ParmParse.cpp commit cf17e95632b8bc26107a9ecd9e8dcae75db47c4e Author: asalmgren Date: Sat Jan 9 13:39:48 2021 -0800 There is no longer a CVODE tutorial (#1715) Docs/sphinx_tutorials/source/CVODE_Tutorial.rst Docs/sphinx_tutorials/source/index.rst commit 4e7f200eb91b1d8689b38f31d2ea1e29535094c0 Author: mic84 Date: Fri Jan 8 17:00:27 2021 -0800 CMake: offer option to exclude Src/Amr from build (#1714) Docs/sphinx_documentation/source/BuildingAMReX.rst Src/CMakeLists.txt Tools/CMake/AMReXConfig.cmake.in Tools/CMake/AMReXOptions.cmake commit 42f045ea6d39b7a1f02b2a92612352cff585a9b7 Author: Weiqun Zhang Date: Fri Jan 8 16:45:55 2021 -0800 Always have Abort_device etc. defined for GPU build (#1712) It used to be these functions are defined only if NDEBUG is defined. However, the cmake build system does not store `NDEBUG` in `AMReX_Config.H. It's possible that AMReX is built with `NDEBUG` and `Abort_device` is not defined, but the user's code may be built without `NDEBUG`. Then `amrex::Abort` defined in `AMReX.H` would fail to compile. Always having these device functions defined can solve the issue. Src/Base/AMReX.H Src/Base/AMReX.cpp commit 147934bcca2551e24c16bb0b5d4a50575a8a2eec Author: Weiqun Zhang Date: Fri Jan 8 14:57:07 2021 -0800 Prefetch not supported on Windows (#1713) On-demand page migration is not supported on Windows. Src/Base/AMReX_BaseFab.H commit 7eb484bb8fdaec3c46f66921b9baefe7a3b00580 Author: Emmanuel Motheau Date: Fri Jan 8 13:10:39 2021 -0800 Update write plot file to ascii tool (#1700) A flag (fast = 1) has been added to the WritePlotfileToAscii tool to propose to export data via Amrex::Print() instead of the 4 i,j,k,n imbricated loops. Indeed, on large plotfiles, exporting data to an ascii file is extremely slow with the procedure originally implemented in the tool. Tools/Postprocessing/C_Src/WritePlotfileToASCII.cpp commit 3daa8ced947e315f8c3cf7506e2eec45c6f8fbf4 Author: Weiqun Zhang Date: Fri Jan 8 10:34:38 2021 -0800 Tweak some floating point comparisons (#1710) Src/AmrCore/AMReX_FillPatchUtil_I.H Src/Base/AMReX_RealBox.H Src/Base/AMReX_RealBox.cpp Src/EB/AMReX_EB_utils.cpp commit 6a773594766d842c866cd3cbb74f2fe6f09884d6 Author: Weiqun Zhang Date: Thu Jan 7 14:48:37 2021 -0800 Make bisect more robust (#1708) If the lower and upper bounds are almost equal, break out of the bisect iteration. Src/Base/AMReX_Algorithm.H Src/Base/AMReX_Geometry.cpp commit be30569c55ce9ecf1f6576b4d94cd9040b3d8490 Author: Weiqun Zhang Date: Thu Jan 7 14:17:12 2021 -0800 Support single precision build with configure (#1706) * Add new arguments `--single-precision` and `--single-precision-particles` to `configure`. * Add amrex::almostEqual for comparing floating point numbers. * Fix double to float conversion warnings. * Add CI testing single precision build .github/workflows/linux.yml Src/Amr/AMReX_Amr.cpp Src/Amr/AMReX_AmrLevel.cpp Src/Amr/AMReX_AuxBoundaryData.cpp Src/Amr/AMReX_StateData.H Src/Amr/AMReX_StateData.cpp Src/Amr/AMReX_extrapolater_3D_K.H Src/AmrCore/AMReX_AmrCore.cpp Src/AmrCore/AMReX_Cluster.H Src/AmrCore/AMReX_ErrorList.cpp Src/Base/AMReX_Algorithm.H Src/Base/AMReX_BaseFab.H Src/Base/AMReX_BoxList.cpp Src/Base/AMReX_COORDSYS_2D_C.H Src/Base/AMReX_CoordSys.cpp Src/Base/AMReX_DistributionMapping.cpp Src/Base/AMReX_FArrayBox.cpp Src/Base/AMReX_FilCC_C.cpp Src/Base/AMReX_Geometry.cpp Src/Base/AMReX_MFCopyDescriptor.cpp Src/Base/AMReX_MultiFab.cpp Src/Base/AMReX_ParallelDescriptor.cpp Src/Base/AMReX_RealBox.cpp Src/Base/AMReX_VisMF.cpp Src/Boundary/AMReX_InterpBndryData_3D_K.H Src/Boundary/AMReX_MacBndry.cpp Src/F_Interfaces/AmrCore/AMReX_FlashFluxRegister.cpp Src/LinearSolvers/MLMG/AMReX_MLABecLaplacian.cpp Src/LinearSolvers/MLMG/AMReX_MLALap_1D_K.H Src/LinearSolvers/MLMG/AMReX_MLALap_2D_K.H Src/LinearSolvers/MLMG/AMReX_MLALap_3D_K.H Src/LinearSolvers/MLMG/AMReX_MLCellABecLap.cpp Src/LinearSolvers/MLMG/AMReX_MLCellLinOp.cpp Src/LinearSolvers/MLMG/AMReX_MLLinOp_K.H Src/LinearSolvers/MLMG/AMReX_MLMG.H Src/LinearSolvers/MLMG/AMReX_MLMG.cpp Src/LinearSolvers/MLMG/AMReX_MLMGBndry.cpp Src/LinearSolvers/MLMG/AMReX_MLMG_1D_K.H Src/LinearSolvers/MLMG/AMReX_MLMG_2D_K.H Src/LinearSolvers/MLMG/AMReX_MLMG_3D_K.H Src/LinearSolvers/MLMG/AMReX_MLNodeLap_2D_K.H Src/LinearSolvers/MLMG/AMReX_MLNodeLap_3D_K.H Src/LinearSolvers/MLMG/AMReX_MLNodeLap_K.H Src/LinearSolvers/MLMG/AMReX_MLNodeLaplacian.H Src/LinearSolvers/MLMG/AMReX_MLNodeLaplacian.cpp Src/LinearSolvers/MLMG/AMReX_MLNodeTensorLap_2D_K.H Src/LinearSolvers/MLMG/AMReX_MLNodeTensorLap_3D_K.H Src/LinearSolvers/MLMG/AMReX_MLNodeTensorLaplacian.cpp Src/LinearSolvers/MLMG/AMReX_MLPoisson.cpp Src/LinearSolvers/MLMG/AMReX_MLPoisson_2D_K.H Src/LinearSolvers/MLMG/AMReX_MLTensorOp.cpp Src/LinearSolvers/MLMG/AMReX_MLTensor_2D_K.H Src/LinearSolvers/MLMG/AMReX_MLTensor_3D_K.H Src/LinearSolvers/Projections/AMReX_MacProjector.cpp Src/LinearSolvers/Projections/AMReX_NodalProjector.H Src/LinearSolvers/Projections/AMReX_NodalProjector.cpp Src/Particle/AMReX_ParticleContainerI.H Src/Particle/AMReX_ParticleHDF5.H Src/Particle/AMReX_ParticleIO.H Src/Particle/AMReX_ParticleInit.H Src/Particle/AMReX_TracerParticle_mod_K.H Src/Particle/AMReX_TracerParticles.cpp Tests/FillBoundaryComparison/main.cpp Tools/C_util/WritePlotFile.cpp Tools/libamrex/configure.py Tutorials/Amr/Advection_AmrCore/Source/main.cpp Tutorials/Amr/Advection_AmrLevel/Source/AmrLevelAdv.cpp Tutorials/Amr/Advection_AmrLevel/Source/main.cpp Tutorials/Basic/HeatEquation_EX1_C/Source/main.cpp Tutorials/Basic/HeatEquation_EX1_CF/Source/main.cpp Tutorials/Basic/HeatEquation_EX2_C/Source/main.cpp Tutorials/Basic/HeatEquation_EX2_CF/Source/main.cpp Tutorials/Basic/HeatEquation_EX3_C/Source/main.cpp Tutorials/Blueprint/HeatEquation_EX1_C/Source/main.cpp Tutorials/EB/CNS/Source/CNS_advance.cpp Tutorials/EB/CNS/Source/main.cpp Tutorials/GPU/CNS/Source/main.cpp Tutorials/GPU/EBCNS/Source/main.cpp Tutorials/LinearSolvers/MAC_Projection_EB/main.cpp Tutorials/LinearSolvers/Nodal_Projection_EB/main.cpp commit f84b7b891cfa55affc25426288b4d3881e4be2a7 Author: Andrew Myers Date: Wed Jan 6 15:34:03 2021 -0800 Fix logic in applying periodic bcs when subcycling is on (#1705) Src/Particle/AMReX_ParticleUtil.H commit 20119b405e406040637daff2f4a591670de9093d Author: Axel Huebl Date: Wed Jan 6 11:55:01 2021 -0800 CMake<3.18: Fix Linker Regression (#1701) Roll-back to explicit `-Wl,` linker flags for CMake<3.18 Tools/CMake/AMReX_Config.cmake commit 887a4ce2a3e995592d9dcfd4318682526c297546 Author: Weiqun Zhang Date: Wed Jan 6 10:13:23 2021 -0800 ParallelDescriptor::ReduceReal wrappers (#1703) Src/Base/AMReX_ParallelDescriptor.H Src/Base/AMReX_ParallelDescriptor.cpp commit c36b253b295032f83e13edd9de78c40e0e660a10 Author: Luca Fedeli Date: Tue Jan 5 17:47:19 2021 +0100 Fix several warnings related to double-->float conversions (#1698) ## Summary I tried to compile `WarpX` in single precision and I noticed several warnings related to casts from `double` to `float` in `AMReX`. This PR should fix several of these warnings. ## Additional background I compiled `WarpX` with `make DIM=3 PRECISION=FLOAT USE_SINGLE_PRECISION_PARTICLES=TRUE` using the compiler `g++ (Ubuntu 10.2.0-13ubuntu1) 10.2.0` Src/Base/AMReX_FilCC_3D_C.H Src/Base/AMReX_MultiFabUtil_3D_C.H Src/LinearSolvers/MLMG/AMReX_MLMG.H Src/Particle/AMReX_ParticleContainerI.H Src/Particle/AMReX_ParticleIO.H commit 6c6924e66f26586d03d4be661739e7ab600986fa Author: Robert Maynard Date: Tue Jan 5 11:21:23 2021 -0500 Support when CMake 3.18 policy 105 is set to new (#1696) When AMReX is using a newer version of CMake (3.18+) it sets policy `105` to new, causing `target_link_options` rules to be applied to CUDA device linking. When that happens we need to make sure that host only linking flags are excluded. Tools/CMake/AMReX_Config.cmake commit 8aa9810b22c37855f6b0602403138e7ffcea4a7b Author: kngott <30483578+kngott@users.noreply.github.com> Date: Tue Jan 5 08:20:42 2021 -0800 Update FabReduce-isum test in ParallelReduce tutorial. (#1695) Tutorials/GPU/ParallelReduce/main.cpp commit 00c116c09110fb93428624da5cf5a7f0cdaabe43 Author: Matt Larsen Date: Tue Jan 5 08:09:58 2021 -0800 changing blueprint ghost type to int (#1694) This MR changes the type of the ghost field used by conduit blueprint from `double` to `int`. Ascent expects integers, and this reduces the storage space required for the field. Src/Extern/Conduit/AMReX_Conduit_Blueprint.cpp commit 5baf0f7fee46a02caa349d4f8aaae26c192acd05 Author: Axel Huebl Date: Mon Jan 4 16:10:03 2021 -0800 CI: Brew Returns Non-Zero If Already Installed (#1697) Homebrew has the unusual behavior to return a non-zero error code in `brew install` if the package is already installed. Since the base image in CI is somewhat fluid and also the status that `brew update` leaves is very dynamic, we just change this ourselves now to be a non-error. .github/workflows/dependencies/dependencies_mac.sh commit 7eafbe22b93d3d9404efd1de5ea830be4915e372 Author: Weiqun Zhang Date: Sun Jan 3 10:12:03 2021 -0800 Optimized function for filling BaseFab of GpuArray (#1692) Add amrex::fill for BaseFab of GpuArray (a.k.a. AoS). The GPU implementation uses shared memory. Tests show it's more than 10x faster than a naive implementation for GpuArray. Also added are a few functions in DPC++ so that I do not have to look up the documentation every time I need to use things like threadIdx and blockIdx. Src/Base/AMReX_BaseFabUtility.H Src/Base/AMReX_FabArray.H Src/Base/AMReX_GpuTypes.H Src/Base/AMReX_MultiFabUtil.H commit 693700e6afb62df0531601cb48c2a5ab272784ea Author: Weiqun Zhang Date: Sat Jan 2 09:42:15 2021 -0800 BaseFab::minmax (#1691) Add BaseFab::minmax function and use it in VisMF. Src/Base/AMReX_BaseFab.H Src/Base/AMReX_VisMF.cpp commit 9ce1e4e19e64d7ae57d49a4dd0d17aab5a77a376 Author: Emmanuel Motheau Date: Sat Jan 2 09:40:45 2021 -0800 New feature in the fextract tool to directly export in the csv format (#1634) This PR adds a feature to the fextract tool to directly export in the csv format. Docs/sphinx_documentation/source/Post_Processing.rst Tools/Plotfile/fextract.cpp commit 3e38f80ba5e42cd66d98692edd4cef7075661a5b Author: Weiqun Zhang Date: Sat Jan 2 09:25:05 2021 -0800 Update CHANGES (#1693) CHANGES commit 63a567f7b00e7e7592686a8268ba796c3b886481 Author: Weiqun Zhang Date: Mon Dec 28 09:23:53 2020 -0800 DPCPP: GNU Make USE_GPU variable (#1689) Set USE_GPU=TRUE for dpcpp in Make.defs just like cuda and hip. Tools/GNUMake/Make.defs commit 6309a8b7a142d9bdcaedf9e95df15d6db8d9a8f5 Author: Weiqun Zhang Date: Sat Dec 26 12:20:00 2020 -0800 Optimization of MLCellLinOp::applyBC (#1690) Launch one kernel per box to work on all faces and all components. If the number of boxes exceeds the kernel fusing threshold, the kernels from all boxes will be fused. .github/workflows/linux.yml Src/AmrCore/AMReX_TagBox.cpp Src/Base/AMReX_Box.H Src/Base/AMReX_GpuContainers.H Src/LinearSolvers/MLMG/AMReX_MLCellLinOp.H Src/LinearSolvers/MLMG/AMReX_MLCellLinOp.cpp Src/Particle/AMReX_ParticleTransformation.H commit 8bcc31cf0b4319afe633c1c791cdbb30b7bd7118 Author: Max Katz Date: Thu Dec 24 09:12:05 2020 -0800 Remove old launch macros (#1687) Removes some old macros that were used exclusively for the GPU pragma. Also removes `box_threads_and_blocks()` which was only used by that functionality. Src/Base/AMReX_BLFort.H Src/Base/AMReX_GpuDevice.H Src/Base/AMReX_GpuDevice.cpp commit d3fd6462b288249b992baa5f4e9c03d6d8945979 Author: kngott <30483578+kngott@users.noreply.github.com> Date: Wed Dec 23 17:12:26 2020 -0800 unsigned long long iMultiFab sum. (#1684) Changes iMultiFab sum to use hardware supported unsigned long long. Using ParallelReduce, this changes sum time on Cori (default CUDA, 10.1) from 0.07546 to 0.001185, 63x increase. Src/Base/AMReX_iMultiFab.cpp commit 021a9a3d9a5678984afd64fdffdea0c5d4912c44 Author: Weiqun Zhang Date: Wed Dec 23 14:44:23 2020 -0800 Sanitize the input of mkconfig (#1685) The mkconfig.py script takes CPPFLAGS and generates AMReX_Config.H. If CPPFLAGS contains -I flags (as advised by homebrew) and the input is not santized, it would generate ill-formed code. Tools/libamrex/mkconfig.py commit b0977fd7aadb9021196807e9ffe1c8bfa2aaa53c Author: Andy Nonaka Date: Wed Dec 23 14:38:33 2020 -0800 WritePlotfileToASCII Dimensionality Checking (#1686) Added dimensionality error checking to WritePlotfileToASCII.cpp Also supports older FBoxLib plotfile types now. Tools/Postprocessing/C_Src/WritePlotfileToASCII.cpp commit c1713daf9e6ea60b54eb201ab0b18161a2e28c28 Author: Weiqun Zhang Date: Wed Dec 23 14:22:42 2020 -0800 Gpu::Reduce cleanup (#1681) Remove unused parameter in blockReduce_partial. Move the test for full block into Gpu::Handler class. Src/Base/AMReX_GpuReduce.H Src/Base/AMReX_GpuTypes.H commit cf82b6bcd2fdae99c1d78e856e4ee06bcb762995 Author: Weiqun Zhang Date: Wed Dec 23 14:20:28 2020 -0800 New versions of Reduce::Min, Max and MinMax taking callable (#1679) ## Summary Previously, Reduce::Min, Max and MinMax take a pointer to the data. New versions these functions taking a callable object is added to provide more flexibility. Note that Reduce::Sum already has the callable version. Also modify the implementation of the CPU versions of these functions to use OpenMP reduction instead of atomic when we can. Add optional initial value to some functions. Fix the order of template parameters for consistence. ## Checklist The proposed changes: - [x] add new capabilities to AMReX Src/Base/AMReX_Reduce.H commit c98be7422796a0e29cb539f805d91d19c2a5ba01 Author: Johannes Blaschke Date: Wed Dec 23 11:43:48 2020 -0800 update SumNeighbors to reflect recent Particle API changes (#1680) Src/Particle/AMReX_NeighborParticlesCPUImpl.H commit 7e48821802e61c999dcd6bdc7c0471d82e2fee55 Author: Weiqun Zhang Date: Wed Dec 23 10:28:22 2020 -0800 Gpu::AsyncArray -> Gpu::Buffer (#1676) because there is no need for a host callback function to free memory. Src/Base/AMReX_FabArrayUtility.H commit d080afba2402f0f46d03d80ab6d3264ef1968bad Author: Max Katz Date: Tue Dec 22 20:31:14 2020 -0800 Only issue prefetch if arena uses managed memory (#1678) ## Summary Before issuing a managed memory prefetch we check that the arena uses managed memory. This allows the CUDA implementation to still work with `amrex.the_arena_is_managed=0` while calling this API. ## Checklist The proposed changes: - [x] fix a bug or incorrect behavior in AMReX Src/Base/AMReX_BaseFab.H Src/Base/AMReX_FabArrayUtility.H commit 88abc1e88eb34f7f7db0ecf110d75347a9d384a0 Author: Weiqun Zhang Date: Tue Dec 22 15:22:00 2020 -0800 CUDA CI: add flags to catch common errors (#1677) ## Summary Add `--Werror ext-lambda-captures-this` (nvcc 11 only) and `--Werror cross-execution-space-call` to CUDA CI tests. .github/workflows/linux.yml commit 0f820789c57213729b6598f3fa912bd751916dea Author: Weiqun Zhang Date: Tue Dec 22 15:19:49 2020 -0800 GNU Make: Fix the library flags (#1641) Previously for GPU build, the GNU Make system was very sensitive on which variable to use for adding extra `libraries`. Before a certain location, `LIBRARIES` should be used, but after that `libraries` should be used. This is very error prone. This PR tries to fix this. Now the user should be able to use `LIBRARIES` anywhere. The use of `libraries` should be avoided because it is still the case that it can only be used after a certain location in the make file. Nevertheless, this PR is backward compatible and it does not force the use of `LIBRARIES`. Tools/GNUMake/Make.defs Tools/GNUMake/Make.rules commit a4a5adc298abf8ff6b74306fea7448fdff8c7502 Author: Weiqun Zhang Date: Tue Dec 22 14:54:48 2020 -0800 DPCPP: add sync to mimic CUDA NULL stream behavior (#1675) Src/Base/AMReX_GpuDevice.H commit 6c18f6c72f7a965aeed920aba3bf3b6bd6cc64d0 Author: Axel Huebl Date: Tue Dec 22 14:51:07 2020 -0800 NVCC 11.2: CUDA/C++17 (#1665) This adds CI coverage for C++17 builds with NVCC. Several apps expressed interest to transition to C++17 code-bases in the near future and this adds test coverage. Co-authored-by: Weiqun Zhang .github/workflows/linux.yml Src/Particle/AMReX_NeighborList.H Src/Particle/AMReX_ParticleLocator.H Tutorials/GPU/CNS/Source/CNS.cpp commit c8d01767451b03df74afb6f3dfd3da609ca1ffdb Author: Andrew Myers Date: Tue Dec 22 14:31:51 2020 -0800 Remove noexcept from this functor to work around cuda 11 bug. (#1674) Also uses variadic form of `amrex::max`. Tutorials/Particles/NeighborList/MDParticleContainer.cpp commit c777f74b02667f8c9a4929d00f7fb9e835900c53 Author: Weiqun Zhang Date: Tue Dec 22 13:13:36 2020 -0800 PolymorphicArray4 (#1643) PolymorphicArray4 is a thin layer around Array4. It allows us to access both AoS and SoA data with operator(). For example, constexpr int ncomp = 2; Box box(...); BaseFab soa_fab(box, ncomp); auto soa_array = makePolymorphic(soa_fab.array()); int i = ...; int j = ...; int k = ...; soa_array(i,j,k,0) = 0; soa_array(i,j,k,1) = 1; BaseFab> aos_fab(box,1); auto aos_array = makePolymorphic(aos_fab.array()); aos_array(i,j,k,0) = 0; aos_array(i,j,k,1) = 1; Src/Base/AMReX_Array.H Src/Base/AMReX_Array4.H commit afda0382ee0598ea0018894c0a3495c693225bc2 Author: Weiqun Zhang Date: Tue Dec 22 09:33:54 2020 -0800 Remove the check on ref ratio being <= 12 (#1672) Src/Amr/AMReX_Amr.cpp Src/AmrCore/AMReX_AmrMesh.cpp commit 1a8e3e70691a4cdf2183a455d13f2ac529705535 Author: Axel Huebl Date: Mon Dec 21 22:44:25 2020 -0800 CI: OpenMP on Windows (Clang) (#1671) - fix MSVC entry back to MSVC - add OpenMP to Clang .github/workflows/windows.yml commit 6680398a1a4f00424a35cd6d989ffa0f22963197 Author: Weiqun Zhang Date: Mon Dec 21 21:01:01 2020 -0800 Unsigned long long for counting number of particles on GPU (#1670) Because long does not have native atomicAdd support, it's much slower than doing reduction using unsigned long long. Src/Particle/AMReX_ParticleContainerI.H commit c9a571f79d44daba2cae2a4be743be8e29ea348d Author: mic84 Date: Mon Dec 21 18:49:32 2020 -0800 CMake: dowstream projects must enable Fortran if AMReX_FORTRAN=ON (#1669) * CMake: disable Fortran by default * CMake: dowstream projects must enable Fortran if AMReX_FORTRAN=ON Docs/sphinx_documentation/source/BuildingAMReX.rst Src/CMakeLists.txt Tools/CMake/AMReXConfig.cmake.in Tools/CMake/AMReXOptions.cmake commit 4e8386ecf2a5575c7e5e7c7d8c741d4f361bb658 Author: Johannes Blaschke Date: Mon Dec 21 16:16:12 2020 -0800 fix neighbor particle 'GetNeighbors' return type (#1668) Src/Particle/AMReX_NeighborParticles.H commit 754451b9325690e63d0ab45f1d68ba1bae5e284a Author: kngott <30483578+kngott@users.noreply.github.com> Date: Mon Dec 21 14:36:16 2020 -0800 FaceLinear fused interpolater variation (with performance comparison!) (#1663) Add a `interp_arr` version of `FaceLinear` to test generality of the new API (and performance comparison, as `FaceDivFree `stencil can't be used in regular API for comparison). Also includes a bit of `interp_arr` cleanup. Src/AmrCore/AMReX_Interpolater.H Src/AmrCore/AMReX_Interpolater.cpp commit 7b9eb82bec95a5c0e729b3086d614cbdfe2bd71b Author: Andrew Myers Date: Mon Dec 21 14:24:42 2020 -0800 Allow alternate form of "check_pair" function (#1667) This allows users to use a different form of `check_pair` functor that passes the particle indices. This facilitates the creation of "half" neighbor lists. Src/Particle/AMReX_NeighborList.H commit 10ed0e029b6278a8dcf2e4af46623a53f00f1641 Author: Weiqun Zhang Date: Mon Dec 21 13:42:47 2020 -0800 ParallelFor for Reduction (#1658) Add capability for `ParallelFor` to safely do reduction using `deviceReduceSum`, `Min`, etc. The user passes `Gpu::KernelInfo{}.setReduction(true)` to notify `ParallelFor` that this is a parallel reduction, and gives `ParallelFor` a callable that takes `Gpu::Handler`. A `Gpu::Handler` is needed to call `deviceReduceSum`. Also add `Gpu::Buffer` class, whose data pointer can be used as a device destination for `deviceReduceSum`. It also has a `copyToHost` method to copy the device result back to the host. See `Tutorials/GPU/ParallelReduce` for examples of how to use `ParallelFor` for reduction. Also note that the reduction function is OpenMP CPU threads safe. Thus the same code can run on with OpenMP when it is not built for GPU. Co-authored-by: Andrew Myers Src/AmrCore/AMReX_TagBox.cpp Src/Base/AMReX.cpp Src/Base/AMReX_Box.H Src/Base/AMReX_Gpu.H Src/Base/AMReX_GpuAsyncArray.H Src/Base/AMReX_GpuAsyncArray.cpp Src/Base/AMReX_GpuAtomic.H Src/Base/AMReX_GpuBuffer.H Src/Base/AMReX_GpuFuse.H Src/Base/AMReX_GpuKernelInfo.H Src/Base/AMReX_GpuLaunchFunctsC.H Src/Base/AMReX_GpuLaunchFunctsG.H Src/Base/AMReX_GpuReduce.H Src/Base/AMReX_GpuTypes.H Src/Base/AMReX_Reduce.H Src/Base/AMReX_Scan.H Src/Base/CMakeLists.txt Src/Base/Make.package Src/EB/AMReX_EB2_GeometryShop.H Tutorials/GPU/ParallelReduce/GNUmakefile Tutorials/GPU/ParallelReduce/main.cpp commit c8cdfa61f443ecde21debc3ad52665bf48eded8b Author: Andrew Myers Date: Sat Dec 19 11:32:26 2020 -0800 Revert tiling change in #1655 (#1664) Src/Particle/AMReX_ParticleContainerI.H commit 875ebff912b5d3f4edc0e6ad7a983a56374a9e69 Author: Axel Huebl Date: Fri Dec 18 20:36:28 2020 -0800 CI: CUDA 9.1 & 11.0 (#1653) Add a recent CUDA 11 release for cutting-edge coverage, too. .github/workflows/dependencies/dependencies_nvcc11.sh .github/workflows/dependencies/dependencies_nvcc9.sh .github/workflows/linux.yml commit 25d06e82804e3e1b4f3f1b0cea4e9979ed37373c Author: Axel Huebl Date: Fri Dec 18 15:29:58 2020 -0800 CI: Lib + Plotfiles Install (#1662) Test the new install of plotfile tools works in CI. .github/workflows/linux.yml commit cc61f30f4266407225bd8c7354565931b275a07f Author: mic84 Date: Fri Dec 18 13:56:37 2020 -0800 CMake: streamline install of plotfile tools (#1661) CMakeLists.txt Docs/sphinx_documentation/source/BuildingAMReX.rst Src/CMakeLists.txt Tools/CMake/AMReXConfig.cmake.in Tools/CMake/AMReXInstallHelpers.cmake Tools/Plotfile/CMakeLists.txt commit 1c291c0b6bb52be354b77b2abb646216dcb62ff7 Author: kngott <30483578+kngott@users.noreply.github.com> Date: Fri Dec 18 10:39:38 2020 -0800 Add new clean commands to docs. (#1660) Docs/sphinx_documentation/source/BuildingAMReX.rst commit 3b897e1b7e04222ae718c00010085908f7693b41 Author: mic84 Date: Thu Dec 17 17:14:36 2020 -0800 Fix how fextract handle the tolerance options (#1659) Tools/Plotfile/fextract.cpp commit de1701ed9d6cbdf61a8c4c3015c1cf5a9b00f7f5 Author: jmsexton03 Date: Thu Dec 17 17:00:36 2020 -0500 AssignDensity and AssignCellDensitySingleLevel copy -> ParallelCopy (#1633) Src/AmrCore/AMReX_AmrParticles.H Src/Particle/AMReX_ParticleContainerI.H commit 07a21fd454317b7b434240d548765c4c4ae4464a Author: Andrew Myers Date: Thu Dec 17 13:29:53 2020 -0800 Have AssignCellDensitySingleLevel respect gpu launch guards. (#1655) Src/Particle/AMReX_ParticleContainerI.H commit e21901db47b85704e29b3800fcad7318b9544e76 Author: Weiqun Zhang Date: Thu Dec 17 12:27:08 2020 -0800 Clean up GPU pragma and CUDA Fortran (#1656) Clean up the remaining GPU pragma and CUDA Fortran. This also fixes a bug in #1650 for codes using the Fortran 77 version of filcc. The proposed changes: - [x] fix a bug or incorrect behavior in AMReX Docs/sphinx_documentation/source/GPU.rst Docs/sphinx_documentation/source/GPU_Chapter.rst Src/Base/AMReX_BCUtil.cpp Src/Base/AMReX_GpuDevice.cpp Src/Base/AMReX_PhysBCFunct.H Src/Base/AMReX_PhysBCFunct.cpp Src/Base/AMReX_filcc_mod.F90 Src/Base/AMReX_fort_mod.F90 Src/Base/Make.package Tools/GNUMake/Make.defs Tools/GNUMake/comps/hpcsdk.mak Tools/GNUMake/comps/ibm.mak Tools/GNUMake/comps/pgi.mak Tutorials/GPU/Launch/Readme.md commit 63db5cbe23e2ddc419e6895ec69610388519ca9d Author: mic84 Date: Thu Dec 17 11:14:35 2020 -0800 Add features to fextract (#1657) Add the following features to `fextract`: - user option to set the number of decimals in the output of reals - user option to set a cut-off tolerance: anything smaller than the tolerance gets printed as zero - user option to disable the job summary report at the end of the output The job summary report is disabled by default in contrast with the current behaviour. Tools/Plotfile/fextract.cpp commit 1f8891e49f23397b5633b34dae4437619d3b46a8 Author: Andrew Myers Date: Thu Dec 17 10:40:55 2020 -0800 Fix parallel add bug when both USE_GPU and USE_OMP are defined. (#1654) We cannot assume copies are thread safe any time AMREX_USE_GPU is defined, whether or not USE_OMP is also on. Note that this only would have been an issue when compiling with OMP but running on only one thread. The proposed changes: - [x] fix a bug or incorrect behavior in AMReX Src/Base/AMReX_FabArrayBase.cpp commit e63397244cd19a67d1e76fdedce73667d41e8f2c Author: Andy Nonaka Date: Thu Dec 17 10:38:23 2020 -0800 Write a plotfile to ASCII (#1638) This utility reads in a single-level plotfile, copies it to a MultiFab with a single box, then writes out all the data in 'i j k comp ' format to the screen The user can modify this cpp file to write out on certain components, coordinates, row/column formatting, etc. Usage: ./WritePlotfileToASCII2d.gnu.MPI.ex infile=inputFileName Docs/sphinx_documentation/source/Post_Processing.rst Docs/sphinx_documentation/source/Post_Processing_Chapter.rst Docs/sphinx_documentation/source/index.rst Tools/Postprocessing/C_Src/GNUmakefile Tools/Postprocessing/C_Src/WritePlotfileToASCII.cpp commit 7b24139e3214b417b54f8d6238131dc484bc6e66 Author: mic84 Date: Wed Dec 16 12:31:01 2020 -0800 CMake: AMRDATA needs Fortran (#1645) Tools/CMake/AMReXOptions.cmake commit 75373313e44803c3732096a2dee413b52ed2d62e Author: mic84 Date: Wed Dec 16 12:30:36 2020 -0800 CMake: fix few quirks with GPU backends variables (#1561) Docs/sphinx_documentation/source/BuildingAMReX.rst Tools/CMake/AMReXConfig.cmake.in Tools/CMake/AMReXOptions.cmake commit b851ce78a2f3f26d53eb101aa0a0266284f44bfb Author: kngott <30483578+kngott@users.noreply.github.com> Date: Wed Dec 16 12:23:39 2020 -0800 FaceLinear: Avoid accessing face outside of coarse box. (#1651) ## Summary Fix an index out-of-bound trigger in FaceLinear interpolater. ## Checklist The proposed changes: - [x] fix a bug or incorrect behavior in AMReX - [ ] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] are described in the proposed changes to the AMReX documentation, if appropriate Src/AmrCore/AMReX_Interp_1D_C.H Src/AmrCore/AMReX_Interp_2D_C.H Src/AmrCore/AMReX_Interp_3D_C.H commit 08e27d80f3cb0151566b05f964fb4fc30296f56c Author: Axel Huebl Date: Wed Dec 16 12:09:48 2020 -0800 CI: fix macOS (#1652) * CI: fix macOS brew did again some updates and things fail. let's revert our current work-around. * Brew: Add gfortran not pulled automatically anymore .github/workflows/dependencies/dependencies_mac.sh commit d7b6b97354c02f2e37761f8afae0fca36e7a4550 Author: Candace Gilet Date: Wed Dec 16 14:13:52 2020 -0500 Fix NodalProjector::getGradPhi for multilevel case (#1642) ## Summary Two bug fixes for multilevel in NodalProjector: 1. Ensure NodalProjector::getGradPhi() returns the correct result. Previously could return \sum{sigma grad phi} / (2**dim sigma_coarse). 2. Average down the complete projected velocity at the end. For alpha!=1, this ensures the projected velocity is averaged down correctly. This could change the results of multilevel regression tests. If the application uses the affected outputs from NodalProjector as is, then changes are expected. If the application subsequently averages down the outputs, then changes greater than roundoff level are not expected. ## Checklist The proposed changes: - [x] fix a bug or incorrect behavior in AMReX - [ ] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] are described in the proposed changes to the AMReX documentation, if appropriate Src/LinearSolvers/MLMG/AMReX_MLNodeLap_2D_K.H Src/LinearSolvers/MLMG/AMReX_MLNodeLap_3D_K.H Src/LinearSolvers/MLMG/AMReX_MLNodeLaplacian.H Src/LinearSolvers/MLMG/AMReX_MLNodeLaplacian.cpp Src/LinearSolvers/MLMG/AMReX_MLNodeLinOp.H Src/LinearSolvers/Projections/AMReX_NodalProjector.cpp commit 70b7ab9999522a51666027ae3f7f3dc527541290 Author: Max Katz Date: Wed Dec 16 09:10:42 2020 -0800 Remove GPU pragma support from GNU make build system (#1650) This is no longer used. Tools/F_scripts/gpu_fortran.py Tools/F_scripts/write_cuda_headers.py Tools/GNUMake/Make.defs Tools/GNUMake/Make.rules Tools/GNUMake/sites/Make.llnl commit affcccf3c900d1d29349aaeb0c5f21c9dd63e637 Author: Max Katz Date: Wed Dec 16 09:08:45 2020 -0800 Remove GPU pragma macros from ArrayLim (#1649) These are no longer used. Src/Base/AMReX_ArrayLim.H commit 78eae95a21df8fa1d0e2a7c86b0437dd68e8aa7a Author: Max Katz Date: Wed Dec 16 09:06:31 2020 -0800 Remove custom MFIter reduction code (#1648) This is no longer in use by any of the astro codes. Src/Base/AMReX_BLFort.H Src/Base/AMReX_MFIter.H Src/Base/AMReX_MFIter.cpp commit 2f9277a64058c91a5d0153b77bd2476bd1f85690 Author: Axel Huebl Date: Tue Dec 15 17:06:11 2020 -0800 CMake: Fix Typos in SYCL Warnings (#1647) Just a typo and a missing space in a warning. Tools/CMake/AMReXOptions.cmake commit 006569febeae1329fc0a3e0c2fb17c77e42539c7 Author: Weiqun Zhang Date: Tue Dec 15 16:30:54 2020 -0800 DPCPP MPI wrapper (#1646) ## Summary Replace mpiicpx with mpiicpc because mpiicpc is shipped by Intel, whereas mpiicpx is a script that is only available on JLSE testbeds. ## Checklist The proposed changes: - [x] fix a bug or incorrect behavior in AMReX - [ ] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] are described in the proposed changes to the AMReX documentation, if appropriate Co-authored-by: Michele Rosso Tools/CMake/AMReXOptions.cmake Tools/CMake/AMReXSYCL.cmake Tools/GNUMake/Make.defs Tutorials/GPU/CNS/Source/CNS_bcfill.cpp Tutorials/GPU/CNS/Source/hydro/CNS_hydro_K.H commit cd2e06e47cb586d0b8e2ef3eb0c534548634ad97 Author: Weiqun Zhang Date: Tue Dec 15 10:44:19 2020 -0800 Remove support for old versions of DPCPP (#1644) Src/Base/AMReX_Extension.H Src/Base/AMReX_FBI.H Src/Base/AMReX_GpuPrint.H Src/Base/AMReX_GpuReduce.H Src/Base/AMReX_RandomEngine.H Src/Base/AMReX_Scan.H commit cd31efb8aee323dafddb6040c0613667cfb5382c Author: Weiqun Zhang Date: Tue Dec 15 09:14:51 2020 -0800 HIP and DPCPP support in configure (#1640) Tools/GNUMake/comps/hip.mak Tools/libamrex/configure.py Tools/libamrex/mkconfig.py commit 66e52d44979d14859e007d769fb1b550ec591cc7 Author: Weiqun Zhang Date: Mon Dec 14 15:45:16 2020 -0800 Fused FillBoundary (#1570) ## Summary Add a new FillBoundary function for a Vector of FabArrays. There are two implementations in this PR. The one being used simply calls `FillBoundary_nowait` and `FillBoundary_finish`. The other version aggregates the MPI messages from multiple FabArrays into one. Tests on summit shows the latter is slower. But I don't want to lose the code, so it is kept for future investigation. ## Checklist The proposed changes: - [ ] fix a bug or incorrect behavior in AMReX - [x] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] are described in the proposed changes to the AMReX documentation, if appropriate Src/Base/AMReX_Array4.H Src/Base/AMReX_BaseFab.H Src/Base/AMReX_Box.H Src/Base/AMReX_FabArrayBase.H Src/Base/AMReX_FabArrayBase.cpp Src/Base/AMReX_FabArrayCommI.H Src/Base/AMReX_MultiFab.H Src/Base/AMReX_MultiFab.cpp Src/Base/AMReX_NonLocalBCImpl.H Src/Base/AMReX_ParallelDescriptor.H Src/Base/AMReX_ParallelDescriptor.cpp commit 6a212600436d83fcfb656ce72d6c810c1b7c1cdd Author: kngott <30483578+kngott@users.noreply.github.com> Date: Mon Dec 14 13:51:13 2020 -0800 Divergence Free Stencil and API (#1483) API for multi-dimensional, face-based, coarse-to-fine interpolation (InterpFromCoarse and FillPatchTwoLevels). Includes a variation of Vanella et. al. stencil (doi:10.1016/j.jcp.2010.05.003, section 3.2), but instead of interpolating from a far coarse cell to solve interior closure problem (highly biased result dependent on chosen coarse cell), this uses least squares with initial guess equal to the average of fine face values across the cell. (Mathematica notebooks of solution are available). Src/AmrCore/AMReX_FillPatchUtil.H Src/AmrCore/AMReX_FillPatchUtil_1d.F90 Src/AmrCore/AMReX_FillPatchUtil_I.H Src/AmrCore/AMReX_INTERP_1D.F90 Src/AmrCore/AMReX_Interp_1D_C.H Src/AmrCore/AMReX_Interp_2D_C.H Src/AmrCore/AMReX_Interp_3D_C.H Src/AmrCore/AMReX_Interpolater.H Src/AmrCore/AMReX_Interpolater.cpp Src/Base/AMReX_BoxArray.H Src/Base/AMReX_BoxArray.cpp Src/Base/AMReX_GpuLaunch.H Tests/DivFreePatch/GNUmakefile Tests/DivFreePatch/Make.package Tests/DivFreePatch/inputs Tests/DivFreePatch/main.cpp commit 78d6f64fadc606d65e30d4da5f218e941bc7ea59 Author: Weiqun Zhang Date: Mon Dec 14 10:19:51 2020 -0800 Constant coefficient MacProjector (#1636) Optimization of constant coefficient MAC projection by using constant coefficient Poisson solver. Src/LinearSolvers/MLMG/AMReX_MLCellABecLap.cpp Src/LinearSolvers/Projections/AMReX_MacProjector.H Src/LinearSolvers/Projections/AMReX_MacProjector.cpp commit 4794c1d1c2e1fd83bc5ac6be8ff71c4893861128 Author: Weiqun Zhang Date: Mon Dec 14 08:39:07 2020 -0800 Option to allow multiple active MFIters (#1631) Add MFIter::allowMultipleMFIters(bool) so that users can disable assertion on the number of active MFIters. This is needed by FLASH because multiple MFIters (one for each level) are built in its iterator. Docs/sphinx_documentation/source/Basics.rst Src/Base/AMReX_MFIter.H Src/Base/AMReX_MFIter.cpp Src/F_Interfaces/Base/AMReX_multifab_fi.cpp Src/F_Interfaces/Base/AMReX_multifab_mod.F90 commit 8e73aa00126634cd83d4ca9c46fbac20da924e4e Author: drangara <69211175+drangara@users.noreply.github.com> Date: Mon Dec 14 11:37:42 2020 -0500 Convert AMReX_eb_to_pvd.F90 to cpp (#1627) ## Summary Convert the fortran file that writes the EB surface for paraview visualization to cpp ## Additional background Tested with the visualization files produced by `mfix/benchamrk/07-hopper` and `mfix/tutorials/clr/prototype`. Paraview visualization seems to look "good". When comparing the actual data from the files I do see some differences b/w the fortran and cpp version, but I believe the differences can be attributed to the precision of datatypes used in the fortran vs cpp code. These changes should not affect the solution results, just affects visualization. Src/EB/AMReX_EBToPVD.H Src/EB/AMReX_EBToPVD.cpp Src/EB/AMReX_WriteEBSurface.cpp Src/EB/AMReX_WriteEB_F.H Src/EB/AMReX_eb_to_pvd.F90 Src/EB/CMakeLists.txt Src/EB/Make.package commit c0a931f0ca46d62210e5b8f2e92c6952cfd60685 Author: Weiqun Zhang Date: Sat Dec 12 14:42:57 2020 -0800 Fix EB bug in #1629 (#1635) For EB nodal solver with constant sigma, the domain ghost cells of sigma should be set to zero. Src/LinearSolvers/MLMG/AMReX_MLNodeLaplacian.cpp commit f6f883bfedf2ecf3900c7f79f1f4df3ec92cca76 Author: Weiqun Zhang Date: Sat Dec 12 10:45:05 2020 -0800 Nodal solver with constant coefficient (#1629) * Add specialization for constant coefficient case in the nodal solver. * Update NodalProjector. * Fix some warnings in 1D. Src/LinearSolvers/MLMG/AMReX_MLABecLap_1D_K.H Src/LinearSolvers/MLMG/AMReX_MLALap_1D_K.H Src/LinearSolvers/MLMG/AMReX_MLLinOp.cpp Src/LinearSolvers/MLMG/AMReX_MLNodeLap_1D_K.H Src/LinearSolvers/MLMG/AMReX_MLNodeLap_2D_K.H Src/LinearSolvers/MLMG/AMReX_MLNodeLap_3D_K.H Src/LinearSolvers/MLMG/AMReX_MLNodeLaplacian.H Src/LinearSolvers/MLMG/AMReX_MLNodeLaplacian.cpp Src/LinearSolvers/MLMG/AMReX_MLNodeTensorLap_1D_K.H Src/LinearSolvers/MLMG/AMReX_MLNodeTensorLaplacian.cpp Src/LinearSolvers/MLMG/AMReX_MLPoisson_1D_K.H Src/LinearSolvers/MLMG/AMReX_MLTensorOp.cpp Src/LinearSolvers/Projections/AMReX_NodalProjector.H Src/LinearSolvers/Projections/AMReX_NodalProjector.cpp commit 85a5f3be91eb9d58845f96c2907af8ee65661f87 Author: Andrew Myers Date: Sat Dec 12 09:44:58 2020 -0800 fix bug in pshifting neighbors introduced in #1590 (#1632) Src/Particle/AMReX_ParticleCommunication.H commit 52d32c6a07cd14498129ec3f03ae69a06a27573e Author: asalmgren Date: Fri Dec 11 21:10:38 2020 -0800 When we create a new level of refinement, we need to set the "level_s… (#1630) …teps" of that new level appropriately. ## Summary ## Additional background ## Checklist The proposed changes: - [x] fix a bug or incorrect behavior in AMReX - [ ] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] are described in the proposed changes to the AMReX documentation, if appropriate Src/Amr/AMReX_Amr.cpp commit 389a0cc49f018ea86df0067bc8182336426fe414 Author: Andrew Myers Date: Fri Dec 11 16:24:59 2020 -0800 Expose the index of the neighboring particle to users of the iterator. (#1628) The proposed changes: - [ ] fix a bug or incorrect behavior in AMReX - [x] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] are described in the proposed changes to the AMReX documentation, if appropriate Src/Particle/AMReX_NeighborList.H commit 73509be24826d16680289e5555cdb8db1e2217e5 Author: Weiqun Zhang Date: Thu Dec 10 10:02:10 2020 -0800 Documentation on avoiding continue and return inside GPU launch macros. (#1617) Docs/sphinx_documentation/source/GPU.rst commit 728db75fa70fcf10fb7fcb53210ba988ce24f1e4 Author: Weiqun Zhang Date: Thu Dec 10 10:01:16 2020 -0800 Implement overset mask for MLPoisson too (#1604) Src/LinearSolvers/CMakeLists.txt Src/LinearSolvers/MLMG/AMReX_MLABecLap_1D_K.H Src/LinearSolvers/MLMG/AMReX_MLABecLap_2D_K.H Src/LinearSolvers/MLMG/AMReX_MLABecLap_3D_K.H Src/LinearSolvers/MLMG/AMReX_MLABecLaplacian.H Src/LinearSolvers/MLMG/AMReX_MLABecLaplacian.cpp Src/LinearSolvers/MLMG/AMReX_MLCellABecLap.H Src/LinearSolvers/MLMG/AMReX_MLCellABecLap.cpp Src/LinearSolvers/MLMG/AMReX_MLCellABecLap_1D_K.H Src/LinearSolvers/MLMG/AMReX_MLCellABecLap_2D_K.H Src/LinearSolvers/MLMG/AMReX_MLCellABecLap_3D_K.H Src/LinearSolvers/MLMG/AMReX_MLCellABecLap_K.H Src/LinearSolvers/MLMG/AMReX_MLPoisson.H Src/LinearSolvers/MLMG/AMReX_MLPoisson.cpp Src/LinearSolvers/MLMG/AMReX_MLPoisson_1D_K.H Src/LinearSolvers/MLMG/AMReX_MLPoisson_2D_K.H Src/LinearSolvers/MLMG/AMReX_MLPoisson_3D_K.H Src/LinearSolvers/MLMG/Make.package commit 8db1187e6faa4d6628d47ddb28dcc23200516451 Author: Axel Huebl Date: Thu Dec 10 09:47:03 2020 -0800 Warning in SP: large ICC float constant (#1626) Fix an ICC warning for a too-large float constant when compiling for single precision in WarpX. Src/Amr/AMReX_Amr.cpp commit cd88adc2e64456116fcaf0dcce4669921bdb2296 Author: Weiqun Zhang Date: Wed Dec 9 18:57:46 2020 -0800 Fuse nodal jacobi smoother for GPU (#1625) ## Summary Fuse nodal jacobi smoother kernels by switch the order of the sweep and MFIter loops. For Amr-Wind's ABL test with `amr.n_cell = 256 256 64` and `amr.max_grid_size = 32` on one V100, the nodal project is about 2x faster. ## Checklist The proposed changes: - [ ] fix a bug or incorrect behavior in AMReX - [ ] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] are described in the proposed changes to the AMReX documentation, if appropriate Src/LinearSolvers/MLMG/AMReX_MLNodeLap_1D_K.H Src/LinearSolvers/MLMG/AMReX_MLNodeLap_2D_K.H Src/LinearSolvers/MLMG/AMReX_MLNodeLap_3D_K.H Src/LinearSolvers/MLMG/AMReX_MLNodeLap_K.H Src/LinearSolvers/MLMG/AMReX_MLNodeLaplacian.cpp commit b846969f308e0402e188b95756b478261a31a8d2 Author: Matt Larsen Date: Wed Dec 9 16:26:11 2020 -0800 don't use geom.size() as the number of levels (#1624) ## Summary The conduit adapter was using the `geom.size()` to determine the number of levels, instead of believing `n_levels`. There was a case where the max levels was greater than the actual refinement level. In this case, geoms was the size of the max_levels in the input deck, but mfs was the size of the active levels. resolves #1623 The proposed changes: - [x] fix a bug or incorrect behavior in AMReX - [ ] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] are described in the proposed changes to the AMReX documentation, if appropriate Src/Extern/Conduit/AMReX_Conduit_Blueprint.cpp commit 923752e81d08a87d8edd7aaf01e92359125e4f86 Author: OscarAntepara <52221614+OscarAntepara@users.noreply.github.com> Date: Wed Dec 9 08:15:17 2020 -0800 Adding eb slopes limiter (#1594) * Adding eb slope limiters * Adding comments and making 3d adjustments * Fixing face centroids computation in the z-direction and correcting the comments Src/EB/AMReX_EB_slopes_K.H commit ffa5387dd6d873987386cd9ebbea3bab9592c9fd Author: Andrew Myers Date: Tue Dec 8 13:36:03 2020 -0800 Support for SoA data for neighbor particles. (#1590) Src/Particle/AMReX_NeighborList.H Src/Particle/AMReX_NeighborParticles.H Src/Particle/AMReX_NeighborParticlesCPUImpl.H Src/Particle/AMReX_NeighborParticlesGPUImpl.H Src/Particle/AMReX_NeighborParticlesI.H Src/Particle/AMReX_ParticleCommunication.H Src/Particle/AMReX_Particles.H Tests/Particles/NeighborParticles/MDParticleContainer.H Tests/Particles/NeighborParticles/MDParticleContainer.cpp Tests/Particles/NeighborParticles/main.cpp Tests/Particles/Redistribute/main.cpp commit a28b02f1d194a0bce76cf784c6dc98cad1339015 Author: Axel Huebl Date: Tue Dec 8 11:13:42 2020 -0800 CMake: Verify DPC++ Compiler ID and at best warn (#1619) * CMake: less stringed MPI+DPCPP Check `find_package(MPI)` is capable to find Intel's MPI with a dpcpp CXX compiler. I would relax this check. Also, `CMAKE_CXX_COMPILER` can be an absolute path, e.g. `-DCMAKE_CXX_COMPILER=$(which dpcpp)`, which is not covered by the original syntax. * DPCPP Compiler Check: Only Warn on ID * Update recommendation * keep leading \n (consistency) Tools/CMake/AMReXOptions.cmake commit e0896cbd9e0432f642d484675e2a4f090ce93e33 Author: Houjun Date: Tue Dec 8 10:53:32 2020 -0800 Fix a bug that causes errors when writing HDF5 plotfile and particle (#1621) * Add HDF5 optimizations * Fix an issue that causes incorrect HDF5 data write with Tutorials/Basic/HeatEquation_EX1_C example code * Minor HDF5 tuning parameter change Co-authored-by: Houjun Tang Src/Base/AMReX_PlotFileUtil.cpp Src/Particle/AMReX_ParticleHDF5.H commit 82ca8fa508cad15c3f63c6233c8b028da5b4e347 Author: Michael Zingale Date: Tue Dec 8 12:34:11 2020 -0500 the A64fx Cray 10.0.1 compilers need the old-style options (#1620) ## Summary The default cray compilers on the Ookami A64fx machine use the old-style options for cray compilers. This logic allows us to build there. ## Additional background ## Checklist The proposed changes: - [x] fix a bug or incorrect behavior in AMReX - [ ] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] are described in the proposed changes to the AMReX documentation, if appropriate Tools/GNUMake/comps/cray.mak commit 91a48a24777dabe03b1ff32bfce05536ce806200 Author: Weiqun Zhang Date: Mon Dec 7 20:00:29 2020 -0800 Fortran Inteface: Porting more MultiFab functions (#1609) Port IntVect ghost cell version of MultiFab::Add, Subtruct, Multiply, Divide, Saxpy, and LinComb to Fortran Interface. Src/F_Interfaces/Base/AMReX_multifab_fi.cpp Src/F_Interfaces/Base/AMReX_multifab_mod.F90 commit 975135ebdec9e0d96f9225a0196fade8ccbd2196 Author: Andrew Myers Date: Mon Dec 7 14:46:19 2020 -0800 [CI] turn off CLEAN option to keep existing files in the deploy target. (#1618) .github/workflows/docs.yml commit 8d45dd95dde0b6cb579c41b13ed3b0b69add297b Author: Axel Huebl Date: Mon Dec 7 13:40:33 2020 -0800 Conduit: Fix unused variables (#1615) ## Summary Removes four (4) unused variable warnings. cc @cyrush @mclarsen ## Additional background ## Checklist The proposed changes: - [ ] fix a bug or incorrect behavior in AMReX - [ ] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] are described in the proposed changes to the AMReX documentation, if appropriate Src/Extern/Conduit/AMReX_Conduit_Blueprint.cpp commit 774ec481be4fdf28bd80f5a381af9b1c97d5b007 Author: Axel Huebl Date: Mon Dec 7 13:32:59 2020 -0800 DPC++: Re-Enable with Work-around. (#1613) My former colleagues shared a work-around that we can use to overwrite the compiler identification: https://gitlab.kitware.com/cmake/cmake/-/issues/21551#note_869580 This regression will also be fixed in CMake 3.19.2 .github/workflows/linux.yml commit ef6ebdbe62366e78d542fa13739e6d0557a3cbf4 Author: Andrew Myers Date: Mon Dec 7 11:48:19 2020 -0800 deploy to docs to seperate 'build' directory. (#1614) .github/workflows/docs.yml commit 2bbaf1bd81b0b8fee3dbd1bc95f2fc5c22009ac3 Author: Andrew Myers Date: Mon Dec 7 10:55:56 2020 -0800 fix and simplify script that builds the documentation (#1612) build_docs.sh commit 6fb23a385bf2fa0bc2e3b64a24e4940b4c0ea16d Author: Andrew Myers Date: Mon Dec 7 10:32:11 2020 -0800 need to specify the target directory on the website repository (#1611) .github/workflows/docs.yml commit 0a402eef6c0f7bdcb609e786d3140a991e99b7b3 Author: Andrew Myers Date: Mon Dec 7 08:59:32 2020 -0800 Migrate building and deploying the documentation to Github CI. (#1603) .github/workflows/dependencies/documentation.sh .github/workflows/docs.yml .travis.yml Docs/sphinx_tutorials/Makefile build_and_deploy.sh build_docs.sh deploy_key.enc commit 81a56bd628b92745c2baef58d8e7b044306cc34c Author: Weiqun Zhang Date: Sun Dec 6 18:46:48 2020 -0800 Add check for plotfile tools (#1610) .github/workflows/linux.yml Tools/Plotfile/fboxinfo.cpp Tools/Plotfile/fextract.cpp Tools/Plotfile/fextrema.cpp commit 65615e97ef62a2cac302ecbe30482d35f0806542 Author: Weiqun Zhang Date: Sun Dec 6 16:05:16 2020 -0800 Fix error in #1599 (#1608) The proposed change fixes an error in #1599. Tools/Plotfile/fsnapshot.cpp commit 88236accbcc263cbf3d4b103189bc7b522e06dfd Author: Andrew Myers Date: Sun Dec 6 14:33:50 2020 -0800 Remove alternative operators from AMReX (#1599) Operators like `and`, `or`, and `not` are allowed in the C++ standard; however, they cause problems on Windows and with our clang Windows CI. This PR removes them from AMReX in favor of `&&`, `||`, and `!`, and also configures the CI to throw a compilation error if they are re-introduced into the codebase. Application codes are still free to use these if they want. Co-authored-by: Weiqun Zhang .github/workflows/linux.yml .github/workflows/macos.yml Src/AmrCore/AMReX_AmrCore.cpp Src/AmrCore/AMReX_AmrParGDB.H Src/AmrCore/AMReX_Cluster.cpp Src/AmrCore/AMReX_FillPatchUtil_I.H Src/Base/AMReX_Array4.H Src/Base/AMReX_AsyncOut.cpp Src/Base/AMReX_BaseFab.H Src/Base/AMReX_BlockMutex.cpp Src/Base/AMReX_BoxArray.H Src/Base/AMReX_BoxArray.cpp Src/Base/AMReX_CuptiTrace.cpp Src/Base/AMReX_FArrayBox.H Src/Base/AMReX_FArrayBox.cpp Src/Base/AMReX_FBI.H Src/Base/AMReX_FabArray.H Src/Base/AMReX_FabArrayBase.cpp Src/Base/AMReX_FabArrayUtility.H Src/Base/AMReX_FilCC_C.cpp Src/Base/AMReX_FileSystem.cpp Src/Base/AMReX_Geometry.cpp Src/Base/AMReX_GpuAtomic.H Src/Base/AMReX_GpuLaunchFunctsG.H Src/Base/AMReX_GpuLaunchMacrosG.H Src/Base/AMReX_GpuReduce.H Src/Base/AMReX_IArrayBox.cpp Src/Base/AMReX_MultiFab.cpp Src/Base/AMReX_MultiFabUtil.cpp Src/Base/AMReX_PCI.H Src/Base/AMReX_ParallelDescriptor.cpp Src/Base/AMReX_PlotFileDataImpl.cpp Src/Base/AMReX_Reduce.H Src/Base/AMReX_Slopes_K.H Src/Base/AMReX_TypeTraits.H Src/Base/AMReX_Vector.H Src/Base/AMReX_VisMF.cpp Src/Boundary/AMReX_InterpBndryData_2D_K.H Src/Boundary/AMReX_InterpBndryData_3D_K.H Src/EB/AMReX_EB2_2D_C.H Src/EB/AMReX_EB2_2D_C.cpp Src/EB/AMReX_EB2_3D_C.H Src/EB/AMReX_EB2_3D_C.cpp Src/EB/AMReX_EB2_Level.H Src/EB/AMReX_EBAmrUtil.cpp Src/EB/AMReX_EBFluxRegister_2D_C.H Src/EB/AMReX_EBFluxRegister_3D_C.H Src/EB/AMReX_EBMultiFabUtil.cpp Src/EB/AMReX_EBMultiFabUtil_2D_C.H Src/EB/AMReX_EBMultiFabUtil_3D_C.H Src/EB/AMReX_EB_slopes_K.H Src/EB/AMReX_EB_triGeomOps_K.H Src/EB/AMReX_EB_utils.cpp Src/EB/AMReX_WriteEBSurface.cpp Src/EB/AMReX_algoim_K.H Src/Extern/HYPRE/AMReX_Habec_2D_K.H Src/Extern/HYPRE/AMReX_Habec_3D_K.H Src/Extern/HYPRE/AMReX_HypreNodeLap.cpp Src/F_Interfaces/AmrCore/AMReX_FlashFluxRegister.cpp Src/LinearSolvers/MLMG/AMReX_MLABecLap_1D_K.H Src/LinearSolvers/MLMG/AMReX_MLABecLap_2D_K.H Src/LinearSolvers/MLMG/AMReX_MLABecLap_3D_K.H Src/LinearSolvers/MLMG/AMReX_MLABecLaplacian.cpp Src/LinearSolvers/MLMG/AMReX_MLALap_1D_K.H Src/LinearSolvers/MLMG/AMReX_MLALap_2D_K.H Src/LinearSolvers/MLMG/AMReX_MLALap_3D_K.H Src/LinearSolvers/MLMG/AMReX_MLCellABecLap.cpp Src/LinearSolvers/MLMG/AMReX_MLEBABecLap.cpp Src/LinearSolvers/MLMG/AMReX_MLEBABecLap_2D_K.H Src/LinearSolvers/MLMG/AMReX_MLEBABecLap_3D_K.H Src/LinearSolvers/MLMG/AMReX_MLEBABecLap_K.H Src/LinearSolvers/MLMG/AMReX_MLEBTensor_2D_K.H Src/LinearSolvers/MLMG/AMReX_MLEBTensor_3D_K.H Src/LinearSolvers/MLMG/AMReX_MLLinOp.cpp Src/LinearSolvers/MLMG/AMReX_MLLinOp_K.H Src/LinearSolvers/MLMG/AMReX_MLMG.cpp Src/LinearSolvers/MLMG/AMReX_MLMG_2D_K.H Src/LinearSolvers/MLMG/AMReX_MLMG_3D_K.H Src/LinearSolvers/MLMG/AMReX_MLNodeLap_2D_K.H Src/LinearSolvers/MLMG/AMReX_MLNodeLap_3D_K.H Src/LinearSolvers/MLMG/AMReX_MLNodeLap_K.H Src/LinearSolvers/MLMG/AMReX_MLNodeLaplacian.cpp Src/LinearSolvers/MLMG/AMReX_MLNodeLinOp.cpp Src/LinearSolvers/MLMG/AMReX_MLNodeTensorLap_2D_K.H Src/LinearSolvers/MLMG/AMReX_MLNodeTensorLap_3D_K.H Src/LinearSolvers/MLMG/AMReX_MLPoisson_1D_K.H Src/LinearSolvers/MLMG/AMReX_MLPoisson_2D_K.H Src/LinearSolvers/MLMG/AMReX_MLPoisson_3D_K.H Src/LinearSolvers/MLMG/AMReX_MLTensorOp.cpp Src/LinearSolvers/MLMG/AMReX_MLTensor_2D_K.H Src/LinearSolvers/MLMG/AMReX_MLTensor_3D_K.H Src/Particle/AMReX_NeighborParticles.H Src/Particle/AMReX_NeighborParticlesCPUImpl.H Src/Particle/AMReX_NeighborParticlesGPUImpl.H Src/Particle/AMReX_NeighborParticlesI.H Src/Particle/AMReX_Particle.H Src/Particle/AMReX_ParticleCommunication.H Src/Particle/AMReX_ParticleCommunication.cpp Src/Particle/AMReX_ParticleContainerI.H Src/Particle/AMReX_ParticleHDF5.H Src/Particle/AMReX_ParticleIO.H Src/Particle/AMReX_ParticleLocator.H Src/Particle/AMReX_ParticleTile.H Src/Particle/AMReX_ParticleUtil.H Src/Particle/AMReX_WriteBinaryParticleData.H Tests/LinearSolvers/CellOverset/MyTest.cpp Tests/LinearSolvers/EBTensor/MyTest.cpp Tests/LinearSolvers/NodalOverset/MyTest.cpp Tests/LinearSolvers/TensorOverset/MyTest.cpp Tools/Plotfile/fcompare.cpp Tools/Plotfile/fsnapshot.cpp Tools/Postprocessing/C_Src/particle_compare.cpp Tutorials/Amr/Advection_AmrLevel/Source/AmrLevelAdv.cpp Tutorials/LinearSolvers/ABecLaplacian_C/MyTest.cpp Tutorials/Particles/CellSortedParticles/CellSortedPC.cpp Tutorials/Particles/NeighborList/MDParticleContainer.cpp commit ad7974de5b3e68fe2f8fa7c114ffaa460ad166a2 Author: hsitaram Date: Sun Dec 6 14:08:33 2020 -0700 Eb from stl (#1592) ## Summary This is a work-in-progress PR that adds a preliminary set of tools and helper functions for using triangulated STL files in amrex based applications. Coupling with AMReX's EB framework will be added in later pull requests. This PR is done early so as to provide a software framework for incremental changes. As of now, I have added one helper function that provides a nodal multifab with a marker value indicating inside or outside the STL geometry. I guess it can be used for particle-wall collisions. I have also added a test case in the EB folder that generates a nodal multifab with the blanked values for an airfoil. The proposed changes: - [ ] fix a bug or incorrect behavior in AMReX - [x] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] are described in the proposed changes to the AMReX documentation, if appropriate Co-authored-by: Hariswaran Sitaraman Co-authored-by: Hariswaran Sitaraman Src/EB/AMReX_EB_STL_utils.H Src/EB/AMReX_EB_STL_utils.cpp Src/EB/AMReX_EB_triGeomOps_K.H Src/EB/CMakeLists.txt Src/EB/Make.package Tutorials/EB/STLtest/GNUmakefile Tutorials/EB/STLtest/Make.package Tutorials/EB/STLtest/airfoil.stl Tutorials/EB/STLtest/inputs Tutorials/EB/STLtest/main.cpp commit 8dc48ad8385cd395310567bf72a36470236e0ce2 Author: Weiqun Zhang Date: Fri Dec 4 17:18:32 2020 -0800 DPCPP: run stencil rap on device (#1606) An earlier compiler bug has been fixed. The stencil rap kernel can run on GPU now. Src/LinearSolvers/MLMG/AMReX_MLNodeLaplacian.cpp commit 0490136d291cc3b354558f091391277f16a46c5a Author: Weiqun Zhang Date: Fri Dec 4 17:11:29 2020 -0800 Remove DPCPP hacks (#1605) Because the Intel graphics driver has increased the kernel parameters to 2K by default, these hacks are no longer needed. Src/Base/AMReX_GpuLaunchFunctsG.H Src/EB/AMReX_EB2_3D_C.cpp Src/EB/AMReX_EB2_Level.cpp Src/LinearSolvers/MLMG/AMReX_MLABecLaplacian.cpp Src/LinearSolvers/MLMG/AMReX_MLEBABecLap.cpp Src/LinearSolvers/MLMG/AMReX_MLEBTensorOp.cpp Src/LinearSolvers/MLMG/AMReX_MLTensorOp.cpp commit 13d67fd18e736e96bfe4e67a6387750380900384 Author: Axel Huebl Date: Fri Dec 4 10:25:55 2020 -0800 CI: Disable DPC++ oneAPI 2020-11 (#1602) Someone patched in a dedicated CMake compiler ID for Intel DPC++ but did not add full compiler support. Release: 2021.1.1 Ref.: https://gitlab.kitware.com/cmake/cmake/-/issues/21551 .github/workflows/linux.yml commit a168f5927cee0d87ff839b23128c8fc878622313 Author: mic84 Date: Thu Dec 3 16:19:42 2020 -0800 CMake: bring back check on CUDA computing capabilities (#1600) Tools/CMake/AMReX_SetupCUDA.cmake commit 2a070723e249e3e4cf2befd61a576330fc280d80 Author: Andrew Myers Date: Thu Dec 3 14:05:14 2020 -0800 move the memcpy wrapper into amrex::Gpu (#1598) Src/Base/AMReX_GpuUtility.H commit 5b436831f47158a50738dcd64a6fe579c9ef8861 Author: Andrew Myers Date: Thu Dec 3 12:35:20 2020 -0800 Add wrapper for memcpy. (#1597) `std::memcpy` does not seem to work in device code in our HIP CI. This adds a wrapper that reverts to the global namespace `memcpy` in HIP device code. The proposed changes: - [ ] fix a bug or incorrect behavior in AMReX - [x] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] are described in the proposed changes to the AMReX documentation, if appropriate Src/Base/AMReX_GpuUtility.H commit ef665e2a2fcc245162767b3613aba48bbc59f958 Author: Axel Huebl Date: Thu Dec 3 11:16:38 2020 -0800 Fix: Windows and/or includes (iso646) (#1593) ## Summary Add missing includes to `AMReX_Extension.H`, which pulls the `` include on Windows for support for `and`/`or`. ## Additional background Seen with Clang 11.0.0 on x64 Windows with Visual Studio 2017 on conda-forge during a WarpX build. See also #947 ## Checklist The proposed changes: - [x] fix a bug or incorrect behavior in AMReX - [ ] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] are described in the proposed changes to the AMReX documentation, if appropriate Src/Base/AMReX_Extension.H Src/Base/AMReX_Vector.H Src/Particle/AMReX_ParticleTile.H commit 5c1addce382f86f0701e6bddc209522cfe70c52d Author: Axel Huebl Date: Thu Dec 3 10:33:42 2020 -0800 CI: Tutorial with OMP (#1591) Making sure the OpenMP Tutorials build as well. .github/workflows/linux.yml Tutorials/Particles/ElectromagneticPIC/CMakeLists.txt commit 0d000b5397b5f6ef94924d85b65846c5811faf86 Author: Weiqun Zhang Date: Thu Dec 3 10:23:15 2020 -0800 AMReX_Config.H (#1566) Based on discussions with @sayerhs, @ax3l, and @mic84, We have decided to make the following changes to the build systems. All headers in Src/ will include AMReX_Config.H, which will be generated by the build systems (both gnu make and cmake). The file will include all the macros we currently pass as compiler options. The purpose of this is to solve some inconsistency issues in the current systems. We support both super build and using AMReX as a library. Moreover, we do not want to force libamrex users to use the same build tool that is used to build the library. In this PR, we are put all the macro definitions in AMReX_Config.H that is included by every AMReX header. Co-authored-by: Axel Huebl Co-authored-by: Michele Rosso Src/Amr/AMReX_Amr.H Src/Amr/AMReX_AmrLevel.H Src/Amr/AMReX_AuxBoundaryData.H Src/Amr/AMReX_Derive.H Src/Amr/AMReX_Extrapolater.H Src/Amr/AMReX_LevelBld.H Src/Amr/AMReX_PROB_AMR_F.H Src/Amr/AMReX_StateData.H Src/Amr/AMReX_StateDescriptor.H Src/Amr/AMReX_extrapolater_1D_K.H Src/Amr/AMReX_extrapolater_2D_K.H Src/Amr/AMReX_extrapolater_3D_K.H Src/Amr/AMReX_extrapolater_K.H Src/AmrCore/AMReX_AmrCore.H Src/AmrCore/AMReX_AmrMesh.H Src/AmrCore/AMReX_AmrParGDB.H Src/AmrCore/AMReX_AmrParticles.H Src/AmrCore/AMReX_Cluster.H Src/AmrCore/AMReX_ErrorList.H Src/AmrCore/AMReX_FLUXREG_F.H Src/AmrCore/AMReX_FillPatchUtil.H Src/AmrCore/AMReX_FillPatchUtil_F.H Src/AmrCore/AMReX_FillPatchUtil_I.H Src/AmrCore/AMReX_FluxReg_1D_C.H Src/AmrCore/AMReX_FluxReg_2D_C.H Src/AmrCore/AMReX_FluxReg_3D_C.H Src/AmrCore/AMReX_FluxReg_C.H Src/AmrCore/AMReX_FluxRegister.H Src/AmrCore/AMReX_INTERP_F.H Src/AmrCore/AMReX_Interp_1D_C.H Src/AmrCore/AMReX_Interp_2D_C.H Src/AmrCore/AMReX_Interp_3D_C.H Src/AmrCore/AMReX_Interp_C.H Src/AmrCore/AMReX_Interpolater.H Src/AmrCore/AMReX_TagBox.H Src/Base/AMReX.H Src/Base/AMReX_Algorithm.H Src/Base/AMReX_Arena.H Src/Base/AMReX_Array.H Src/Base/AMReX_Array4.H Src/Base/AMReX_ArrayLim.H Src/Base/AMReX_AsyncOut.H Src/Base/AMReX_BArena.H Src/Base/AMReX_BCRec.H Src/Base/AMReX_BCUtil.H Src/Base/AMReX_BC_TYPES.H Src/Base/AMReX_BLBackTrace.H Src/Base/AMReX_BLFort.H Src/Base/AMReX_BLProfiler.H Src/Base/AMReX_BLassert.H Src/Base/AMReX_BackgroundThread.H Src/Base/AMReX_BaseFab.H Src/Base/AMReX_BaseFabUtility.H Src/Base/AMReX_BaseUmap.H Src/Base/AMReX_BaseUmap_f.H Src/Base/AMReX_BlockMutex.H Src/Base/AMReX_Box.H Src/Base/AMReX_BoxArray.H Src/Base/AMReX_BoxDomain.H Src/Base/AMReX_BoxIterator.H Src/Base/AMReX_BoxList.H Src/Base/AMReX_CArena.H Src/Base/AMReX_CONSTANTS.H Src/Base/AMReX_COORDSYS_1D_C.H Src/Base/AMReX_COORDSYS_2D_C.H Src/Base/AMReX_COORDSYS_3D_C.H Src/Base/AMReX_COORDSYS_C.H Src/Base/AMReX_CoordSys.H Src/Base/AMReX_CudaGraph.H Src/Base/AMReX_CuptiTrace.H Src/Base/AMReX_DArena.H Src/Base/AMReX_Dim3.H Src/Base/AMReX_DistributionMapping.H Src/Base/AMReX_EArena.H Src/Base/AMReX_Exception.H Src/Base/AMReX_Extension.H Src/Base/AMReX_FACopyDescriptor.H Src/Base/AMReX_FArrayBox.H Src/Base/AMReX_FPC.H Src/Base/AMReX_FabArray.H Src/Base/AMReX_FabArrayBase.H Src/Base/AMReX_FabArrayUtility.H Src/Base/AMReX_FabConv.H Src/Base/AMReX_FabFactory.H Src/Base/AMReX_FilCC_1D_C.H Src/Base/AMReX_FilCC_2D_C.H Src/Base/AMReX_FilCC_3D_C.H Src/Base/AMReX_FilCC_C.H Src/Base/AMReX_FilND_C.H Src/Base/AMReX_FileSystem.H Src/Base/AMReX_ForkJoin.H Src/Base/AMReX_Functional.H Src/Base/AMReX_Geometry.H Src/Base/AMReX_Gpu.H Src/Base/AMReX_GpuAllocators.H Src/Base/AMReX_GpuAssert.H Src/Base/AMReX_GpuAsyncArray.H Src/Base/AMReX_GpuAtomic.H Src/Base/AMReX_GpuComplex.H Src/Base/AMReX_GpuContainers.H Src/Base/AMReX_GpuControl.H Src/Base/AMReX_GpuDevice.H Src/Base/AMReX_GpuElixir.H Src/Base/AMReX_GpuError.H Src/Base/AMReX_GpuFuse.H Src/Base/AMReX_GpuKernelInfo.H Src/Base/AMReX_GpuLaunch.H Src/Base/AMReX_GpuLaunchFunctsC.H Src/Base/AMReX_GpuLaunchFunctsG.H Src/Base/AMReX_GpuLaunchGlobal.H Src/Base/AMReX_GpuLaunchMacrosC.H Src/Base/AMReX_GpuLaunchMacrosG.H Src/Base/AMReX_GpuMemory.H Src/Base/AMReX_GpuPrint.H Src/Base/AMReX_GpuQualifiers.H Src/Base/AMReX_GpuRange.H Src/Base/AMReX_GpuReduce.H Src/Base/AMReX_GpuTypes.H Src/Base/AMReX_GpuUtility.H Src/Base/AMReX_IArrayBox.H Src/Base/AMReX_INT.H Src/Base/AMReX_IndexSequence.H Src/Base/AMReX_IndexType.H Src/Base/AMReX_IntConv.H Src/Base/AMReX_IntVect.H Src/Base/AMReX_LayoutData.H Src/Base/AMReX_Lazy.H Src/Base/AMReX_Loop.H Src/Base/AMReX_MFCopyDescriptor.H Src/Base/AMReX_MFIter.H Src/Base/AMReX_Machine.H Src/Base/AMReX_Machine.cpp Src/Base/AMReX_MakeType.H Src/Base/AMReX_Math.H Src/Base/AMReX_MemPool.H Src/Base/AMReX_MemProfiler.H Src/Base/AMReX_MultiFab.H Src/Base/AMReX_MultiFabUtil.H Src/Base/AMReX_MultiFabUtil_1D_C.H Src/Base/AMReX_MultiFabUtil_2D_C.H Src/Base/AMReX_MultiFabUtil_3D_C.H Src/Base/AMReX_MultiFabUtil_C.H Src/Base/AMReX_MultiFabUtil_nd_C.H Src/Base/AMReX_NFiles.H Src/Base/AMReX_NonLocalBC.H Src/Base/AMReX_NonLocalBCImpl.H Src/Base/AMReX_OpenMP.H Src/Base/AMReX_Orientation.H Src/Base/AMReX_PODVector.H Src/Base/AMReX_ParallelContext.H Src/Base/AMReX_ParallelDescriptor.H Src/Base/AMReX_ParallelReduce.H Src/Base/AMReX_ParmParse.H Src/Base/AMReX_Partition.H Src/Base/AMReX_Periodicity.H Src/Base/AMReX_PhysBCFunct.H Src/Base/AMReX_PlotFileDataImpl.H Src/Base/AMReX_PlotFileUtil.H Src/Base/AMReX_Print.H Src/Base/AMReX_REAL.H Src/Base/AMReX_Random.H Src/Base/AMReX_RandomEngine.H Src/Base/AMReX_RealBox.H Src/Base/AMReX_RealVect.H Src/Base/AMReX_Reduce.H Src/Base/AMReX_SPACE.H Src/Base/AMReX_Scan.H Src/Base/AMReX_Slopes_K.H Src/Base/AMReX_ThirdPartyProfiling.H Src/Base/AMReX_TinyProfiler.H Src/Base/AMReX_Tuple.H Src/Base/AMReX_TypeTraits.H Src/Base/AMReX_Utility.H Src/Base/AMReX_Vector.H Src/Base/AMReX_VectorIO.H Src/Base/AMReX_VisMF.H Src/Base/AMReX_ccse-mpi.H Src/Base/AMReX_filcc_f.H Src/Base/AMReX_iMultiFab.H Src/Base/AMReX_parstream.H Src/Boundary/AMReX_BndryData.H Src/Boundary/AMReX_BndryRegister.H Src/Boundary/AMReX_BoundCond.H Src/Boundary/AMReX_FabSet.H Src/Boundary/AMReX_InterpBndryData.H Src/Boundary/AMReX_InterpBndryData_1D_K.H Src/Boundary/AMReX_InterpBndryData_2D_K.H Src/Boundary/AMReX_InterpBndryData_3D_K.H Src/Boundary/AMReX_InterpBndryData_K.H Src/Boundary/AMReX_LOUtil_K.H Src/Boundary/AMReX_LO_BCTYPES.H Src/Boundary/AMReX_MacBndry.H Src/Boundary/AMReX_Mask.H Src/Boundary/AMReX_MultiMask.H Src/Boundary/AMReX_YAFluxRegister.H Src/Boundary/AMReX_YAFluxRegister_1D_K.H Src/Boundary/AMReX_YAFluxRegister_2D_K.H Src/Boundary/AMReX_YAFluxRegister_3D_K.H Src/Boundary/AMReX_YAFluxRegister_K.H Src/CMakeLists.txt Src/EB/AMReX_EB2.H Src/EB/AMReX_EB2_2D_C.H Src/EB/AMReX_EB2_3D_C.H Src/EB/AMReX_EB2_C.H Src/EB/AMReX_EB2_GeometryShop.H Src/EB/AMReX_EB2_Graph.H Src/EB/AMReX_EB2_IF.H Src/EB/AMReX_EB2_IF_AllRegular.H Src/EB/AMReX_EB2_IF_Base.H Src/EB/AMReX_EB2_IF_Box.H Src/EB/AMReX_EB2_IF_Complement.H Src/EB/AMReX_EB2_IF_Cylinder.H Src/EB/AMReX_EB2_IF_Difference.H Src/EB/AMReX_EB2_IF_Ellipsoid.H Src/EB/AMReX_EB2_IF_Extrusion.H Src/EB/AMReX_EB2_IF_Intersection.H Src/EB/AMReX_EB2_IF_Lathe.H Src/EB/AMReX_EB2_IF_Plane.H Src/EB/AMReX_EB2_IF_Polynomial.H Src/EB/AMReX_EB2_IF_Rotation.H Src/EB/AMReX_EB2_IF_Scale.H Src/EB/AMReX_EB2_IF_Sphere.H Src/EB/AMReX_EB2_IF_Spline.H Src/EB/AMReX_EB2_IF_Torus.H Src/EB/AMReX_EB2_IF_Translation.H Src/EB/AMReX_EB2_IF_Union.H Src/EB/AMReX_EB2_Level.H Src/EB/AMReX_EB2_MultiGFab.H Src/EB/AMReX_EBAmrUtil.H Src/EB/AMReX_EBCellFlag.H Src/EB/AMReX_EBDataCollection.H Src/EB/AMReX_EBFArrayBox.H Src/EB/AMReX_EBFabFactory.H Src/EB/AMReX_EBFluxRegister.H Src/EB/AMReX_EBFluxRegister_2D_C.H Src/EB/AMReX_EBFluxRegister_3D_C.H Src/EB/AMReX_EBFluxRegister_C.H Src/EB/AMReX_EBInterpolater.H Src/EB/AMReX_EBMultiFabUtil.H Src/EB/AMReX_EBMultiFabUtil_2D_C.H Src/EB/AMReX_EBMultiFabUtil_3D_C.H Src/EB/AMReX_EBMultiFabUtil_C.H Src/EB/AMReX_EBSupport.H Src/EB/AMReX_EB_slopes_K.H Src/EB/AMReX_EB_utils.H Src/EB/AMReX_MultiCutFab.H Src/EB/AMReX_WriteEBSurface.H Src/EB/AMReX_WriteEB_F.H Src/EB/AMReX_algoim.H Src/EB/AMReX_algoim_K.H Src/EB/AMReX_distFcnElement.H Src/Extern/Conduit/AMReX_Conduit_Blueprint.H Src/Extern/HYPRE/AMReX_Habec_2D_K.H Src/Extern/HYPRE/AMReX_Habec_3D_K.H Src/Extern/HYPRE/AMReX_Habec_K.H Src/Extern/HYPRE/AMReX_Hypre.H Src/Extern/HYPRE/AMReX_HypreABecLap.H Src/Extern/HYPRE/AMReX_HypreABecLap2.H Src/Extern/HYPRE/AMReX_HypreABecLap3.H Src/Extern/HYPRE/AMReX_HypreIJIface.H Src/Extern/HYPRE/AMReX_HypreNodeLap.H Src/Extern/HYPRE/CMakeLists.txt Src/Extern/PETSc/AMReX_PETSc.H Src/Extern/PETSc/CMakeLists.txt Src/Extern/ProfParser/AMReX_AVGDOWN_F.H Src/Extern/ProfParser/AMReX_BLProfStats.H Src/Extern/ProfParser/AMReX_BLProfUtilities.H Src/Extern/ProfParser/AMReX_BLWritePlotFile.H Src/Extern/ProfParser/AMReX_CommProfStats.H Src/Extern/ProfParser/AMReX_RegionsProfStats.H Src/Extern/ProfParser/CMakeLists.txt Src/Extern/SENSEI/AMReX_AmrDataAdaptor.H Src/Extern/SENSEI/AMReX_AmrInSituBridge.H Src/Extern/SENSEI/AMReX_AmrMeshDataAdaptor.H Src/Extern/SENSEI/AMReX_AmrMeshInSituBridge.H Src/Extern/SENSEI/AMReX_InSituBridge.H Src/Extern/SENSEI/AMReX_InSituUtils.H Src/Extern/SENSEI/CMakeLists.txt Src/Extern/SWFFT/Dfft.H Src/Extern/amrdata/AMReX_AmrData.H Src/Extern/amrdata/AMReX_AmrvisConstants.H Src/Extern/amrdata/AMReX_DataServices.H Src/Extern/amrdata/AMReX_XYPlotDataList.H Src/Extern/hpgmg/BL_HPGMG.H Src/F_Interfaces/AmrCore/AMReX_FAmrCore.H Src/F_Interfaces/AmrCore/AMReX_FlashFluxRegister.H Src/F_Interfaces/Base/AMReX_FPhysBC.H Src/F_Interfaces/CMakeLists.txt Src/LinearSolvers/MLMG/AMReX_MLABecLap_1D_K.H Src/LinearSolvers/MLMG/AMReX_MLABecLap_2D_K.H Src/LinearSolvers/MLMG/AMReX_MLABecLap_3D_K.H Src/LinearSolvers/MLMG/AMReX_MLABecLap_K.H Src/LinearSolvers/MLMG/AMReX_MLABecLaplacian.H Src/LinearSolvers/MLMG/AMReX_MLALap_1D_K.H Src/LinearSolvers/MLMG/AMReX_MLALap_2D_K.H Src/LinearSolvers/MLMG/AMReX_MLALap_3D_K.H Src/LinearSolvers/MLMG/AMReX_MLALap_K.H Src/LinearSolvers/MLMG/AMReX_MLALaplacian.H Src/LinearSolvers/MLMG/AMReX_MLCGSolver.H Src/LinearSolvers/MLMG/AMReX_MLCellABecLap.H Src/LinearSolvers/MLMG/AMReX_MLCellLinOp.H Src/LinearSolvers/MLMG/AMReX_MLEBABecLap.H Src/LinearSolvers/MLMG/AMReX_MLEBABecLap_2D_K.H Src/LinearSolvers/MLMG/AMReX_MLEBABecLap_3D_K.H Src/LinearSolvers/MLMG/AMReX_MLEBABecLap_K.H Src/LinearSolvers/MLMG/AMReX_MLEBTensorOp.H Src/LinearSolvers/MLMG/AMReX_MLEBTensor_2D_K.H Src/LinearSolvers/MLMG/AMReX_MLEBTensor_3D_K.H Src/LinearSolvers/MLMG/AMReX_MLEBTensor_K.H Src/LinearSolvers/MLMG/AMReX_MLLinOp.H Src/LinearSolvers/MLMG/AMReX_MLLinOp_F.H Src/LinearSolvers/MLMG/AMReX_MLLinOp_K.H Src/LinearSolvers/MLMG/AMReX_MLMG.H Src/LinearSolvers/MLMG/AMReX_MLMGBndry.H Src/LinearSolvers/MLMG/AMReX_MLMG_1D_K.H Src/LinearSolvers/MLMG/AMReX_MLMG_2D_K.H Src/LinearSolvers/MLMG/AMReX_MLMG_3D_K.H Src/LinearSolvers/MLMG/AMReX_MLMG_K.H Src/LinearSolvers/MLMG/AMReX_MLNodeLap_1D_K.H Src/LinearSolvers/MLMG/AMReX_MLNodeLap_2D_K.H Src/LinearSolvers/MLMG/AMReX_MLNodeLap_3D_K.H Src/LinearSolvers/MLMG/AMReX_MLNodeLap_K.H Src/LinearSolvers/MLMG/AMReX_MLNodeLaplacian.H Src/LinearSolvers/MLMG/AMReX_MLNodeLinOp.H Src/LinearSolvers/MLMG/AMReX_MLNodeTensorLap_1D_K.H Src/LinearSolvers/MLMG/AMReX_MLNodeTensorLap_2D_K.H Src/LinearSolvers/MLMG/AMReX_MLNodeTensorLap_3D_K.H Src/LinearSolvers/MLMG/AMReX_MLNodeTensorLap_K.H Src/LinearSolvers/MLMG/AMReX_MLNodeTensorLaplacian.H Src/LinearSolvers/MLMG/AMReX_MLPoisson.H Src/LinearSolvers/MLMG/AMReX_MLPoisson_1D_K.H Src/LinearSolvers/MLMG/AMReX_MLPoisson_2D_K.H Src/LinearSolvers/MLMG/AMReX_MLPoisson_3D_K.H Src/LinearSolvers/MLMG/AMReX_MLPoisson_K.H Src/LinearSolvers/MLMG/AMReX_MLTensorOp.H Src/LinearSolvers/MLMG/AMReX_MLTensor_1D_K.H Src/LinearSolvers/MLMG/AMReX_MLTensor_2D_K.H Src/LinearSolvers/MLMG/AMReX_MLTensor_3D_K.H Src/LinearSolvers/MLMG/AMReX_MLTensor_K.H Src/LinearSolvers/Projections/AMReX_MacProjector.H Src/LinearSolvers/Projections/AMReX_NodalProjector.H Src/Particle/AMReX_ArrayOfStructs.H Src/Particle/AMReX_BinIterator.H Src/Particle/AMReX_DenseBins.H Src/Particle/AMReX_NeighborList.H Src/Particle/AMReX_NeighborParticles.H Src/Particle/AMReX_NeighborParticlesCPUImpl.H Src/Particle/AMReX_NeighborParticlesGPUImpl.H Src/Particle/AMReX_ParGDB.H Src/Particle/AMReX_ParIter.H Src/Particle/AMReX_Particle.H Src/Particle/AMReX_ParticleBufferMap.H Src/Particle/AMReX_ParticleCommunication.H Src/Particle/AMReX_ParticleHDF5.H Src/Particle/AMReX_ParticleIO.H Src/Particle/AMReX_ParticleInit.H Src/Particle/AMReX_ParticleLocator.H Src/Particle/AMReX_ParticleMPIUtil.H Src/Particle/AMReX_ParticleMesh.H Src/Particle/AMReX_ParticleReduce.H Src/Particle/AMReX_ParticleTile.H Src/Particle/AMReX_ParticleTransformation.H Src/Particle/AMReX_ParticleUtil.H Src/Particle/AMReX_Particle_mod_K.H Src/Particle/AMReX_Particles.H Src/Particle/AMReX_SparseBins.H Src/Particle/AMReX_StructOfArrays.H Src/Particle/AMReX_TracerParticle_mod_K.H Src/Particle/AMReX_TracerParticles.H Src/Particle/AMReX_WriteBinaryParticleData.H Src/Particle/CMakeLists.txt Src/SDC/AMReX_SDCstruct.H Tools/CMake/AMReXGenerateConfigHeader.cmake Tools/CMake/AMReXInstallHelpers.cmake Tools/CMake/AMReXSetDefines.cmake Tools/CMake/AMReX_Config.H.in Tools/CMake/AMReX_Config.cmake Tools/CMake/AMReX_Defines.cmake Tools/CMake/AMReX_Utils.cmake Tools/CMake/modify_installed_headers.cmake Tools/GNUMake/Make.defs Tools/GNUMake/Make.rules Tools/libamrex/mkconfig.py Tutorials/Amr/Advection_AmrLevel/Source/AmrLevelAdv.H commit 5fb87ea35fa521e9ce97dc9d36f21eda5e2343be Author: Shreyas Ananthan Date: Thu Dec 3 09:19:23 2020 -0700 Restore reproducibility of solutions with MacProjector reuse (#1596) Src/LinearSolvers/Projections/AMReX_MacProjector.cpp commit ae4cc9a1488719e284409b078887866fe65709d9 Author: mic84 Date: Wed Dec 2 15:12:33 2020 -0800 CMake: output missing components list when AMReX is not found (#1572) Tools/CMake/AMReXConfig.cmake.in Tools/CMake/AMReXInstallHelpers.cmake commit 75c0896594cd38d883d7907466657eac3ca1a188 Author: Axel Huebl Date: Wed Dec 2 15:09:17 2020 -0800 CI: Windows + Clang (#1588) Add Clang as Windows target. This compiler offers potentially a route to modern OpenMP. (MSVC: 2.0 standard, LLVM/Clang: 3.1 standard) .github/workflows/windows.yml commit 91bdc795d598b8bbf58d640d572f27dfa5ae31a5 Author: Shreyas Ananthan Date: Wed Dec 2 14:44:56 2020 -0700 MacProjector: Update setDomainBC API (#1589) Allow `setDomainBC` to be called before `setUMAC` has been called. Src/LinearSolvers/Projections/AMReX_MacProjector.cpp commit 820a30d7966426a96e24e3963b37f3789eec258f Author: Andrew Myers Date: Wed Dec 2 13:26:03 2020 -0800 Fix InitNRandomPerCell for single precision particles (#1582) * due to roundoff errors, we need to redraw these numbers sometimes when single precision particles are used. * missing ; Src/Particle/AMReX_ParticleInit.H commit dc0ec74c4d120509f9754fb7d43af6d275d9c1a4 Author: Shreyas Ananthan Date: Wed Dec 2 13:35:05 2020 -0700 DPCPP: Update CMake options to allow building with MPI+DPCPP (#1586) Tools/CMake/AMReXOptions.cmake Tools/CMake/AMReXSYCL.cmake commit 89220ed73d9fe74618386f8b1bfafd30760d6fde Author: Andrew Myers Date: Tue Dec 1 10:31:23 2020 -0800 Reimplement NumParticlesAtLevel (#1581) This removes a workaround that had this function fall back to CPUs for CUDA (but not HIP or DPC++). Additionally, this re-implements the function in terms of ReduceTuple to avoid multiple memcpy operations in the case of multiple boxes. Src/Particle/AMReX_ParticleContainerI.H Tests/Particles/ParticleReduce/main.cpp commit bbcce736ea9497cdb4722e8b4f4c2ee0d7c193ca Author: Robert Maynard Date: Tue Dec 1 13:24:16 2020 -0500 eval_genex correctly replaces the matching compiler version (#1580) Previously it would place the resulting output in the variable `1` instead of `_in`. Additionally it would try to add an extra `>`. Tools/CMake/AMReXGenexHelpers.cmake commit 9a523cf95dc3046081b0830cf435a3455929728d Author: Robert Maynard Date: Tue Dec 1 13:20:19 2020 -0500 Add msvc cuda support (#1573) ## Summary This set of changes has allowed me to build AMReX on windows using CUDA 11.X and VS2019. ## Checklist The proposed changes: - [x] fix a bug or incorrect behavior in AMReX - [ ] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] are described in the proposed changes to the AMReX documentation, if appropriate Src/AmrCore/AMReX_ErrorList.cpp Src/Base/AMReX.H Src/Base/AMReX.cpp Src/Base/AMReX_BLBackTrace.cpp Src/Base/AMReX_Extension.H Src/Base/AMReX_GpuDevice.cpp Src/Base/AMReX_MultiFab.H Tools/CMake/AMReX_Config.cmake commit 1cb99250ecd4da445cfc8afe325664030c49c4f0 Author: Weiqun Zhang Date: Tue Dec 1 10:12:38 2020 -0800 Update CHANGES (#1584) CHANGES commit 94ca530cb43c497cf22dfbfbd13dfd087f7d910d Author: Andrew Myers Date: Mon Nov 30 21:55:26 2020 -0500 Fix bug affecting async IO + single precision particles (#1583) The proposed changes: - [x] fix a bug or incorrect behavior in AMReX - [ ] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] are described in the proposed changes to the AMReX documentation, if appropriate Src/Particle/AMReX_WriteBinaryParticleData.H commit d9af9ab7f134b04bc5e3cc57362e82a64fecef0a Author: Weiqun Zhang Date: Mon Nov 30 12:49:50 2020 -0800 Fix warnings for DPCPP (#1579) Fix unused variable. Do not pass `-fsycl-unnamed-lambda` to `mpiicpx` for linking to get rid of warning on unknown argument. Src/Base/AMReX_FBI.H Tools/GNUMake/Make.defs Tools/GNUMake/Make.rules Tutorials/GPU/CNS/Source/CNS.H commit 07afa8ec8f59403ac54c2b2c566c1c88b1863112 Author: Weiqun Zhang Date: Mon Nov 30 10:25:56 2020 -0800 Template specialization of MPI communication functions for char (#1569) * Add template specialization of MPI communication functions for char. Select a proper data type according to the size. * Use the new functions in FabArray and Particle communication. And this also fixes a bug in Particle communication. (The counts should be divided by the size of data type.) * Due to a gcc bug on explicit specialization in a namespace, workaround was implemented in AMReX_ParallelDescriptor.H and .cpp. Src/Base/AMReX_Arena.H Src/Base/AMReX_Arena.cpp Src/Base/AMReX_FabArrayCommI.H Src/Base/AMReX_ParallelDescriptor.H Src/Base/AMReX_ParallelDescriptor.cpp Src/Particle/AMReX_ParticleCommunication.H commit 50c40c46382a83cdc98fd59e337c5037aba07c1e Author: Weiqun Zhang Date: Mon Nov 30 10:16:16 2020 -0800 EB2::PolyIF (#1567) The existing PolynomialIF contains a dynamic size vector. This makes it hard to support it on GPU. We have used an undocumented feature to get this run on GPU. However, this code can no longer compile, because the GNU make system now makes cross execution space calls an error. So we disable GPU support for PolynomialIF and add a new PolyIF class with fixed size array and therefore GPU support. Src/EB/AMReX_EB2_IF_Polynomial.H commit 26e4d30f346cb89921502e409da1b933a60f9c81 Author: Weiqun Zhang Date: Mon Nov 30 10:11:37 2020 -0800 Add .gitattributes (#1578) Make github detect *.H as C++. This file is borrowed from Castro. .gitattributes commit a77968674f3d9ec1d4298d99a506dd1951f88c67 Author: Shreyas Ananthan Date: Mon Nov 30 09:17:48 2020 -0700 Tools/Plotfile: Fix build errors in single precision mode (#1577) This PR fixes one build error and one warning observed when compiling `Tools/Plotfile` utilities with `-DAMREX_USE_FLOAT` Tools/Plotfile/fcompare.cpp Tools/Plotfile/fextract.cpp commit 82eceae29419ceec24450926af1f5445ac425d00 Author: Shreyas Ananthan Date: Mon Nov 30 09:14:54 2020 -0700 Arena: Add accessor to query Arena attributes through info object (#1575) ## Summary This PR adds an accessor to the `Arena` class that can be used to query the attributes of the Arena (e.g., `device_use_managed_memory` flag) from the application codes. ## Additional background Currently, there is no easy way to determine if `amrex::The_Arena()` is using managed memory. Applications would have to use a combination of `AMREX_USE_HIP` compile-time definition and `amrex.the_arena_is_managed` ParmParse variable to determine this. Testing individual pointers with `amrex::Gpu::isManaged()` is possible only on CUDA/HIP but not DPC++. Also it is costly compared to just querying the `Arena` object. Allowing read-only access to `ArenaInfo` object will allow a wide variety of checks in application code. ## Checklist The proposed changes: - [ ] fix a bug or incorrect behavior in AMReX - [X] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] are described in the proposed changes to the AMReX documentation, if appropriate Src/Base/AMReX_Arena.H commit acdb7eac1d5a1744e87c0caafd1c43cc5e1171e9 Author: Luca Fedeli Date: Mon Nov 30 16:58:32 2020 +0100 Fix some warnings related to double-->float conversions (#1571) I tried to compile `WarpX` in single precision and I noticed several warnings related to casts from `double` to `float` in `AMReX`. This PR should fix some of these warnings. I plan to continue in the next days with other PRs, if you agree. Src/AmrCore/AMReX_AmrMesh.H Src/Base/AMReX_CoordSys.H Src/Base/AMReX_FilCC_2D_C.H Src/Base/AMReX_Geometry.H Src/Base/AMReX_MultiFabUtil_2D_C.H Src/Base/AMReX_MultiFabUtil_nd_C.H commit abdab63376660238396626ee064130a5c1826024 Author: Shreyas Ananthan Date: Sun Nov 29 14:04:44 2020 -0700 Fix assertion check for MacProjection (#1576) Fixes bug introduced in #1574 Src/LinearSolvers/Projections/AMReX_MacProjector.cpp commit abeb357db57261fe8a8d811e161da0c6c2814d92 Author: Shreyas Ananthan Date: Sun Nov 29 09:53:42 2020 -0700 Allow MacProjection reuse (#1574) This commit modifies the MacProjector interface to allow reuse of the MacProjector. The changes to the API are backwards compatible. - Add a new constructor that does not require specification of beta, umac etc. - Add `initProjector` to explicitly initialize the projection instance at app level - Add methods to set MAC velocities, divU etc. Src/LinearSolvers/Projections/AMReX_MacProjector.H Src/LinearSolvers/Projections/AMReX_MacProjector.cpp commit cc8f323c426f3957619c93477af2b9e32e3b49c2 Author: Axel Huebl Date: Mon Nov 23 22:27:06 2020 -0800 CI: Make DPC++ Pretty (#1568) * CI: Make DPC++ Pretty Just combing the show pony. * oneAPI Activation: still needs set +e .github/workflows/linux.yml commit f068164b513c12828dac069510d64f8b4912028e Author: mic84 Date: Mon Nov 23 14:48:58 2020 -0800 Remove SUNDIALS support (#1559) ## Summary SUNDIALS support in AMReX consisted of only few Fortran interfaces to SUNDIALS functions for the convenience of app codes. Since there are no AMReX functionalities depending on SUNDIALS, support for SUNDIALS will be removed. ## Additional background ## Checklist The proposed changes: - [ ] fix a bug or incorrect behavior in AMReX - [ ] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] are described in the proposed changes to the AMReX documentation, if appropriate Docs/sphinx_documentation/source/BuildingAMReX.rst Docs/sphinx_documentation/source/SUNDIALS.rst Docs/sphinx_documentation/source/SUNDIALS_CVODE.rst Docs/sphinx_documentation/source/SUNDIALS_top.rst Docs/sphinx_tutorials/source/SUNDIALS_Tutorial.rst Src/CMakeLists.txt Src/Extern/CVODE/Make.package Src/Extern/CVODE/cvode_interface.f90 Src/Extern/CVODE/fnvector_serial.f90 Src/Extern/CVODE/integrator_stats.f90 Src/Extern/CVODE/sundials_fdlsmat.f90 Src/Extern/SUNDIALS/CMakeLists.txt Src/Extern/SUNDIALS/Make.package Src/Extern/SUNDIALS/arkode_interface.f90 Src/Extern/SUNDIALS/cvode_interface.f90 Tools/CMake/AMReXConfig.cmake.in Tools/CMake/AMReXOptions.cmake Tools/CMake/AMReXThirdPartyLibraries.cmake Tools/CMake/FindSUNDIALS.cmake Tools/GNUMake/Make.defs Tools/GNUMake/packages/Make.cvode Tools/GNUMake/packages/Make.sundials Tutorials/CMakeLists.txt Tutorials/CVODE/EX1_F/GNUmakefile Tutorials/CVODE/EX1_F/Make.package Tutorials/CVODE/EX1_F/inputs Tutorials/CVODE/EX1_F/integrate_ode.f90 Tutorials/CVODE/EX1_F/main.cpp Tutorials/CVODE/EX1_F/myfunc_F.H Tutorials/CVODE/EX1_F/ode_mod.f90 Tutorials/CVODE/EX2_F/GNUmakefile Tutorials/CVODE/EX2_F/Make.package Tutorials/CVODE/EX2_F/inputs Tutorials/CVODE/EX2_F/integrate_ode_no_jac.f90 Tutorials/CVODE/EX2_F/integrate_ode_with_jac.f90 Tutorials/CVODE/EX2_F/main.cpp Tutorials/CVODE/EX2_F/myfunc_F.H Tutorials/CVODE/EX2_F/ode_mod.f90 Tutorials/SUNDIALS/EX-CUSOLVER/GNUmakefile Tutorials/SUNDIALS/EX-CUSOLVER/Make.CVODE Tutorials/SUNDIALS/EX-CUSOLVER/Make.package Tutorials/SUNDIALS/EX-CUSOLVER/README.md Tutorials/SUNDIALS/EX-CUSOLVER/extern_probin.template Tutorials/SUNDIALS/EX-CUSOLVER/inputs Tutorials/SUNDIALS/EX-CUSOLVER/inputs_128 Tutorials/SUNDIALS/EX-CUSOLVER/inputs_256 Tutorials/SUNDIALS/EX-CUSOLVER/inputs_32 Tutorials/SUNDIALS/EX-CUSOLVER/inputs_64 Tutorials/SUNDIALS/EX-CUSOLVER/main.cpp Tutorials/SUNDIALS/EX-CUSOLVER/make_cuda.sh Tutorials/SUNDIALS/EX-CUSOLVER/make_cuda_cusolver.sh Tutorials/SUNDIALS/EX-CUSOLVER/make_serial.sh Tutorials/SUNDIALS/EX-CUSOLVER/react_cuda.cpp Tutorials/SUNDIALS/EX-CUSOLVER/react_cuda_cusolver.cpp Tutorials/SUNDIALS/EX-CUSOLVER/react_serial.cpp Tutorials/SUNDIALS/EX-CUSOLVER/react_system.F90 Tutorials/SUNDIALS/EX-CUSOLVER/react_utils.F90 Tutorials/SUNDIALS/EX-CUSOLVER/test_react.H Tutorials/SUNDIALS/EX-CUSOLVER/test_react_F.H Tutorials/SUNDIALS/EX1_C/CMakeLists.txt Tutorials/SUNDIALS/EX1_C/GNUmakefile Tutorials/SUNDIALS/EX1_C/Make.package Tutorials/SUNDIALS/EX1_C/SetIC.f90 Tutorials/SUNDIALS/EX1_C/inputs Tutorials/SUNDIALS/EX1_C/inputs_2box Tutorials/SUNDIALS/EX1_C/inputs_non_vectorized Tutorials/SUNDIALS/EX1_C/main.cpp Tutorials/SUNDIALS/EX1_C/myfunc_F.H Tutorials/SUNDIALS/EX1_CUDA/GNUmakefile Tutorials/SUNDIALS/EX1_CUDA/Make.package Tutorials/SUNDIALS/EX1_CUDA/SetIC.f90 Tutorials/SUNDIALS/EX1_CUDA/inputs Tutorials/SUNDIALS/EX1_CUDA/main.cpp Tutorials/SUNDIALS/EX1_CUDA/myfunc_F.H Tutorials/SUNDIALS/EX1_CUDA/ode_mod.f90 Tutorials/SUNDIALS/EX1_F/GNUmakefile Tutorials/SUNDIALS/EX1_F/Make.package Tutorials/SUNDIALS/EX1_F/inputs Tutorials/SUNDIALS/EX1_F/integrate_ode.f90 Tutorials/SUNDIALS/EX1_F/main.cpp Tutorials/SUNDIALS/EX1_F/myfunc_F.H Tutorials/SUNDIALS/EX1_F/ode_mod.f90 Tutorials/SUNDIALS/EX2_F/GNUmakefile Tutorials/SUNDIALS/EX2_F/Make.package Tutorials/SUNDIALS/EX2_F/inputs Tutorials/SUNDIALS/EX2_F/integrate_ode_no_jac.f90 Tutorials/SUNDIALS/EX2_F/integrate_ode_with_jac.f90 Tutorials/SUNDIALS/EX2_F/main.cpp Tutorials/SUNDIALS/EX2_F/myfunc_F.H Tutorials/SUNDIALS/EX2_F/ode_mod.f90 commit e1ff59c8d9207a135f25284380a1ba8fc15abb1f Author: Roberto Porcu <53792251+rporcu@users.noreply.github.com> Date: Sun Nov 22 23:19:06 2020 -0500 allow MLEBABecLap to solve for multiple components (#1557) ## Summary MLLinOp::getNComp() is returning 1 and right now MLEBABecLap is working only with MultiFabs having 1 component. ## Additional background This PR has the goal to generalize MLEBABecLap solver for a generic N number of components. ## Checklist The proposed changes: - [ ] fix a bug or incorrect behavior in AMReX - [x] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] are described in the proposed changes to the AMReX documentation, if appropriate Co-authored-by: Weiqun Zhang Src/LinearSolvers/MLMG/AMReX_MLEBABecLap.H Src/LinearSolvers/MLMG/AMReX_MLEBABecLap.cpp commit 1f8163c397e1aa33fd633219166b87d4633a0b46 Author: Weiqun Zhang Date: Sun Nov 22 19:40:00 2020 -0800 DPCPP: MPI (#1556) Allow MPI for dpcpp build in gnu make assuming mpiicpx is used. Tools/GNUMake/Make.defs Tools/GNUMake/Make.machines Tools/GNUMake/comps/dpcpp.mak Tools/GNUMake/sites/Make.alcf commit a85fd8d7971f9448b8a00e796a392bbe84ae0f97 Author: Axel Huebl Date: Fri Nov 20 10:28:25 2020 -0800 CMake: OpenMP for Fortran (#1563) * CMake: OpenMP for Fortran Fix missing activation. * Update Tools/CMake/AMReXParallelBackends.cmake Co-authored-by: mic84 * CMakeL OpenMP Fortran fix Config, too Co-authored-by: mic84 Tools/CMake/AMReXConfig.cmake.in Tools/CMake/AMReXParallelBackends.cmake commit 7a0fea34be764441014e2ba7dac44fecbdbbabfb Author: Weiqun Zhang Date: Thu Nov 19 11:01:07 2020 -0800 Non-local boundary conditions (#1544) * Non-local boundary conditions * Rotate90 fills the lo-x and lo-y boundary regions by rotating the data around (x=0,y=0) by 90 degrees in either direction. It also fills the corner of lo-x and lo-y boundary region by rotating the data by 180 degrees. * Rotate180 fills the lo-x boundary by rotating the data around (x=0,y=L_y/2) by 180 degrees. * Fill the polar boundaries of the spherical coordinates (theta, phi, r). The lo-x boundary is filled with f(-x,y) = f(x,mod(y+pi,2*pi)), and the hi-x boundary is filled with f(pi+x,y) = f(pi-x,mod(y+pi,2*pi)). * GPU support for NonLocalBC Src/Base/AMReX_FBI.H Src/Base/AMReX_FabArrayBase.H Src/Base/AMReX_FabArrayBase.cpp Src/Base/AMReX_NonLocalBC.H Src/Base/AMReX_NonLocalBCImpl.H Src/Base/CMakeLists.txt Src/Base/Make.package commit b5e08d92541f24e3ee3cb8d46f5b49c0147538bd Author: Weiqun Zhang Date: Thu Nov 19 10:28:53 2020 -0800 Remove deprecated MFUtil::convert function. One could use amrex::cast for FabArray casting. (#1523) Src/Base/AMReX_MultiFabUtilI.H commit 907baaede901d52601f120d606224bccd04d9611 Author: Andrew Myers Date: Thu Nov 19 13:24:16 2020 -0500 If TheZeroVector() is passed in to SortParticlesByBin, do nothing instead of crashing. (#1564) Src/Particle/AMReX_ParticleContainerI.H Src/Particle/AMReX_Particles.H commit 6038a3102fc55db861d69b242eecc35119675b96 Author: Weiqun Zhang Date: Wed Nov 18 09:49:08 2020 -0800 Scan size type (#1521) Make the size type a template parameter to support the scan of more than INT_MAX elements. Src/Base/AMReX_Scan.H commit ff92f1c129ea149eead13dc264eabf15a705e74d Author: mic84 Date: Wed Nov 18 09:14:51 2020 -0800 CMake: AMReX_MPI options depends on SYCL, not HIP (#1555) Tools/CMake/AMReXOptions.cmake commit 7cf969f1b656f9ab17975c90952360fb400dd712 Author: Axel Huebl Date: Wed Nov 18 09:14:25 2020 -0800 CMake: Fix Single-Precision Switch (#1550) * CMake: Fix Single-Precision Switch Since the latest refactoring, single-recision builds were not activated. This fixes it. * AMReX_Config.H: add AMREX_SINGLE_PRECISION_PARTICLES Src/Particle/CMakeLists.txt Tools/CMake/AMReX_Config.H.in Tools/CMake/AMReX_Defines.cmake commit 14c3003f8f4dda9ef9db0c04ba613e378677358a Author: Weiqun Zhang Date: Wed Nov 18 09:03:38 2020 -0800 Refactor FabArray's communication functions (#1553) Remove duplication and allow for more flexibility for extension. Src/AmrCore/AMReX_TagBox.cpp Src/Base/AMReX_FBI.H Src/Base/AMReX_FabArray.H Src/Base/AMReX_FabArrayCommI.H Src/Base/AMReX_MultiFab.cpp Src/Base/AMReX_MultiFabUtil.cpp Src/Base/AMReX_iMultiFab.cpp Src/Boundary/AMReX_YAFluxRegister.cpp commit 079f599f74057ea93d8c60b82760b8747b21849e Author: Weiqun Zhang Date: Tue Nov 17 18:40:51 2020 -0800 DPCPP: Fortran (#1554) Switch from gfortran to ifx that comes with oneapi in GNU make. Tools/GNUMake/comps/dpcpp.mak commit e38742e322f6fbdf0e0fb090271d229fc2c834b9 Author: mic84 Date: Tue Nov 17 13:26:24 2020 -0800 CMake: improvements to handling of buildInfo (#1552) Tools/CMake/AMReXBuildInfo.cmake commit 477df17dd96607c385fc9410c4fe7746f121b58d Author: Weiqun Zhang Date: Tue Nov 17 11:19:19 2020 -0800 Fix BaseFab::maxabs (#1546) ## Summary For the host version, call host function LoopOnCpu instead of host device function Loop because the function we pass to it is a host function. ## Checklist The proposed changes: - [x] fix a bug or incorrect behavior in AMReX - [ ] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] are described in the proposed changes to the AMReX documentation, if appropriate Src/Base/AMReX_BaseFab.H commit cd6f4d0bdbc389f672e3d6a3b8d286729caca3b6 Author: Andrew Myers Date: Tue Nov 17 13:33:00 2020 -0500 Fix bug in the 'intarray' method of StructOfArrays (#1548) Thanks to @rporcu for finding this. The proposed changes: - [x] fix a bug or incorrect behavior in AMReX - [ ] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] are described in the proposed changes to the AMReX documentation, if appropriate Src/Particle/AMReX_StructOfArrays.H commit b329cfe0bc4036aae7b69dc94b7bb34f8ab040c9 Author: Axel Huebl Date: Mon Nov 16 21:07:35 2020 -0800 CMake: generate_buildinfo target (#1545) This commit generates a static library `buildInfo::` for the passed user target. This target is then automatically linked. Also, the generated `.cpp` file is now scoped properly, avoiding collisions. This solves the following corner cases: - workflows with OBJECT libraries, that cannot be passed as targets since they have no `PRE_BUILD` custom command support - workflows with multiple targets that are not linked together (those call `generate_buildinfo` multiple times) - workflows with multiple targets that are linked together (those call `generate_buildinfo` once and re-use the target) Tools/CMake/AMReXBuildInfo.cmake commit 5076d33c583d88e0fb78214470715c2d09c083cd Author: Shreyas Ananthan Date: Mon Nov 16 10:00:02 2020 -0700 fcompare: fix logic error when comparing both rel/abs tolerances (#1542) PR #1537 introduced the option to allow tolerance checks on both absolute and relative tolerances. However, the check used `and` instead of `or` to allow tests to pass when either absolute or relative error was below user-specified tolerance. ## Checklist The proposed changes: - [X] fix a bug or incorrect behavior in AMReX - [ ] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] are described in the proposed changes to the AMReX documentation, if appropriate Tools/Plotfile/fcompare.cpp commit 95b97799f2e611d19a4b685f725bfa3e5e12dc7f Author: Weiqun Zhang Date: Sun Nov 15 17:11:12 2020 -0800 nvcc --Werror cross-execution-space-call (#1540) * nvcc --Werror cross-execution-space-call * GNU make: Turn on the flag by default. Use `GPU_ERROR_CROSS_EXECUTION_SPACE_CALL=FALSE` to turn it off. * CMake: Add AMReX_ERROR_CROSS_EXECUTION_SPACE_CALL, which if off by default (just like AMReX_ERROR_CAPTURE_THIS). * update cmake documentation Docs/sphinx_documentation/source/GPU.rst Tools/CMake/AMReX_SetupCUDA.cmake Tools/GNUMake/Make.defs Tools/GNUMake/comps/nvcc.mak commit aef49304278572eb3c4a5db75462c8c213d35221 Author: Max Katz Date: Sun Nov 15 11:22:29 2020 -0800 Don't throw C++ std error when not config checking (#1543) Tools/GNUMake/comps/gnu.mak Tools/GNUMake/comps/nvcc.mak commit c80007df2f90e373f3d947fa7f621ab902378f3c Author: Shreyas Ananthan Date: Thu Nov 12 17:36:06 2020 -0700 fcompare: Add option for an absolute tolerance check (#1537) This PR adds a command line option to `fcompare` that takes in a user-specified tolerance for absolute error and adds logic to compare both absolute and relative errors against user-specified tolerances. Since `-a` is already used as a short-form for `--allow_diff_grids`, I have only used `--abs_tol` as the option without a short version when parsing command line options. Tools/Plotfile/fcompare.cpp commit cdb236ee435635f8007d2be97c0f6f1df13ecba0 Author: Weiqun Zhang Date: Thu Nov 12 08:50:35 2020 -0800 Remove call to cudaDeviceSetCacheConfig (#1536) ## Summary No longer set CUDA device cache configuration to prefer L1 cache. This does not appear to affect any kernels in a negative way. With more shared memory available, reduction functions are faster in some cases. ## Additional background ## Checklist The proposed changes: - [ ] fix a bug or incorrect behavior in AMReX - [ ] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] are described in the proposed changes to the AMReX documentation, if appropriate Src/Base/AMReX_GpuDevice.cpp commit f55a950e0977a4b062bba6ff9787ba84e9f7da92 Author: Weiqun Zhang Date: Thu Nov 12 08:11:02 2020 -0800 Fix multi-component nodal solver tutorial (#1535) ## Summary `MultiFab::setVal` should not be called inside `MFIter`. ## Additional background This was caught by #1530. ## Checklist The proposed changes: - [x] fix a bug or incorrect behavior in AMReX - [ ] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] are described in the proposed changes to the AMReX documentation, if appropriate Tutorials/LinearSolvers/MultiComponent/MCNodalLinOp.cpp commit a01697fda4a2f38a1e7e3c849bc52baca2c2999d Author: Axel Huebl Date: Tue Nov 10 18:18:03 2020 -0800 Ascent: Check Bounds Particle Reals (#1515) Access of particle variable names with a bound-check, so users definitely pass the right amount of names. Src/Extern/Conduit/AMReX_Conduit_Blueprint_ParticlesI.H commit d9f60b4cb934852cb87653fbc84df22c98cfe0e0 Author: Andrew Myers Date: Tue Nov 10 09:02:30 2020 -0800 Allow for particle containers constructed with the same AmrCore... (#1532) Src/AmrCore/AMReX_AmrParGDB.H Src/Particle/AMReX_ParGDB.H Src/Particle/AMReX_Particles.H commit 10c2a5e5949213d607305f771edab433285166d7 Author: asalmgren Date: Mon Nov 9 17:58:38 2020 -0800 Update the docs to include hypre.adjust_singular_matrix (#1534) ## Summary adds documentation of hypre.adjust_singular_matrix ## Additional background ## Checklist The proposed changes: - [ ] fix a bug or incorrect behavior in AMReX - [ ] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [X] are described in the proposed changes to the AMReX documentation, if appropriate Docs/sphinx_documentation/source/LinearSolvers.rst commit 59b8bf1cc1ebf6dac76602a24a4ce696080b070c Author: Weiqun Zhang Date: Mon Nov 9 15:59:56 2020 -0800 DPC++: fix scan (#1533) * Use atomics instead of volatile to read status written to global memory by other blocks * Workaround `Random()` bug in the scan test * Limit the memory usage in the scan test Src/Base/AMReX_Scan.H Tutorials/GPU/ParallelScan/main.cpp commit 15ab2368225b81777e2810e0e02b1293364bb575 Author: Weiqun Zhang Date: Mon Nov 9 14:02:21 2020 -0800 Fix #1530 (#1531) `FillPatchIterator` is derived from `MFIter`. We need to reset `MFIter::depth` so that the ctor of `FillPatchIterator` can start `MFIter`. Src/Amr/AMReX_AmrLevel.cpp commit b94e1d2fca2bffd2b0e826eebfca7d63290efd40 Author: Weiqun Zhang Date: Mon Nov 9 10:30:06 2020 -0800 Assert no nested MFIters (#1530) ## Summary Add assertion to catch nested MFIters (e.g., MultiFab functions are called inside MFIter). ## Checklist The proposed changes: - [ ] fix a bug or incorrect behavior in AMReX - [x] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] are described in the proposed changes to the AMReX documentation, if appropriate Src/Base/AMReX_MFIter.H Src/Base/AMReX_MFIter.cpp commit ad7c4eac3475fb7ddfc2abe8c74400d0274f8398 Author: Weiqun Zhang Date: Mon Nov 9 10:29:06 2020 -0800 Add anonymous namespace back to parameters in AMReX_Amr.cpp (#1529) Src/Amr/AMReX_Amr.H Src/Amr/AMReX_Amr.cpp commit eb29a4f6b40e6458323db681795c1a12e1403c1c Author: Weiqun Zhang Date: Mon Nov 9 07:51:46 2020 -0800 Free MPI_Datatype defined by AMReX (#1527) ## Summary This fixes a one-time memory leak in MPI_Datatypes defined by AMReX as reported in #1525. ## Checklist The proposed changes: - [x] fix a bug or incorrect behavior in AMReX - [ ] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] are described in the proposed changes to the AMReX documentation, if appropriate Src/Base/AMReX_ParallelDescriptor.cpp commit 0fd351b118b695d9c439c33a0f1ff9c6f3f00aeb Author: Weiqun Zhang Date: Sun Nov 8 10:27:36 2020 -0800 Update OLCF makefile (#1528) ## Summary * Remove summitdev and peak. * Environment variable OLCF_MODULEPATH_ROOT is used in addition to host name to detect OLCF machines. ## Additional background ## Checklist The proposed changes: - [ ] fix a bug or incorrect behavior in AMReX - [ ] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] are described in the proposed changes to the AMReX documentation, if appropriate Tools/GNUMake/Make.machines Tools/GNUMake/sites/Make.olcf commit 6cc5104e57548e575717051b04361ffed0fee9c2 Author: Weiqun Zhang Date: Sun Nov 8 10:26:56 2020 -0800 Remove getGridSize that is no longer used. (#1524) Src/Base/AMReX_GpuLaunch.H Src/Base/AMReX_GpuLaunch.cpp Src/Base/CMakeLists.txt Src/Base/Make.package commit 86abbe0221c1c3c53e69d740cdc24190ce1376c5 Author: Weiqun Zhang Date: Sat Nov 7 14:57:59 2020 -0800 Support CUDA arch < 6.0 (#1512) ## Summary * Implement atomicAdd for CUDA arch < 6.0. * Remove the CUDA arch check in CMake. ## Checklist The proposed changes: - [ ] fix a bug or incorrect behavior in AMReX - [x] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] are described in the proposed changes to the AMReX documentation, if appropriate Src/Base/AMReX_BaseFab.H Src/Base/AMReX_GpuAtomic.H Src/Base/AMReX_GpuDevice.H Src/Base/AMReX_GpuDevice.cpp Tools/CMake/AMReX_SetupCUDA.cmake commit ec4424eb66366c5b798639a100b30c6c01938f79 Author: Axel Huebl Date: Sat Nov 7 14:56:34 2020 -0800 Change Spelling: Env AMREX_CUDA_ARCH (#1522) ## Summary Make the _environment variable_ that sets a default CUDA architecture all-caps, as this is way more common in Unix. Typical Values: `7.0` or `Volta` (i.e. for V100) ## Additional background ## Checklist The proposed changes: - [ ] fix a bug or incorrect behavior in AMReX - [ ] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] are described in the proposed changes to the AMReX documentation, if appropriate Docs/sphinx_documentation/source/GPU.rst Tools/CMake/AMReX_SetupCUDA.cmake commit 1b00f34ef1e170d4e2b398045e9dd45b4ba8ec34 Author: Weiqun Zhang Date: Fri Nov 6 18:18:27 2020 -0800 make clean & make cleanconfig (#1520) ## Summary `make clean` now does what `make realclean` does. `make realclean` is kept. `make cleanconfig` is introduced to do what `make clean` used to do. For most users, `make clean` is the one that should be used. Also add some tests in makefiles to avoid error messages in case when `make clean` is run on a makefile with the default not suitable for the system. ## Checklist The proposed changes: - [ ] fix a bug or incorrect behavior in AMReX - [ ] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] are described in the proposed changes to the AMReX documentation, if appropriate Tools/GNUMake/Make.defs Tools/GNUMake/Make.rules Tools/GNUMake/README.md Tools/GNUMake/comps/hip.mak Tools/GNUMake/comps/nvcc.mak Tools/GNUMake/sites/Make.unknown commit d9ac9a6c844a5055cdd2b332d101096da3ae5d72 Author: mic84 Date: Fri Nov 6 13:47:58 2020 -0800 CMake: fix options name in Test directory (#1514) Tests/CMakeLists.txt Tests/HDF5Benchmark/CMakeLists.txt commit 6dc26f704a386a03120f71395f00186049499264 Author: mic84 Date: Fri Nov 6 13:47:15 2020 -0800 CMake: add component check for Fortran (#1511) Docs/sphinx_documentation/source/BuildingAMReX.rst Tools/CMake/AMReXConfig.cmake.in commit 9f2822bdcdd5604eea85a6206413684fb0137ea4 Author: mic84 Date: Fri Nov 6 13:44:41 2020 -0800 CMake: fix some quirks with options output (#1516) Tools/CMake/AMReXOptions.cmake commit 79d991ca8518943155e312a9a7566380e563bc93 Author: Shreyas Ananthan Date: Fri Nov 6 12:41:36 2020 -0700 hypre: Adjust singular matrices before hypre solves (#1519) Src/Extern/HYPRE/AMReX_Hypre.H Src/Extern/HYPRE/AMReX_HypreABecLap3.cpp Src/Extern/HYPRE/AMReX_HypreIJIface.H Src/Extern/HYPRE/AMReX_HypreIJIface.cpp Src/Extern/HYPRE/AMReX_HypreNodeLap.cpp Src/LinearSolvers/MLMG/AMReX_MLCellABecLap.cpp commit fe6058b946fb4334b74c42fedde919836f82066d Author: Shreyas Ananthan Date: Fri Nov 6 12:04:55 2020 -0700 hypre: Update nodal Laplacian interface to accept custom options (#1518) HypreNodeLap class creates the hypre IJ instance in its constructor. Therefore, the custom option namespace specified by user must be part of the constructor arguments. Src/Extern/HYPRE/AMReX_HypreNodeLap.H Src/Extern/HYPRE/AMReX_HypreNodeLap.cpp Src/LinearSolvers/MLMG/AMReX_MLLinOp.H Src/LinearSolvers/MLMG/AMReX_MLMG.cpp Src/LinearSolvers/MLMG/AMReX_MLNodeLinOp.H Src/LinearSolvers/MLMG/AMReX_MLNodeLinOp.cpp commit 7b7c1c8e6addc152fb9c0059577799bfe8dc6d93 Author: Weiqun Zhang Date: Thu Nov 5 16:27:28 2020 -0800 DPC++: sync after htod_memcpy on null stream (#1517) ## Summary DPC++ does not have the concept of null stream. Adding a sync after htod_memcpy on the "null" stream will help eliminate potential bugs (e.g., htod_memcpy is called before MFIter that uses non-null stream). ## Additional background ## Checklist The proposed changes: - [ ] fix a bug or incorrect behavior in AMReX - [ ] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] are described in the proposed changes to the AMReX documentation, if appropriate Src/Base/AMReX_GpuDevice.H Src/Base/AMReX_Reduce.H commit 8035b01d86a7f8f4542bec6aa167dc864c8b3209 Author: Shreyas Ananthan Date: Thu Nov 5 15:06:36 2020 -0700 CMake: Fix ENABLE_HYPRE to AMReX_HYPRE (#1513) Fix the variable being checked to enable HYPRE. ## Checklist The proposed changes: - [X] fix a bug or incorrect behavior in AMReX - [ ] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] are described in the proposed changes to the AMReX documentation, if appropriate Tools/CMake/AMReXThirdPartyLibraries.cmake commit 9aa75b1844e4ecc6c860a2d5d37b57bccedf73ed Author: Max Katz Date: Thu Nov 5 08:59:20 2020 -0800 Fix some build issues at LLNL (#1510) Tools/GNUMake/sites/Make.llnl commit 7ca532e5b299df1d7e9137e61f9271e4341c6189 Author: Andrew Myers Date: Thu Nov 5 08:54:57 2020 -0800 Add timer to neighbor list construction (#1509) Src/Particle/AMReX_NeighborList.H commit 86c6b3d00ae8d8e232b345b7060d9188fc838136 Author: mic84 Date: Wed Nov 4 19:39:39 2020 -0800 CMake: overhaul options (#1490) * Doc: move ENABLE_CUDA_FASTMATH to the GPU options section * CMake: change all options to include AMReX_ namespace * CMake: remove ENABLE_ACC option * CMake: make AMReX_SPACEDIM a multi-valued string * Doc: add option AMReX_ENABLE_TESTS * CMake: rename AMReX_Options.cmake to AMReXOptions.cmake * CMake: refactor GPU-related options * CMake: MPI must be turned off if DPCPP is enabled * CMake: set via precision via selection of value * CMake: forgot to update Particles/CMakeLists.txt * CMake: append prefix to TL_PROFILE * Update Docs/sphinx_documentation/source/AMReX_Profiling_Tools.rst Co-authored-by: Axel Huebl * Update Docs/sphinx_documentation/source/AMReX_Profiling_Tools.rst Co-authored-by: Axel Huebl * Update Docs/sphinx_documentation/source/BuildingAMReX.rst Co-authored-by: Axel Huebl * CMake: components in Config file shall not have namespace prefix * Revert "CMake: components in Config file shall not have namespace prefix" This reverts commit 924cf15f2a926467f3924a76f80595c0d1e53a7a. * Update Tools/CMake/AMReX_SetupCUDA.cmake Co-authored-by: Axel Huebl * CMake: update Doc * Update Tools/CMake/AMReX_SetupCUDA.cmake Co-authored-by: Axel Huebl Co-authored-by: Axel Huebl .github/workflows/linux.yml .github/workflows/macos.yml .github/workflows/windows.yml CHANGES CMakeLists.txt Docs/sphinx_documentation/source/AMReX_Profiling_Tools.rst Docs/sphinx_documentation/source/Basics.rst Docs/sphinx_documentation/source/BuildingAMReX.rst Docs/sphinx_documentation/source/GPU.rst Docs/sphinx_documentation/source/Visualization.rst Src/Amr/CMakeLists.txt Src/AmrCore/CMakeLists.txt Src/Base/CMakeLists.txt Src/Boundary/CMakeLists.txt Src/CMakeLists.txt Src/EB/CMakeLists.txt Src/Extern/HYPRE/CMakeLists.txt Src/Extern/PETSc/CMakeLists.txt Src/Extern/ProfParser/CMakeLists.txt Src/Extern/amrdata/CMakeLists.txt Src/F_Interfaces/CMakeLists.txt Src/LinearSolvers/CMakeLists.txt Src/Particle/CMakeLists.txt Tests/CMakeLists.txt Tools/CMake/AMReXConfig.cmake.in Tools/CMake/AMReXInstallHelpers.cmake Tools/CMake/AMReXOptions.cmake Tools/CMake/AMReXParallelBackends.cmake Tools/CMake/AMReXSYCL.cmake Tools/CMake/AMReXThirdPartyLibraries.cmake Tools/CMake/AMReX_Config.cmake Tools/CMake/AMReX_Defines.cmake Tools/CMake/AMReX_Options.cmake Tools/CMake/AMReX_SetupCUDA.cmake Tools/Plotfile/CMakeLists.txt Tutorials/Amr/Advection_AmrCore/CMakeLists.txt Tutorials/Amr/Advection_AmrLevel/CMakeLists.txt Tutorials/Basic/HeatEquation_EX1_C/CMakeLists.txt Tutorials/Basic/HeatEquation_EX1_CF/CMakeLists.txt Tutorials/Basic/HeatEquation_EX1_F/CMakeLists.txt Tutorials/Basic/HeatEquation_EX2_CF/CMakeLists.txt Tutorials/Basic/HeatEquation_EX3_C/CMakeLists.txt Tutorials/Basic/HelloWorld_F/CMakeLists.txt Tutorials/Basic/main_C/CMakeLists.txt Tutorials/Basic/main_F/CMakeLists.txt Tutorials/Blueprint/AssignMultiLevelDensity/CMakeLists.txt Tutorials/Blueprint/CellSortedParticles/CMakeLists.txt Tutorials/Blueprint/HeatEquation_EX1_C/CMakeLists.txt Tutorials/CMakeLists.txt Tutorials/EB/CNS/CMakeLists.txt Tutorials/EB/GeometryGeneration/CMakeLists.txt Tutorials/ForkJoin/MLMG/CMakeLists.txt Tutorials/ForkJoin/Simple/MyTest.cpp Tutorials/FortranInterface/Advection_F/CMakeLists.txt Tutorials/FortranInterface/Advection_octree_F/CMakeLists.txt Tutorials/FortranInterface/Advection_octree_F2/CMakeLists.txt Tutorials/GPU/CNS/CMakeLists.txt Tutorials/GPU/EBCNS/CMakeLists.txt Tutorials/GPU/Launch/CMakeLists.txt Tutorials/LinearSolvers/ABecLaplacian_C/CMakeLists.txt Tutorials/LinearSolvers/ABecLaplacian_F/CMakeLists.txt Tutorials/LinearSolvers/MAC_Projection_EB/CMakeLists.txt Tutorials/LinearSolvers/Nodal_Projection_EB/CMakeLists.txt Tutorials/Particles/CellSortedParticles/CMakeLists.txt Tutorials/Particles/ElectromagneticPIC/CMakeLists.txt Tutorials/Particles/ElectrostaticPIC/CMakeLists.txt Tutorials/Particles/NeighborList/CMakeLists.txt Tutorials/SUNDIALS/EX1_C/CMakeLists.txt commit 22e6f93f2f66d24e364f35226b3f0d990182e316 Author: Weiqun Zhang Date: Wed Nov 4 17:00:42 2020 -0800 New Reduce::Sum function taking lambda function (#1508) ## Checklist The proposed changes: - [ ] fix a bug or incorrect behavior in AMReX - [x] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] are described in the proposed changes to the AMReX documentation, if appropriate Src/Base/AMReX_Reduce.H commit ae9d0c696dad43b4477517728bf0f2ef978165db Author: Weiqun Zhang Date: Wed Nov 4 14:50:18 2020 -0800 GNU Make: AMREX_CUDA_ARCH (#1507) ## Summary Option to use AMREX_CUDA_ARCH in addition to CUDA_ARCH to set CUDA arch. If AMREX_CUDA is set, it will be used over CUDA_ARCH. ## Additional background The CMake system is also planning to support AMREX_CUDA_ARCH. ## Checklist The proposed changes: - [ ] fix a bug or incorrect behavior in AMReX - [x] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [x] are described in the proposed changes to the AMReX documentation, if appropriate Docs/sphinx_documentation/source/BuildingAMReX.rst Tools/GNUMake/Make.defs commit 1ab6d82e749b039b0417fa6c539c04d464ba95ac Author: Andrew Myers Date: Wed Nov 4 10:54:39 2020 -0800 Fix bug in new assertion (#1506) Src/Particle/AMReX_ParticleContainerI.H commit db41d3d75449b162ebe4113656da49536149e165 Author: Max Katz Date: Wed Nov 4 09:23:41 2020 -0800 Add option for device sync at beginning and end of TinyProfiler region (#1505) ## Summary Now when setting `tiny_profiler.device_synchronize_on_region_end = 1` in the inputs file, we will synchronize before calling nvtxRangePop() and nvtxRangePush(), which means that TINY_PROFILE regions will include the full kernel time, rather than just the kernel launch time. ## Additional background ## Checklist The proposed changes: - [ ] fix a bug or incorrect behavior in AMReX - [x] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] are described in the proposed changes to the AMReX documentation, if appropriate Src/Base/AMReX_TinyProfiler.H Src/Base/AMReX_TinyProfiler.cpp commit 81fdb9a6254fee22daf3c32843c619e08d527bbc Author: Max Katz Date: Wed Nov 4 07:50:41 2020 -0800 Guard MFIter reduction macros against USE_GPU_PRAGMA (#1504) Src/Base/AMReX_BLFort.H commit 461ca5b42dfeb3ac24ab0aa344d3fc00129cc8b4 Author: Weiqun Zhang Date: Tue Nov 3 15:39:57 2020 -0800 GNU make: adjust multiple target rules (#1500) ## Summary Remove `/` from the object file name so that it works for multiple targets in different directories. This should not affect its current usage. ## Checklist The proposed changes: - [ ] fix a bug or incorrect behavior in AMReX - [ ] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] are described in the proposed changes to the AMReX documentation, if appropriate Tools/GNUMake/Make.rules commit 3ed4b614a91e8b49d3e6f9b434493cf8f1c9464a Author: mic84 Date: Tue Nov 3 14:09:34 2020 -0800 CMake; bump up hypre minimi required version (#1503) Tools/CMake/AMReXThirdPartyLibraries.cmake commit f4c041f181dd1bb5743483e4b86f19cbb6e40fb2 Author: Andrew Myers Date: Tue Nov 3 13:54:38 2020 -0800 assert if aos.size() != soa.size() (#1502) Src/Particle/AMReX_ParticleContainerI.H commit b5a506997ce5db41bc46d6d64a238053fa2ea1ea Author: Weiqun Zhang Date: Tue Nov 3 13:53:23 2020 -0800 Fix warnings in AMReX_Machine.cpp (#1499) Src/Base/AMReX_GpuDevice.cpp Src/Base/AMReX_Machine.H Src/Base/AMReX_Machine.cpp Src/LinearSolvers/MLMG/AMReX_MLLinOp.cpp commit e61266427dc11ffdfd2bad6577ac204e19ad5e88 Author: Andrew Myers Date: Tue Nov 3 13:52:40 2020 -0800 Use Long type for pindex to avoid potential overflow. (#1501) Src/Particle/AMReX_ParticleContainerI.H commit 7117a8f9006ad2d647c3e6412aee3ec3047d6def Author: Weiqun Zhang Date: Tue Nov 3 09:01:25 2020 -0800 Update GNU make for Cori GPU node (#1497) ## Summary Set CUDA_ARCH to 80 for A100 on Cori. ## Checklist The proposed changes: - [ ] fix a bug or incorrect behavior in AMReX - [ ] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] are described in the proposed changes to the AMReX documentation, if appropriate Tools/GNUMake/Make.machines Tools/GNUMake/sites/Make.nersc commit 24cacefc388e9811f52c1ebbceeb06cca90ba9b6 Author: Axel Huebl Date: Tue Nov 3 08:07:50 2020 -0800 Lin. Solvers (MLMG): Include Order (#1496) ## Summary Include own headers before stdlib headers to catch missing includes early. This avoid problems with less common compilers and environments. Carved out of #1485 ## Additional background ## Checklist The proposed changes: - [ ] fix a bug or incorrect behavior in AMReX - [ ] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] are described in the proposed changes to the AMReX documentation, if appropriate Src/LinearSolvers/MLMG/AMReX_MLCGSolver.H Src/LinearSolvers/MLMG/AMReX_MLCGSolver.cpp Src/LinearSolvers/MLMG/AMReX_MLLinOp.cpp Src/LinearSolvers/MLMG/AMReX_MLNodeLaplacian.cpp commit ee3076a8306269bf6d18ffee3f4d4944c660241d Author: Max Katz Date: Tue Nov 3 08:06:58 2020 -0800 GNUMake: Merge Ascent and Peak with Summit (#1493) ## Summary These are all systems with the same architecture so we can share the build setup. ## Checklist The proposed changes: - [ ] fix a bug or incorrect behavior in AMReX - [ ] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] are described in the proposed changes to the AMReX documentation, if appropriate Tools/GNUMake/sites/Make.olcf commit 43f112fe4b9deda918f3d3961d2979e124d0645d Author: Weiqun Zhang Date: Mon Nov 2 13:00:42 2020 -0800 Update CHANGES for AMReX 20.11 (#1494) CHANGES commit db23e834e9cdc24884b689e0bdbd285ba6124562 Author: Shreyas Ananthan Date: Mon Nov 2 11:00:19 2020 -0700 HYPRE: Fix issues with hypre IJ interface (#1495) This commit fixes two issues with the hypre IJ interface - Fixes a memory leak in HypreIJIface as it was calling parse_inputs in constructor - Sets default number of iterations and tolerance for BoomerAMG when used as a preconditioner Src/Extern/HYPRE/AMReX_HypreIJIface.cpp commit f29a0c9d1b8e6356f8007a0bcb9cfc1000f2a0ac Author: Weiqun Zhang Date: Fri Oct 30 15:45:23 2020 -0700 Change default hypre interface to ij for non-EB (#1492) Src/LinearSolvers/MLMG/AMReX_MLMG.H commit 93ffe6391b8a18c749610cabe05c785b6325378d Author: Weiqun Zhang Date: Fri Oct 30 10:39:39 2020 -0700 Fix DEBUG build for DPC++ (#1491) This fixes a bug introduced in #1489. Src/Base/AMReX.cpp commit 82c7a83ff6dd43be71d5960ad422f4db88d929e7 Author: Weiqun Zhang Date: Fri Oct 30 08:45:24 2020 -0700 Changes to make hipcc --save-temps work. (#1489) ## Summary It does not like AMREX_GPU_DEVICE_EXTERNAL. Also changed is the device version of `Error`, `Abort` and `Warning`. If `NDEBUG` is defined, they do not do anything. ## Checklist The proposed changes: - [ ] fix a bug or incorrect behavior in AMReX - [x] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] are described in the proposed changes to the AMReX documentation, if appropriate Src/Base/AMReX.H Src/Base/AMReX.cpp Tools/GNUMake/Make.defs commit 69c496f8e8e65f7fe4e9cd5afd4e9e89fa4ad9ba Author: Weiqun Zhang Date: Thu Oct 29 12:32:24 2020 -0700 Fix GPU race condition for EB (#1487) A GPU race condition was introduced in #1451 trying to fix faces and cells next to covered cells that were converted from small cut cells. It is fixed by launching a separate kernel for each direction. Src/EB/AMReX_EB2_2D_C.cpp Src/EB/AMReX_EB2_3D_C.cpp commit 147149885e7ca02dcfdc88f1f4953bf26dc10dc8 Author: Axel Huebl Date: Thu Oct 29 11:30:41 2020 -0700 Ascent: Use Default Name for Mesh Ghost (#1488) ## Summary Use the default name for ghost cells in Ascent. The current name is non-default and causes issues at domain-boundaries during volume rendering (unless one overwrites the `ghost_field_name` keyword in the options). Thanks to @mclarsen and @cyrush for debugging and hinting this with me. ## Additional background Seen with WarpX. Pre-PR: ![replay_000400(9)](https://user-images.githubusercontent.com/1353258/97613195-9eed1c80-19d5-11eb-9934-cafeffc51458.png) Post-PR: ![replay_000400(10)](https://user-images.githubusercontent.com/1353258/97614230-eb852780-19d6-11eb-8dcd-4ca4d86d2e21.png) ## Checklist The proposed changes: - [x] fix a bug or incorrect behavior in AMReX - [ ] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] are described in the proposed changes to the AMReX documentation, if appropriate Src/Extern/Conduit/AMReX_Conduit_Blueprint.cpp commit 32052a90a11da0dec54c353e0d6e29758e52b9a1 Author: Axel Huebl Date: Wed Oct 28 16:08:03 2020 -0700 CI: Check More Warnings (#1486) ## Summary Checks for more warnings that are likely to uncover nifty bugs: `-Wshadow -Woverloaded-virtual -Wunreachable-code` Also checks for superfluous `;` with `-Wextra-semi` (only a flag in Clang). ## Additional background Does not yet add `-Wall -Wextra -Wpedantic` because there are quite a few places where we need to add casts to avoid signed-unsigned-comparisons, etc. WarpX' CI covers currently: * GNU: `-Wall -Wextra -Wpedantic -Wshadow -Woverloaded-virtual -Wunreachable-code` * Clang: `-Wall -Wextra -Wpedantic -Wshadow -Woverloaded-virtual -Wextra-semi -Wunreachable-code` Even with super-builds, we current include AMReX with `-isystem` and build AMReX with default compiler flags; this enables us to suppress most AMReX build warnings besides the ones that drop-through from macros defined in headers, which atm. are all fixed for the code sections we use. ## Checklist The proposed changes: - [ ] fix a bug or incorrect behavior in AMReX - [ ] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] are described in the proposed changes to the AMReX documentation, if appropriate .github/workflows/linux.yml .github/workflows/macos.yml Src/Particle/AMReX_NeighborParticles.H Tutorials/Basic/PrefixSum_MultiFab/main.cpp Tutorials/Particles/CellSortedParticles/cell_sorted_F.H Tutorials/Particles/ElectromagneticPIC/Exec/OpenMP/em_pic_F.H Tutorials/Particles/ElectrostaticPIC/ElectrostaticParticleContainer.cpp Tutorials/Particles/NeighborList/MDParticleContainer.cpp commit da1063cc5c16dcf519e74d630f898e7b23711c76 Author: Weiqun Zhang Date: Wed Oct 28 14:03:39 2020 -0700 HIP GNU Make: need to override COMP for regression tests (#1467) Tools/GNUMake/Make.defs commit ee020a50235341408297ff7018c536486eb30754 Author: Axel Huebl Date: Wed Oct 28 13:38:34 2020 -0700 CI: -Werror for host compilers (#1484) ## Summary Add more warning coverage in CI for CMake builds. ## Additional background ## Checklist The proposed changes: - [ ] fix a bug or incorrect behavior in AMReX - [ ] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] are described in the proposed changes to the AMReX documentation, if appropriate .github/workflows/linux.yml .github/workflows/macos.yml commit 0186e2f103f800280d8edfc5dffcd9280f132f42 Author: Weiqun Zhang Date: Mon Oct 26 11:18:29 2020 -0700 Fix warning in FabSet for single precision (#1481) * Fix warning in FabSet for single precision * Constify constants Co-authored-by: Axel Huebl Src/Boundary/AMReX_FabSet.cpp commit 6f07fe4e84a4dd57a0e89c2d2cb84c9953fda85d Author: Weiqun Zhang Date: Fri Oct 23 16:23:17 2020 -0700 Workaround for Mac CI (#1482) .github/workflows/dependencies/dependencies_mac.sh commit 61ef93db305fccfe81404686d6878f1f805fee02 Author: Axel Huebl Date: Fri Oct 23 13:29:11 2020 -0700 CMake 3.18+: CUDA Arch Policy (OLD) (#1480) Keep the old policy for now to avoid setting the code generation flags twice. We should be able to transition this well, but I need to first find out how device LTO generation flags are handled here, if at all, and how to restore the default-detection of the local GPU architecture, e.g. on Summits head-nodes. CMakeLists.txt commit f4d73901d209caa80951b4842f0e46848104a98f Author: asalmgren Date: Fri Oct 23 10:13:49 2020 -0700 More fixes to documenting hypre (previous text got lost). (#1477) Docs/sphinx_documentation/source/LinearSolvers.rst commit f42ab5658751c51d6792d7dd7f1999c4c3310143 Author: Luca Fedeli Date: Thu Oct 22 17:23:11 2020 +0200 Add fill method to GpuArray (#1472) ## Summary I would like to propose to add a `fill` method to `GpuArray`. This PR implements just that. ## Additional background I am wondering if it would be better to replace `i < N` with `i < amrex::max(N,std::size_t{1})`, in order to deal with the case `N=0`. ## Checklist The proposed changes: - [ ] fix a bug or incorrect behavior in AMReX - [X] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] are described in the proposed changes to the AMReX documentation, if appropriate Src/Base/AMReX_Array.H commit 0cfd7d349279040c1d9b9283f970b333296594d9 Author: asalmgren Date: Thu Oct 22 08:21:53 2020 -0700 Update the docs to reflect the new hypre options (#1473) Docs/sphinx_documentation/source/LinearSolvers.rst commit 6071cf447613beba8753fb0a5409dbe4e299111a Author: Weiqun Zhang Date: Tue Oct 20 18:10:01 2020 -0700 Gpu::Atomic::AddNoRet (#1469) Add Gpu::Atomic::AddNoRet to use HIP's atomicAddNoRet for float, which is much faster than atomicAdd for float that is currently implemented with CAS. Src/Base/AMReX_FBI.H Src/Base/AMReX_GpuAtomic.H Src/Base/AMReX_GpuReduce.H Src/Base/AMReX_MultiFab.cpp Src/Base/AMReX_TypeTraits.H Src/EB/AMReX_EB_utils.cpp Src/Particle/AMReX_DenseBins.H Src/Particle/AMReX_Particle_mod_K.H Tests/Particles/NeighborParticles/MDParticleContainer.cpp Tests/Particles/ParticleMesh/main.cpp Tutorials/Particles/ElectromagneticPIC/Exec/CUDA/em_pic_K.H commit a004707203dcbd3e28c5e637c623d569301c64cd Author: Andy Nonaka Date: Tue Oct 20 18:01:29 2020 -0700 Enforce singular solvable (#1471) ## Summary Option to make the code NOT enforce solvability for singular problems. This is accomplished by calling linop.setEnforceSingularSolvable(false) The default behavior is true, so no results will change in existing codes ## Additional background ## Checklist The proposed changes: - [x] add new capabilities to AMReX Src/LinearSolvers/MLMG/AMReX_MLLinOp.H Src/LinearSolvers/MLMG/AMReX_MLMG.cpp commit 8350a1641044aa222d922e85bd082be5734e2a6a Author: mic84 Date: Tue Oct 20 17:15:07 2020 -0700 CMake: improve HDF5 support (#1468) * CMake: look only for the C component of HDF5 * CMake: use only C-related variables from FindHDF5 * CMake: add Tests/HDF5Benchmark to CTest * CMake: fix typo * Update Tools/CMake/AMReXThirdPartyLibraries.cmake * CMake: use hdf5 imported target if available Co-authored-by: Axel Huebl Tests/CMakeLists.txt Tests/HDF5Benchmark/CMakeLists.txt Tools/CMake/AMReXThirdPartyLibraries.cmake commit c0710a2bc4725aa8e7cab57690b27c73285436ae Author: Shreyas Ananthan Date: Sun Oct 18 14:16:56 2020 -0600 Use rocrand from HIP APT repository (#1466) .github/workflows/dependencies/dependencies_hip.sh commit 4f0052023f6c68ca2898ed99552e641a2e4dcc8f Author: Shreyas Ananthan Date: Sun Oct 18 14:16:03 2020 -0600 Address warnings in CUDA builds (#1465) Src/Base/AMReX.cpp Src/Base/AMReX_FBI.H Tools/Plotfile/fsnapshot.cpp commit aef02b9a4745005856a45417632b964a760dadb0 Author: mic84 Date: Sun Oct 18 11:03:49 2020 -0700 CMake: FindPETSc must not overwrite CMAKE_Fortran_FLAGS (#1464) ## Summary FindPETSc used to set ```CMAKE_Fortran_FLAGS``` to the list of PETSc include directories. This caused problems with the compilation of Fortran files (see Issue #1463 ). This PR fixes this. ## Additional background ## Checklist The proposed changes: - [X] fix a bug or incorrect behavior in AMReX - [ ] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] are described in the proposed changes to the AMReX documentation, if appropriate Tools/CMake/FindPETSc.cmake commit f64fb79d0f8a04a785e7c21f49dd81330ce55692 Author: Axel Huebl Date: Fri Oct 16 18:56:55 2020 -0700 CI: rocRAND v1.8.2 (#1462) Building the development branch is a bit unstable. .github/workflows/dependencies/dependencies_hip.sh commit 0cea78bf2f4f24568a1d71d20e8b2614fe19012c Author: mic84 Date: Fri Oct 16 17:33:42 2020 -0700 CMake: some cleanup (#1460) * CMake: provide function to retrieve AMReX version * CMake: remove unused helper functions CMakeLists.txt Tools/CMake/AMReX_Utils.cmake commit 931cf5993b8ad2d5fda1a853ebabb36dbe2ae5c2 Author: Weiqun Zhang Date: Fri Oct 16 17:30:50 2020 -0700 DPCPP: Scan (#1461) ## Summary Need to use global address space when doing atomic inc on virtual block id. ## Checklist The proposed changes: - [x] fix a bug or incorrect behavior in AMReX - [ ] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] are described in the proposed changes to the AMReX documentation, if appropriate Src/Base/AMReX_Scan.H commit 31ca23418f488fe0b309ef3fc3e99c522ced4921 Author: mic84 Date: Fri Oct 16 14:53:58 2020 -0700 [WIP] CMake: add support for dpc++ beta10 (#1459) * CMake: retrieve dpcpp version * CMake: modify dpc++ flags to account for beta10 version Tools/CMake/AMReXSYCL.cmake commit d036a6df1c6f8f7bb458c7e5253a91f4b6f96b83 Author: Max Katz Date: Fri Oct 16 14:21:28 2020 -0700 Fix potential out of bounds access in #1441 (#1458) ## Summary The user may have given us a vector that has too many values, so we just ignore the remainder. ## Additional background ## Checklist The proposed changes: - [x] fix a bug or incorrect behavior in AMReX - [ ] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] are described in the proposed changes to the AMReX documentation, if appropriate Src/AmrCore/AMReX_ErrorList.H commit 18b293dad9199deb329efecddd56ccce709371f5 Author: Weiqun Zhang Date: Fri Oct 16 10:12:19 2020 -0700 DPCPP beta10 (#1456) ## Summary * Update for DPCPP beta10 * Fix some warnings * Remove some device versions of Random() for HIP and DPC++ ## Checklist The proposed changes: - [ ] fix a bug or incorrect behavior in AMReX - [ ] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] are described in the proposed changes to the AMReX documentation, if appropriate Src/Base/AMReX_Extension.H Src/Base/AMReX_FBI.H Src/Base/AMReX_GpuPrint.H Src/Base/AMReX_GpuReduce.H Src/Base/AMReX_Random.H Src/Base/AMReX_Random.cpp Src/Base/AMReX_RandomEngine.H Src/Base/AMReX_Scan.H Src/Particle/AMReX_BinIterator.H Src/Particle/AMReX_ParticleCommunication.H Src/Particle/AMReX_ParticleContainerI.H Tools/GNUMake/comps/dpcpp.mak commit 15643f4b0ab28dc6b3277c5c44c18ac1f80e1345 Author: mic84 Date: Thu Oct 15 16:21:13 2020 -0700 CMake: prevent in-source builds (#1453) * CMake: prevent in-source builds * CMake: make error message more detailed CMakeLists.txt commit c53b1bbe88bed6b731bae4e11fd9ed0883a6b814 Author: Andy Nonaka Date: Thu Oct 15 14:29:41 2020 -0700 change char * to char const* so SWFFT code compiles with USE_CUDA=TRUE (#1454) Src/Extern/SWFFT/distribution.c commit 48b0a20210b9825c1c403e1e0b8a741ba913dffa Author: Shreyas Ananthan Date: Wed Oct 14 15:00:12 2020 -0600 Make HypreABecLap3::getSolution public for CUDA builds (#1452) Src/Extern/HYPRE/AMReX_HypreABecLap3.H commit 5843b838c1a86ea7a7978d0b6edd9f5e0c03d24a Author: asalmgren Date: Wed Oct 14 09:10:35 2020 -0700 Fix when small_volfrac is used -- we need to adjust the data in cut c… (#1451) Fix when small_volfrac is used -- we need to adjust the data in cut cell neighbors as well as regular neighbors This addresses issue #1450 ## Checklist The proposed changes: - [X] fix a bug or incorrect behavior in AMReX - [ ] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] are described in the proposed changes to the AMReX documentation, if appropriate Src/EB/AMReX_EB2_2D_C.cpp Src/EB/AMReX_EB2_3D_C.cpp commit 79f3a94fd50b6e9bcac6f99d0b03ffb47e9a5a87 Author: Andrew Myers Date: Wed Oct 14 00:39:23 2020 -0700 Allow users to set a Geometry object for ParticleContainer independently of the AmrCore / AmrLevel object. (#1446) Src/AmrCore/AMReX_AmrParGDB.H Src/Particle/AMReX_ParGDB.H Src/Particle/AMReX_Particles.H commit ff17a9295b54a36fa78c4f3c2350c3b3b5dd1338 Author: Houjun Date: Tue Oct 13 12:14:50 2020 -0700 HDF5 plotfile write bug fix and performance improvement (#1448) * Add HDF5 optimizations * Fix an issue that causes incorrect HDF5 data write with Tutorials/Basic/HeatEquation_EX1_C example code Co-authored-by: Houjun Tang Src/Base/AMReX_PlotFileUtil.cpp Src/Particle/AMReX_ParticleHDF5.H Tests/HDF5Benchmark/GNUmakefile commit 6649d227425fdc8ecbed301c4383c8dc74800323 Author: mic84 Date: Tue Oct 13 11:52:09 2020 -0700 CMake: add HIP support (#1316) * CMake: first basic attempt at HIP support * Update Tools/CMake/AMReXParallelBackends.cmake Co-authored-by: Axel Huebl * CI: Add HIP-Clang * fix typo: GCC 9.3 on focal * Fix sources: set two lists * fix target properties: append two lists * CMake: some changes to debug issue with generated files * CI: only run HIP check for now * CMake: fix typo * CMake: let's try this * CMake: other small modifications * CMake: attemp #1 * CMake: attemp #2 * CI: build just the bare minimum for now * CMake: attempt #3 * CMake: missing defines for HIP builds * CMake: defines were still missing :-P * CMake: AMREX_HIP_PLATFORM is actually the HIP_COMPILER * CMake: let's see if this works * Install rocRAND * CMake: re-factor and update defines for HIP compilation * CMake: list against hiprand too * CMake: find and link to hiprand * CMake: add -DNDEBUG to any type of build when HIP is on * CMake: we must manually add rocrand too * CMake: let's see if it works with Fortran enabled * Revert "CMake: let's see if it works with Fortran enabled" This reverts commit 065071539fd166f3bee994c2621d86c773180e3d. * CMake: try this * CMake: this should work too * Revert "Revert "CMake: let's see if it works with Fortran enabled"" This reverts commit 6a92adcc4ddfb06b4130f10d29cc49b298a66e73. * CMake: no fortran for now * CI: turn on linear solvers in HIP check * CI: turn on particles in HIP checks * Revert "CI: turn on linear solvers in HIP check" This reverts commit c723560a7b401266d6259af340130b1065e6be59. The reason is that Linear Solvers tutorials are not HIP-aware yet. * CMake: hipify tutorials * CMake: temporary fix * Revert "CMake: temporary fix" This reverts commit bd6724ed73f7afc82a4dc60c60d4390c6a83808d. * CMake: fix HIP compilation for Tutorials * CMake: some cleanup * CMake: add HIP options to pass in architecture and extra flags * CMake: oops * CMake: try to enable linear solvers * CMake: update ROCm version number in CI check name * CMake: enable Fortran * CMake: disable tutorials for the time being * CMake: commit custom FindHIP.cmake to debug the issue * CMake: trying this * CMake: now it should use the local FindHIP.cmake * CMake: now custom FindHIP.cmake should be able to find helper files * CMake: let's see if we really need this * CMake: let's try this * CMake: temporary fix * Revert "CMake: disable tutorials for the time being" This reverts commit 885b9113b72f54bca9eaf74f4e5b6b90b17032ff. * CMake: Tutorials/Particles/CellSortedParticles do not work with HIP * CMake: remove local FindHIP.cmake. * CMake: HIP arch flags are now PUBLIC and inheritable * CMake: no use for setup_target_for_hip_compilation anymore * CMake: remove unnecessary options for HIP * CMake: check that HIP_COMPILER is the same as CMAKE_CXX_COMPILER * CMake: better this * CI: uncomment all checks * CMake: fix visibility of arch flags for HIP * Doc: add subsection on HIP+CMake * Update Docs/sphinx_documentation/source/GPU.rst Co-authored-by: Shreyas Ananthan * CMake: fix incorrect configuration option passed to cmake * CI: target 'tutorials' no longer exists Co-authored-by: Axel Huebl Co-authored-by: Shreyas Ananthan .github/workflows/dependencies/dependencies_hip.sh .github/workflows/linux.yml Docs/sphinx_documentation/source/GPU.rst Src/CMakeLists.txt Tools/CMake/AMReXParallelBackends.cmake Tools/CMake/AMReXTargetHelpers.cmake Tools/CMake/AMReX_Defines.cmake Tools/CMake/AMReX_Options.cmake Tutorials/CMakeLists.txt Tutorials/Particles/CellSortedParticles/CMakeLists.txt commit 6dc6ce8f55a5ef19f349c267b7bb9fc4bb616b07 Author: Candace Gilet Date: Tue Oct 13 09:48:50 2020 -0400 volume multifab must be defined before calling GetVolume(volume) (#1449) Src/LinearSolvers/Projections/AMReX_NodalProjector.cpp commit 5e6d555d07d3862b5b88b6c38559a59918541194 Author: mic84 Date: Mon Oct 12 17:38:22 2020 -0700 NodalProjector: use volume-weighted average down (#1444) Src/LinearSolvers/Projections/AMReX_NodalProjector.cpp commit 95ddfdb2c1434b1705d84174047f06cd3004174d Author: Weiqun Zhang Date: Wed Oct 7 19:02:08 2020 -0700 Fix EB interpolation from cell centers to faces (#1443) ##Summary There were race conditions that made it fail on AMD GPUs. ## Checklist The proposed changes: - [x] fix a bug or incorrect behavior in AMReX - [ ] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] are described in the proposed changes to the AMReX documentation, if appropriate Src/EB/AMReX_EBMultiFabUtil.cpp Src/EB/AMReX_EBMultiFabUtil_2D_C.H Src/EB/AMReX_EBMultiFabUtil_3D_C.H commit 2c7b5004fb98d92bf185ecc73b6429dfd7ed8b6f Author: Weiqun Zhang Date: Wed Oct 7 15:16:51 2020 -0700 Fix bug in hypre overset solver (#1442) ## Summary ## Additional background ## Checklist The proposed changes: - [x] fix a bug or incorrect behavior in AMReX - [ ] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] are described in the proposed changes to the AMReX documentation, if appropriate Src/Extern/HYPRE/AMReX_Habec_2D_K.H Src/Extern/HYPRE/AMReX_Habec_3D_K.H Src/Extern/HYPRE/AMReX_HypreABecLap3.cpp Src/Extern/PETSc/AMReX_PETSc.cpp commit 3972c6d582f024c4a4d108ff02e66d117e6db1e6 Author: Max Katz Date: Wed Oct 7 15:33:35 2020 -0400 Allow tagging value to vary by level (#1441) ## Summary The new error tagging scheme from #1166 is modified to allow the threshold value to vary by level. Using the example there, if we did: ``` amr.refinement_indicators = flame_tracer lo_temp amr.refine.flame_tracer.max_level = 3 amr.refine.flame_tracer.value_greater = 1.e-6 1.e-5 amr.refine.flame_tracer.field_name = Y(H) amr.refine.lo_temp.max_level = 2 amr.refine.lo_temp.value_less = 1000. amr.refine.lo_temp.field_name = temp ``` Then level 0 would be tagged for refinement for Y(H) >= 1.e-6, while level 1 and all higher levels would be tagged for refinement for Y(H) >= 1.e-5. (If a value for every level is not provided, we assume the last value holds for all remaining levels, similar to how we treat quantities like amr.n_error_buf.) ## Additional background This will help for cases where a field varies substantially on the domain and you want to have better control over refinement so that you don't have too many zones. ## Checklist The proposed changes: - [ ] fix a bug or incorrect behavior in AMReX - [x] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] are described in the proposed changes to the AMReX documentation, if appropriate Src/AmrCore/AMReX_ErrorList.H Src/AmrCore/AMReX_ErrorList.cpp commit 92399587f20875a98655ccbd06a083965cfc9c5c Author: Weiqun Zhang Date: Tue Oct 6 13:30:52 2020 -0700 Hypre with overset (#1439) * Hypre with overset Add overset mask support for hypre. * Keep bottom_verbose restricted to amrex side. Hypre side's verbosity will be controlled by parmparse parameters. Src/Extern/HYPRE/AMReX_Habec_2D_K.H Src/Extern/HYPRE/AMReX_Habec_3D_K.H Src/Extern/HYPRE/AMReX_Hypre.H Src/Extern/HYPRE/AMReX_Hypre.cpp Src/Extern/HYPRE/AMReX_HypreABecLap.cpp Src/Extern/HYPRE/AMReX_HypreABecLap3.H Src/Extern/HYPRE/AMReX_HypreABecLap3.cpp Src/Extern/HYPRE/AMReX_HypreIJIface.cpp Src/LinearSolvers/MLMG/AMReX_MLABecLaplacian.H Src/LinearSolvers/MLMG/AMReX_MLCellABecLap.H Src/LinearSolvers/MLMG/AMReX_MLCellABecLap.cpp Src/LinearSolvers/MLMG/AMReX_MLNodeLinOp.H Src/LinearSolvers/MLMG/AMReX_MLNodeTensorLap_2D_K.H Src/LinearSolvers/MLMG/AMReX_MLNodeTensorLap_3D_K.H Tests/LinearSolvers/CellOverset/MyTest.H Tests/LinearSolvers/CellOverset/MyTest.cpp commit 6a20c352cf54ed9a1393676b1bf19c57941db2af Author: Axel Huebl Date: Mon Oct 5 16:17:43 2020 -0700 GNUmake: -Titan (OLCF) (#1440) R.I.P. Titan. Btw, only found because a colleague of mine could not compile WarpX. Their hostname is called -titan. Tools/GNUMake/Make.machines Tools/GNUMake/sites/Make.olcf commit 9992d844099c3df81d9499519ffeb7f93a201dc6 Author: Shreyas Ananthan Date: Sun Oct 4 18:19:24 2020 -0600 Hypre IJ interface: Enable access to additional solvers and preconditioners available in Hypre (#1437) This PR extends the AMReX _hypre_ IJ interface to allow access to additional solvers and preconditioners available in _hypre_. - Refactored `HypreABecLap3` and `HypreNodeLap` (IJ matrix/vector ParCSR interfaces) and created a new IJ interface class that provides a unified way to access Hypre ParCSR solvers. - In addition to BoomerAMG, adds support for 7 other [ParCSR solvers](https://hypre.readthedocs.io/en/latest/ch-solvers.html) (GMRES (4 variants), PCG, BiCGSTAB, and Hybrid). Also adds support for BoomerAMG and Euclid as preconditioners for solvers that can use a preconditioner. - Adds support to parse user options to configure hypre library via AMReX ParmParse interface and input file, using a custom namespace (default is `hypre`) - Adds `MLMG::setHypreOptionsNamespace` that allow applications to customize the namespace for different types of linear solvers (e.g., `mac_proj.hypre.