AMR — Parallel Adaptive Mesh Refinement

A linear (pointerless) adaptive mesh refinement engine built on a Morton/Z-order–encoded quadtree (2D) / octree (3D). Leaves are stored as a sorted array of (code, level) pairs; refinement, 2:1 balancing, and coarsening operate on that array. The design follows the p4est/t8code family ([BurstWG2011], [Holke2018]) and the non-conforming-AMR constraint framework of [CDK2019]; see REFERENCES.md.

📖 New here? Start with docs/ — getting started, how-tos, and design notes.

Backends

Three implementations share the same algorithm and the same Morton core, each in its own directory:

Dir	Backend	Parallelism	Build	Tests
omp/	shared-memory C++	OpenMP	CMake or g++	Catch2 (omp/test_*.cpp)
mpi/	distributed C++	MPI (+OpenMP per rank)	CMake (-DAMR_BUILD_MPI=ON)	Catch2 (mpi/tests.cpp)
cuda/	single-GPU	CUDA + Thrust	CMake (-DAMR_BUILD_CUDA=ON)	custom harness (cuda/tests.cu)

Each C++ backend has the same module layout: core (Morton + strong types), tree (refine/balance/coarsen), physics (oracles), viz (SVG/VTK), plus main and tests.

simplex/ is separate from the octree: header-only red refinement of unstructured and hybrid meshes, with hanging nodes as weighted constraints, 1-irregular balance and the prolongation; structured seeds (structured.hpp); and one solver cycle, adapt = marking (common/marking.hpp, [Doerfler1996]) → balance → refine. The refinement is element-agnostic. Element shapes come from one model (element.hpp): products of simplices (T3, T4, Q4, H8, prism), refined factor by factor with Freudenthal's subdivision ([Freudenthal1942], [Bey2000]; the tetrahedron's diagonal as in [Bey1995], [Zhang1995]), plus a hand-made pyramid. Any mix of one dimension's shapes stays conforming. On the solver's triangle and tetrahedron meshes it reproduces the MATLAB meshes exactly. Tests: Catch2 (simplex/test*.cpp), built by CMake as amr_simplex_tests; amr_simplex_bench times one solver cycle on growing meshes. amr_simplex {refine|balance|adapt|structured} IN OUT runs it on SMX2 files (simplex/io.hpp), the exchange format MATLAB drives it through.

Invariants

• Sorted leaves: leaf_codes is ascending; refine/coarsen re-establish this.
• Unique leaves: no duplicate codes; no leaf is an ancestor of another.
• 2:1 balance: after balance(), face-adjacent leaves (and, in 3D, edge-adjacent leaves) differ by ≤ 1 level; corner-only neighbours are not constrained.
• Partition of unity (MPI): summed leaf volumes across ranks == 1.0 (checked by verify_global()).

Build & test

OpenMP (omp/)

# Via CMake (fetches Catch2):
cmake -S . -B build -DCMAKE_BUILD_TYPE=Release && cmake --build build
ctest --test-dir build

# Or directly against system Catch2, no network:
g++ -std=c++20 -O2 -fopenmp -Iomp omp/test_*.cpp \
    /usr/lib/libCatch2Main.a /usr/lib/libCatch2.a -o omp/test && omp/test
OMP_NUM_THREADS=8 g++ -std=c++20 -O2 -fopenmp -Iomp omp/main.cpp -o omp/amr && omp/amr

MPI (mpi/)

cmake -S . -B build -DAMR_BUILD_MPI=ON && cmake --build build
mpirun -np 4 ./build/amr_mpi_tests
mpirun -np 4 ./build/amr_mpi

In containers / sandboxes where the shared-memory transport stalls, force TCP: mpirun --mca btl tcp,self -np 4 ./build/amr_mpi_tests.

CUDA (cuda/)

# set the arch to your GPU (default sm_86 → AMR_CUDA_ARCHITECTURES=86):
cmake -S . -B build -DAMR_BUILD_CUDA=ON -DAMR_CUDA_ARCHITECTURES=80 && cmake --build build
./build/amr_cuda   # needs a working CUDA driver matching the runtime

The Thrust device lambdas require nvcc's --extended-lambda, which CMake sets automatically for the CUDA targets. The code compiles and links without a GPU; running needs a driver whose version matches the CUDA runtime.

Benchmarks & docs

• benchmarks/ — scripts, historical data, and the Colab notebook (see its README; new scaling runs are produced by the Stage-2 SLURM harness).
• omp/PARALLELIZATION.md, cuda/WALKTHROUGH.md — implementation notes imported from the earlier development line (marked historical where they describe superseded code).

Status

refine, balance, and coarsen are verified across backends:

• OpenMP: 50038 assertions / 8 cases pass.
• MPI: full suite passes at 1/2/4 ranks (50038 assertions).
• CUDA: compiles and links clean (runtime needs a working GPU).

OpenMP strong scaling (sample)

Fixed demo (max_level=20, ~89.3M leaves), omp/ backend, 32-core host; 3D balance() wall-clock. Final leaf count is identical at every thread count (parallel determinism). Full data: benchmarks/results/omp_scaling_restructured_2026-05-29.csv.

threads	3D balance (ms)	speedup
1	58267	1.0×
2	30305	1.9×
4	15634	3.7×
8	8365	7.0×
16	5927	9.8×
32	4701	12.4×

Proper strong/weak-scaling sweeps (parametrized sizes, MPI ranks, GPU) are produced by the Stage-2 SLURM harness, not this fixed demo.

License

BSD 3-Clause — see LICENSE.