Document current backends and guard Map allocation policies

docs/changelog.md

@@ -6,6 +6,19 @@ Major implementation milestones recorded per-slice (newest first). See `git log`
 
 ## v0.1.711 — 2026-11-07
 
+Native C and LLVM programs gain addressed UDP send and receive through
+`sock_sendto` and `sock_recvfrom`, including numeric IPv4/IPv6 sender metadata,
+binary and empty packets, flags, arena bounds and thread-local errno. A
+zero-capacity receive without sender metadata reports EAGAIN; queued packets
+retain their sender. The loopback gate checks both native backends.
+
+The code-generation guide and tutorial now distinguish implemented backends
+and builtin values from their historical phase summaries. Map storage checks
+bound suffix replacement and small compacted maps, and reject controls that
+restore either previous allocation policy. The suffix parity fixture also
+keeps a copied snapshot across compaction and replacement. The reference
+documents when explicit compaction can invalidate uncopied reads on C.
+
 _A map keeps a value that is the tail of the one it replaces; a compacted
 map's first block is sized to it._
 

docs/codegen.md

@@ -1,6 +1,10 @@
 # Code generation (codegen)
 
-Mere's code-generation strategy, the current state in `mere`, and what's planned. Because the **memory model** and **effect system** at Mere's core only show their real power once codegen is in place, Phase 4 progresses through this in slices.
+Mere's code-generation strategy and backend behavior. C, LLVM IR, Wasm and
+RISC-V are implemented. The phase tables below record how they developed;
+the [standard-library reference](stdlib-reference.md),
+[host support matrix](host-matrix.md) and executable checks describe current
+support for individual operations.
 
 ---
 
@@ -27,8 +31,10 @@ The last stage of a compiler. Mere-ml started as parser → typer → **eval** (
 | Mode | Execution model |
 |---|---|
 | Tree-walking interpreter | Traverses the AST each time, looks up env, boxes/unboxes closure values, runs through OCaml's GC |
-| C codegen (current) | Runs the native binary produced by the C compiler — register allocation, function calls are one `call` instruction |
-| LLVM / Wasm (future) | Adds more optimization (inlining, dead-code elim, loop unrolling) |
+| C (`-c`) | Emits C for a native C compiler |
+| LLVM IR (`-ll`) | Emits LLVM IR for a native LLVM toolchain |
+| Wasm (`-w`) | Emits WAT for a WebAssembly engine and its host imports |
+| RISC-V (`-rv` / `-rv64`) | Emits RV32IM / RV64IM machine code without an external assembler or linker |
 
 ### Distribution
 
@@ -37,51 +43,60 @@ Codegen mode: compile `program.mere` once and you have a self-contained binary.
 
 ### Letting the memory model shine
 
-Mere's **region / view / Trivial[R] / `with` Drop** design ([memory-model.md](memory-model.md)) currently exists as type-level labels; under the hood, OCaml's GC actually manages memory.
-
-```
-// Current (interpreter):
-region R { let n = R.alloc(Node { ... }) in ... }
-// Reality: an ordinary object allocated on OCaml's heap, collected by the GC
-
-// After codegen:
-// region R push/pops a bump allocator and frees everything on scope exit
-// Drop's LIFO execution becomes a mechanical reverse-order walk over a Drop list
-```
-
-As memory-model.md says:
-> Region's real power (bump allocator, bulk free, cache locality) only emerges with **native codegen**.
+The interpreter uses OCaml-managed values. Compiled backends implement region
+allocation and reclamation in their runtimes, and `with` performs scope-bound
+cleanup. Region results and escaped values have backend-specific handling;
+some shapes are refused. See [memory-model.md](memory-model.md) for the
+ownership rules, reclamation mechanisms and executable examples.
 
 ### Becoming a self-contained language
 
-In interpreter mode, "Mere runs on top of OCaml." Once we can emit native binaries via codegen, "Mere is a self-contained language." This is required by the design goal of "both native (LLVM) and Wasm" in the internal design notes.
+The distribution compiler is built with OCaml. Programs compiled to native
+binaries run without the compiler or an OCaml installation. Wasm programs
+need an engine and the imports they use; RISC-V programs need the corresponding
+machine or emulator. See [bare-metal.md](bare-metal.md) for hosted and bare
+RISC-V execution.
 
 ---
 
 ## 3. Mere's codegen strategy
 
-### Multi-stage approach
+### Implemented targets
 
-| Phase | Target | Status | Why |
-|---|---|---|---|
-| 4 (current) | **C** | In progress | The simplest intermediate language; clang/gcc are right there |
-| 5 (future) | **LLVM IR** | Not started | More optimization opportunities; supports both native and JIT |
-| or | **Wasm** | Not started | Browser execution + WASI for self-contained binaries |
+| CLI | Target | Toolchain or host |
+|---|---|---|
+| `-c` | **C** | Clang or GCC |
+| `-ll` | **LLVM IR** | Clang or another LLVM toolchain |
+| `-w` | **Wasm text** | `wat2wasm`, then an engine with the required host imports |
+| `-rv` / `-rv64` | **RISC-V binary** | RV32IM / RV64IM execution; see the bare-metal guide |
 
 ### Why go through C?
 
-- Easier than emitting LLVM / Wasm directly (C is "portable assembly").
-- A C compiler is already on hand (clang/gcc are standard on macOS/Linux).
-- Region runtime and GC helpers can be written in C and FFI-bound.
-- When we later switch to LLVM IR, the C output serves as a "ground truth" for comparison.
+- C output can be inspected and checked with Clang and GCC, including sanitizers.
+- The native C and LLVM paths share C runtime support for facilities such as
+  sockets; LLVM programs using native FFI also link the `--ffi-c` companion.
+- Differential checks compare execution with the interpreter and other backends.
 
 ### Per-AST type annotations (introduced in Phase 4 slice 5)
 
-`Ast.expr` gains a `mutable ty : ty option`; `Typer.infer` records the inferred type on each node. This lets codegen directly read each node's type — tuple shape inference and fn signature inference become clean. The same foundation will be used for LLVM / Wasm migration.
+`Ast.expr` has a `mutable ty : ty option`; `Typer.infer` records the inferred
+type on each node. Code generators use these annotations for concrete tuple
+shapes, function signatures and specialization.
 
 ---
 
-## 4. Current state in mere (as of 2026-06-24, after Phases 4-46)
+## 4. Backend capabilities
+
+The next two phase summaries are historical snapshots from June 2026, not a
+current test count or an exhaustive support list. Current regression examples
+include [builtin values](../test/parity/builtin_as_value.mere),
+[tuple captures](../test/parity/closure_tuple_cap.mere),
+[returned closures and aliases](../test/parity/closure_alias_types.mere),
+[file reads](../test/parity/read_lines.mere) and
+[environment access](../test/parity/dirs_and_env.mere). Run them with
+`sh scripts/parity.sh <file> ...` to compare interpreter, C, LLVM and Wasm.
+
+### Historical overview: June 2026
 
 **Nearly the entire Mere syntax that's expressible in the interpreter** + **Q-010 collections (Vec / OwnedVec / StrBuf / Map)** + **FFI (`extern fn`)** + **float (3 backends)** + **Phase 36's 13 syntactic sugars** is native-compilable and runnable in 3 backends (C / LLVM IR / Wasm). **1573 tests pass**. End-to-end: factorial, fibonacci, linked-list sum, `make_adder` closures, polymorphic variant + record, pattern match (nested / guard / or). Phase 15 added all element types for Vec[R, T] / OwnedVec[T] / StrBuf[R] / Map[R, K, V] (K = int / bool / str / tuple / record / all variants) / higher-order API (vec_set / iter / fold / map / filter) / conversions (vec_to_owned / owned_vec_to_vec / vec_to_list) / `len` ad-hoc polymorphism / with-OwnedVec scope-Drop — all in 3 backends.
 
@@ -150,22 +165,31 @@ In interpreter mode, "Mere runs on top of OCaml." Once we can emit native binari
 | Polymorphic variant monomorphization | `type 'a list = Nil \| Cons of 'a * 'a list` and friends — concrete instantiations are collected from AST + fn signatures and a specialized struct is emitted per instance (`list_int`, `opt_str`, ...). Recursivity is judged on the substituted payload. So `[1, 2, 3]` literals and `'a list`-typed `sum` etc. run natively |
 | Polymorphic record monomorphization | `type 'a Box = { v: 'a }` etc. specialize like variants (`Box_int`, `Box_str`, ...); Record_lit emit reads the mono name from `.ty` and emits e.g. `((Box_int){.v = 42})` |
 | Polymorphic `show` builtin | For each `show` call, the argument type is collected and a specialized `show_T` C fn is auto-generated. Covers int/bool/str/unit/tuple/record/variant (monomorphic + polymorphic instantiations). **`'a list` is special-cased (`[1, 2, 3]` style)**; other recursive variants use `Cons (1, Cons (2, Nil))` style. asprintf-based; heap-allocates |
-| Closure (defunctionalized) | Inner `let n = fn x -> body` is defunctionalized to top level. Free vars are prepended to the C fn's params; call sites are rewritten. Captures are restricted to int/bool/str/unit (tuple/record/function-value captures not yet supported). Multi-level nesting is fine |
+| Closure (lifted) | Inner functions are lifted and captures are passed in their environment. Tuple, record and function-value captures are supported; concrete representation and region escape rules still apply. The parity corpus includes tuple captures and returned closure aliases |
 | First-class fns (Phase A + B) | `T1 -> T2` types become closure structs (`{ void* env; T2 (*fn)(void*, T1); }`). Each top-level fn gets a `_closure_fn` adapter + `_as_value` const; anonymous Fun in expression position is lifted into env struct (heap-alloc) + adapter (`__anon_N_fn`) + closure construction, with captures rewritten to `__env_self->name`. Closure dispatch: `({ __auto_type __c = e; __c.fn(__c.env, x); })`. Direct call (Var head on a known top-level) still takes the fast direct-call path |
 
-### What doesn't work (currently Codegen_error)
-
-- **Phase 35 made nullary factory builtins (vec_new / map_new / strbuf_new etc.) usable as first-class values in all 4 backends** (eta-wrap). But **partial application of multi-arg curried builtins** (`let push_to_v = vec_push v in ...`) is still future work.
-- **Auto scope-bound Drop for OwnedVec** — explicit `with v = ...` already frees at scope end; without it, bulk-freed at main exit (Phase 15.8 / 15.13; DEFERRED §1.3).
-- Inner-lifted fns (`let h = fn ...`) used as VALUES (currently only direct calls; rewriting as an anonymous Fun works).
-- Closure captures of non-primitive types (tuple/record captures; closure-value captures are fine).
-- Nested or-patterns (or inside constructor / tuple / record).
-- **Placing `while cond do body` directly under top-level main** — top-level Let_rec constraint; inside an fn body is fine.
-- **String interpolation with nested string literals** (`"x = {show \"abc\"}"`) — workaround via a let binding.
-- Some stdlib builtins (`read_lines` / `env_var` / `file_exists` etc.) are **interpreter-only**. Phase 34 brought float / libm (sqrt / sin / cos / tan / f_pow / atan2) to the 3 backends. `args` is no longer among them: it reaches all four backends as of v0.1.169 (LLVM stores argc/argv from `main` into globals; Wasm folds the host's `arg_count` / `arg_get`).
-- GC for strings / closure envs / variant nodes (currently a region arena bulk-freed at main exit — suitable for short-lived runs).
-- LLVM / Wasm with payload-mixed variants as Map K (uniform payload only) — C accepts mixed.
-- **`of_json` on LLVM**. `to_json` landed there in v0.1.184 by sharing the `show` emitter — the two differ only in literals — but decoding needs a JSON parser written in the target language, and C and Wasm each have their own hand-written one. `scripts/parity.sh` now reports what each backend did not check, which is where this shows up: six of eighty-three, all but one of them `of_json`.
+### Support boundaries
+
+Builtin values are supported when the backend implements the operation and
+its use has a concrete callable type. Factory values such as `vec_new` and
+`map_new` are supported when their return type is concrete, as in
+[factory_value.mere](../examples/factory_value.mere). An unconstrained
+`let make = vec_new` may retain a polymorphic return type; annotate it or
+pass the builtin to a concretely typed parameter. Wrapping a call in `fn`
+remains useful when an operation has a backend-specific restriction. See the
+standard-library reference for individual APIs.
+
+Floats, `read_lines`, `env_var`, `file_exists` and `args` are implemented on
+the interpreter, C, LLVM and Wasm paths. Host filesystem and environment
+access depend on the selected host. The generated host support matrix and
+the standard-library reference describe these requirements.
+
+Automatic scope cleanup for an ordinary `OwnedVec` binding is not inferred:
+use `with v = owned_vec_new () in body` for scope-bound cleanup. LLVM JSON
+decoding (`of_json` and its siblings) remains unsupported; `to_json` works
+there. Further restrictions, including region-returned functions on LLVM,
+are documented in the memory-model guide. The parity harness reports clean
+backend refusals separately from compilation failures and output differences.
 
 ### CLI
 
@@ -245,21 +269,11 @@ Detailed change history is in [changelog.md](changelog.md).
 
 ---
 
-## 6. Roadmap
-
-### Remaining work (see DEFERRED §1.2 / §1.3)
+## 6. LLVM and Wasm implementation history
 
-| Feature | What's needed |
-|---|---|
-| First-class value use of builtins | Emit per-T closure wrappers for `let f = vec_new in ...` (DEFERRED §1.2). Multi-slice effort |
-| Auto scope-bound Drop for OwnedVec | NLL + move semantics (DEFERRED §1.3); design first |
-| Nested or-patterns | Expand or inside constructors |
-| Inner-lifted fn as value | Generate a closure form alongside `let h = fn ...` when `h` is used as a value (currently rewrite as anonymous Fun) |
-| Complex closure captures | Tuple / record captures (currently int / bool / str / unit / function value only) |
-| float | `Float_lit` / `f_add` etc. → C `double` + `%g` family |
-| Long-running program support | Currently the arena is bulk-freed only at `main` exit. A sub-arena that can drop transient strings (e.g. `show` output) or a proper GC is needed for long-running processes |
-| Remaining stdlib builtins | About 70 builtins (sqrt, str_replace, ...) ported to 3 backends |
-| Payload-mixed variants as Map K on LLVM / Wasm | Loosen the uniform-payload MVP constraint (a larger change) |
+The tables in this section record completed implementation slices. They are
+kept as history; use the capability and support sections above when choosing
+a backend for a program.
 
 ### Phase 5 (LLVM/Wasm migration)
 

docs/stdlib-reference.md

@@ -1832,6 +1832,17 @@ satisfy it and hide the very row it was describing.
 | `map_recycle` | `Map[R, K, V] -> unit` | return the table's storage for reuse |
 | `map_bytes` | `Map[R, K, V] -> int` | bytes this Map currently holds |
 
+**Explicit compaction on C is a reclamation point.** A value read from a
+Map or Vec may still refer to its private arena. Finish using such reads
+before calling `map_compact`, `vec_compact`, `map_recycle` or `vec_recycle`
+on that container: the running stack does not pin those values. To keep an
+immutable snapshot across compaction, copy it out as a region result, for
+example `let saved = region Copy { map_get m key } in ...`. The copy lives
+in the enclosing region. The suspended-coroutine pinning described above
+protects stopped stacks separately. The
+[suffix fixture](../test/parity/map_set_suffix.mere) checks a copied list
+snapshot across replacement and compaction.
+
 **`OwnedVec`**: `owned_vec_get : OwnedVec[T] -> int -> T` joins `owned_vec_new` /
 `push` / `len` / `to_vec`, which are in language-reference.
 

docs/tutorial.md

@@ -649,18 +649,24 @@ mere -ll examples/map_codegen.mere | clang -x ir - -o map && ./map   # → 640
 
 # Wasm codegen (requires wabt / Node.js)
 mere -w examples/vec_codegen_wasm_typed.mere > v.wat
-wat2wasm v.wat -o v.wasm
-node -e 'WebAssembly.instantiate(require("fs").readFileSync("v.wasm"),
-  { env: { puts: () => 0 } }).then(r => console.log(r.instance.exports.main()))'
+wat2wasm --enable-tail-call --enable-threads v.wat -o v.wasm
+node scripts/run_wasm.js v.wasm
 # → 252
 ```
 
-Remaining work (see DEFERRED §1.2 / §1.3):
+Builtin factories can also be used as values on interpreter, C, LLVM and
+Wasm when the factory has a concrete return type, for example
+`let make = (vec_new : unit -> Vec[__heap, int]) in ...`.
+An unconstrained `let make = vec_new` can retain a polymorphic return type
+and be refused by codegen; use an annotation or pass it to a concretely typed
+parameter, as in [factory_value.mere](../examples/factory_value.mere).
+Other builtin values follow the operation's backend support;
+[builtin_as_value.mere](../test/parity/builtin_as_value.mere) is an executable example.
 
-- **First-class value use of builtins** (`let f = vec_new in ...`) is not yet codegen-supported — interpreter only. Workaround: write a wrapper like `fn v -> vec_push v x`.
-- **Auto scope-bound Drop for OwnedVec** is not supported — explicit `with v = owned_vec_new () in body` frees at scope end; without it, bulk-freed at main exit.
-- **The borrow checker doesn't track Vec internals at element granularity** — the mode at the point of borrowing the Vec is machine-checked; details like borrowing a `vec_get` result come later.
-- **Payload-mixed variants as Map K on LLVM / Wasm** are restricted to uniform-payload (MVP); C allows mixed.
+For scope-bound `OwnedVec` cleanup, use
+`with v = owned_vec_new () in body`; an ordinary binding does not infer a
+scope-end drop. See [the memory model](memory-model.md) for borrowing,
+container ownership and reclamation rules.
 
 For the full Q-010 design see the internal design notes.
 
@@ -926,11 +932,20 @@ For details see [codegen.md](codegen.md).
 
 Interpreter mode (`mere file.mere`) and codegen output match for the same program (including formatting like `[1, 2, 3]`). At **4-backend feature parity**, int / functions / strings / tuples / records / variants / closures / polymorphism / recursive variants / complex patterns / show / region / view / `with` Drop / list pretty-printing / Q-010 collections (Vec / OwnedVec / StrBuf / Map) + higher-order API + conversions + len + with-Drop / signature spread / Result helpers / try_or / inner-fn lifting / top-level value-binding globalization / str_compare / FFI (extern fn) all work (reached incrementally through Phases 15-32).
 
-The **feature-parity gap** between interpreter and the 3 codegen backends is now nearly zero; remaining items:
-- **First-class value use of builtins** (`let f = vec_new in ...`) is interpreter-only (future work).
-- **Auto scope-bound Drop for OwnedVec** isn't supported (explicit `with` or main-exit bulk free only; NLL / linear types are future work).
-- `float` / `'a list`-typed builtins (`read_lines` / `args` / `env_var` / `file_exists` etc.) are **interpreter-only** (codegen comes in a separate phase).
-- LLVM / Wasm accept only **uniform payload for Map K with payload variants** (C allows mixed).
+Backend support depends on the operation and its value's shape. Floats,
+builtin factory values, `read_lines`, `args`, `env_var` and `file_exists`
+are implemented on interpreter, C, LLVM and Wasm. Filesystem and environment
+operations use the selected host's facilities. See the
+[standard-library reference](stdlib-reference.md) and
+[host support matrix](host-matrix.md) for individual APIs and host requirements.
+
+Some boundaries remain: `OwnedVec` needs explicit `with` for scope-bound
+cleanup, LLVM does not implement `of_json` decoding, and LLVM refuses a
+function returned from a region block. The [codegen guide](codegen.md) and
+[memory model](memory-model.md) describe these restrictions. Run
+`sh scripts/parity.sh <file> ...` to check a concrete program across the four
+backends; it distinguishes a named refusal from a compilation failure or
+different output.
 
 ## 14. Formatting source (`mere fmt`)