Audit umbrella: documented behaviour the code does not have (schema link restrictions, export flags, doc recursion)

[avrabe]

Umbrella from the inconsistency audit, claim-vs-code lens. Headline instance filed separately: #952 (sql read-only claim + --qualification-mode scope).

Note rivet docs check passes (64 files, 0 violations) — none of these gaps are machine-gated, which is itself the pattern.

Verification status: agent-reported with claim file:line + code file:line + an empirical check for each. I re-ran the sql and --qualification-mode ones myself (#952); the rest are recorded as reported.

1. link-types: source-types / target-types are documented as restrictions and never enforced

schemas/common.yaml:102-106 — the header defining the construct for all 30 shipped schemas: "source-types — (optional) restrict which artifact types may originate / target-types — (optional) restrict which artifact types may be targeted."

rivet-core/src/schema.rs:306-308 declares the fields; no validation path reads them. The only non-render reader is validate.rs:1254-1256, a suppression heuristic for prose-mention-without-typed-link — it uses the restriction to decide when not to warn. The sole enforced restriction is the artifact-type-level link-fields[].target-types (validate.rs:925-926). There is no source-type check anywhere.

Empirical: stpa.yaml:314-318 declares prevents with source-types: [system-constraint, controller-constraint], target-types: [hazard, sub-hazard]. A uca emitting prevents → L-001 (a loss) violates both, and validate returns PASS with zero diagnostics — while the coverage rule still reports the hazard as "needs an incoming prevents link". The illegal link is simultaneously unflagged and uncounted.

Highest-consequence item in this list: a schema author declaring source-types believes the graph is constrained.

2. export --format html — three documented flags silently ignored, config.js never generated

main.rs:742,746,750 document --homepage "written to config.js", --version-label, --versions. They arrive as _homepage, _version_label, _versions_json — underscore-prefixed unused params at main.rs:12114-12116. cmd_export_zola doesn't take them at all.

docs.rs:1807 — "Runtime customization is done entirely through config.js"; :1813 shows a file tree with config.js, requirements.html, matrix.html…; :1804 says "CSS is embedded inline with no external dependencies"; :1951 describes an externalCss mechanism. RIVET_EXPORT, config.js, externalCss and all seven documented CSS classes appear only in docs.rs.

Empirical: export with all three flags produces no config.js and no file containing the URL or label; index.html has zero <style> tags and links href="_assets/styles.css". Real layout is artifacts/index.html, matrix/index.html, etc.

3. Document scanning is non-recursive; the prose says recursive, and skips are silent

docs.rs:920-921 — - docs # loads docs/*.md recursively. document.rs:406 is std::fs::read_dir(dir) — one level. The module's own test comment at :2590 concedes it.

docs/ holds 60 .md but only 21 at top level; 4 frontmattered documents under docs/research/, docs/design/, docs/historical/ are invisible to the link graph, dropped with no warning — contradicting the "Loud-by-default doc scanning" design at docs.rs:625, since a file never enumerated cannot be declined. Knock-on: the exclude: "generated/**" example at :623 can never match.

4. init --vendor-schemas "immune to release-to-release rule drift" — false for bridges

main.rs:303 / docs.rs:790 make that claim. embedded.rs:249 discover_bridges() iterates the compiled-in list and falls back to embedded_bridge() when the on-disk file is absent, so a bridge added in a later release fires on a project vendored before it existed.

Empirical: with common/dev/stpa vendored and no stpa-dev.bridge.yaml, schema rules shows constraint-has-requirement active and rivet schema sources prints stpa-dev.bridge … embedded (compiled into rivet — changes on upgrade; vendor under schemas/ to pin) — the binary contradicting its own help text in one run.

5. ai-generated-needs-review can never fire on a valid store

schemas/common.yaml:430 — "AI-generated artifacts with active/approved status must have a reviewer." The rule body uses when: {field: status, equals: active}, but active is not among the nine values in the status enum (common.yaml:58-67), which is gated at Error severity. Condition::Equals is single-valued, so approved — also named in the description — never matches either.

Empirical: an AI-authored requirement with status: approved and no reviewer produces no such diagnostic (a sibling conditional rule does fire, proving the machinery works); with status: active it errors on the enum.

The shipped review gate on AI-authored artifacts emits zero warnings on every valid store — in a repo whose tool-confidence story leans on AI provenance. check_conditional_consistency (schema.rs:929-981) never cross-checks a when.equals value against the field's allowed-values, so nothing reports the dead rule.

6. AGENTS.md documents the release plan as baseline; the field is release

AGENTS.md:188-198 — "rivet's release plan lives in the baseline field … Scope a release = rivet list --filter '(= baseline "vX.Y.Z-track")'". The model field is release (model.rs:322), with --release, --set-release and a rivet release subcommand. sexpr_eval.rs:581-586 falls unknown bare names through to fields.get(...), so the query returns empty rather than erroring.

Empirical: --filter '(= baseline "v0.35.0-track")' → 0 artifacts; --release v0.35.0 → 12.

Second gap in the same file: AGENTS.md:7-10 says hand-authored content "lives below [the markers] … they survive regeneration" — but this section is at lines 186-205, inside the marker pair (END at :220). managed_section.rs:121 replaces the whole region, so rivet init --agents destroys it.

7. Smaller

• docs.rs:2736-2738 — "At schema load time … the schema is rejected with a diagnostic." It's a validate-time Warning (schema.rs:936), reached from validation not load, and requirements_overlap only fires on identical requirements. Three overstatements in one sentence.
• docs.rs:1085 — "Orphan detection: Artifacts never referenced in any document are flagged." links.rs:216-221 is "no incoming or outgoing links"; documents are not consulted.
• main.rs:1522-1525 — "schemas for every --format json output (validate, stats, coverage, list)". Five ship (incl. query); ~52 --format json sites exist; diff, matrix, audit, commits, impact have none.
• docs.rs:2106,2108,2240 — "The 15-Tool Catalog" / "fifteen tools". rivet mcp --list-tools → 16; the table omits rivet_bundle. CLI_DOC in the same binary says "the 16-tool catalog".
• README.md:89 — rivet init --preset dev # scaffold rivet.yaml + schemas/ + artifacts/. main.rs:4355 guards the only create_dir_all(&schemas_dir) behind if vendor_schemas. A fresh init produces no schemas/.

Explicitly not filed

Stale counts with no behavioural consequence (schema type-count tables, "33 Playwright spec files", "28 Kani harnesses", README.md:152's "v0.5.0 series"), and normative statements whose adjacent comments already disclose non-enforcement (DSGVO at common.yaml:228-230, the SIL/DAL parentheticals). Worth a tidy pass, not a bug.

[avrabe]

Triage note (scheduled routine, first pass on this issue — filed 2026-09-16, zero prior comments).

Umbrella at the "documented-behaviour-not-in-code" lens with six items, one meta-observation (rivet docs check PASSes with 64 files / 0 violations — none of the gaps are machine-gated, which is itself the pattern) and one headline instance already filed separately (#952 — sql write-vs-read-only + --qualification-mode scope). The highest-consequence item is item 1: source-types / target-types are documented as restrictions for all 30 shipped schemas at common.yaml:102-106, and no validation path reads them — the illegal-link fixture in the body is simultaneously unflagged (validate says PASS) and uncounted (the coverage rule still reports the target as needing the link). Item 5 is a different subtlety: the shipped ai-generated-needs-review rule at common.yaml:430 can never fire on a valid store because when: {field: status, equals: active} names a value that isn't in the enum, and check_conditional_consistency doesn't cross-check when.equals against allowed-values.

Verification status: agent-reported with per-item claim file:line + code file:line + empirical check. The maintainer re-ran items in #952 personally; the rest are recorded as reported.

There's no ## Acceptance Criteria block on the body a routine pass can gate on. Sketch matching the umbrella-triage pattern used across adjacent threads (#956, #957):

## Acceptance Criteria — umbrella closure

- [ ] Each of items 1–6 has a follow-up issue filed with its own testable AC block. This umbrella tracks the class; the individual issues carry the fixes.
- [ ] Every fix ships with a **claim-refuting fixture**: an input the documented behaviour says would produce X, driven through the code, asserted to produce X. If it does not, either the code is fixed or the claim is retracted — the umbrella doesn't allow "both surfaces stay drifted".
- [ ] Umbrella is closed only when items 1–6's follow-ups are closed AND a repository-level convention lands: `rivet docs check` (or a sibling gate) is extended to look for the specific class of drift this umbrella surfaces — documented behaviour not backed by a validation or test path.

## Item-level AC — item 1 (link-types source-types / target-types never enforced)

- [ ] `validate.rs` grows a new check `link-source-type-mismatch` (severity: Error) that reads the `source-types` restriction declared on each `link-types` entry and fires on links whose source violates it. Wires through `check_conditional_consistency` or a parallel path — maintainer pick.
- [ ] Regression fixture (the maintainer-constructed one): a `uca` emitting `prevents → L-001` (a `loss`) — `source-types: [system-constraint, controller-constraint]`, `target-types: [hazard, sub-hazard]`. Current behaviour: PASS, zero diagnostics, `L-001` reported as needing `prevents`. Post-fix: FAIL on the illegal `prevents`, `L-001` no longer counted as covered by the illegal link.
- [ ] The `prose-mention-without-typed-link` suppression heuristic at `validate.rs:1254-1256` (currently the only reader of the restriction) is documented in a comment referencing the new check, so future readers see the pair.
- [ ] Commit trailers: `Fixes: REQ-<N>` + `Refs: #958`.

## Item-level AC — item 2 (export --format html: 3 documented flags silently ignored, config.js never generated)

- [ ] Two directions the AC needs to pin — either:
  - **(A)** `main.rs:12114-12116` picks up `_homepage`, `_version_label`, `_versions_json` and threads them into `cmd_export_html`; `config.js` and inline `<style>` (not `_assets/styles.css`) get generated matching `docs.rs:1807-1951`; the `externalCss` mechanism becomes real. Regression: a full export contains `config.js`, and the emitted `index.html` inlines styles with no external `href="_assets/styles.css"`.
  - **(B)** `main.rs:742,746,750` drop the flags from the help; `docs.rs:1807-1951` collapses to describe the actual behaviour (external `styles.css`, no `config.js`, `matrix/index.html` layout); the seven documented CSS classes get their real names.
- [ ] Whichever direction is picked, the counterpart docs / flags are updated to remove the drift — no half-fix that leaves the other surface still lying.
- [ ] Commit trailers: `Fixes: REQ-<N>` + `Refs: #958`.

## Item-level AC — item 3 (document scanning non-recursive, silent skips)

- [ ] `document.rs:406` uses `walkdir::WalkDir` (or equivalent) instead of `std::fs::read_dir(dir)`. All 60 `.md` files under `docs/` become visible to the link graph.
- [ ] Regression fixture: a project with a `.md` file at `docs/research/foo.md` carrying a frontmatter block. Current behaviour: file invisible, no warning. Post-fix: file in the link graph AND appears in `rivet docs check` output.
- [ ] The `exclude: "generated/**"` example at `:623` works as documented (currently can't match because generated/ contents are never enumerated).
- [ ] The "Loud-by-default doc scanning" design at `docs.rs:625` gets a test — a `.md` file inside `docs/` that's malformed produces a warning, not silence.
- [ ] `docs.rs:920-921` comment either matches the code (drops "recursively") or the code matches the comment (goes recursive). Pick one; both surfaces converge.
- [ ] Commit trailers: `Fixes: REQ-<N>` + `Refs: #958`.

## Item-level AC — item 5 (ai-generated-needs-review can never fire)

- [ ] `schemas/common.yaml:430` `when.equals` clause is fixed — either uses a value in the `status` enum (`approved`, `verified`, etc.), or the enum grows `active` (unlikely — the description says "active/approved" but `status` is a lifecycle, not a boolean).
- [ ] `check_conditional_consistency` (`schema.rs:929-981`) cross-checks every `when.equals` value against the field's `allowed-values` and errors on unmatched values at schema-load time. Regression fixture: a scratch schema declaring a rule with `when: {field: status, equals: not-in-enum}` fails to load with a specific error naming the drift.
- [ ] Post-fix behaviour: an AI-authored requirement with `status: approved` and no reviewer fires the rule. The current behaviour (zero diagnostics on every valid store) is asserted as the pre-fix baseline.
- [ ] Commit trailers: `Fixes: REQ-<N>` + `Refs: #958, REQ-059` (or whichever REQ tracks the AI-provenance review-gate — check `#104`'s state).

## Item-level AC — item 6 (AGENTS.md documents baseline; field is release)

- [ ] `AGENTS.md:188-198` is corrected: `baseline` → `release`, the filter example uses `--release`, the query surface names `rivet release` (subcommand).
- [ ] `sexpr_eval.rs:581-586` grows a warning for unknown bare names that fell through to `fields.get(...)` — turning a 0-row query for a typo'd field name into a legible "no field named 'baseline'" instead of silent emptiness.
- [ ] `AGENTS.md:7-10` marker-region issue: the "hand-authored content lives below the markers" claim is either made true (move the section outside the marker pair) or the sentence is corrected. The `managed_section.rs` regeneration is exercised to prove the choice sticks.
- [ ] Commit trailers: `Fixes: REQ-<N>` + `Refs: #958`.

## Item-level AC — item 4 (init --vendor-schemas: "immune to rule drift" false for bridges)

- [ ] Either `main.rs:303` / `docs.rs:790` retract the "immune" claim for bridges, OR `discover_bridges` (`embedded.rs:249`) stops falling back to `embedded_bridge` when a project has vendored schemas (opt-in via a vendored bridge manifest).
- [ ] `rivet schema sources` on a vendored project reports the effective bridge set unambiguously — no "compiled into rivet" line for a project that vendored its schemas without saying so, or an explicit "unvendored fallback" marker if the fallback is preserved.
- [ ] Regression fixture: the maintainer-constructed one (`common`/`dev`/`stpa` vendored, no `stpa-dev.bridge.yaml`) asserts current behaviour, then the fix.
- [ ] Commit trailers: `Fixes: REQ-<N>` + `Refs: #958`.

## Suggested per-item filing order

1. Item 1 (highest-consequence — the illegal-link case is both unflagged and miscounted; schema authors relying on the restrictions have a wrong mental model of the graph)
2. Item 5 (structural — one dead rule + one missing meta-check that would catch the whole class)
3. Item 6 (small fix, high leverage — the query surface goes empty on the documented filter and no warning fires; drives real user friction)
4. Item 3 (silent skips are actionable and legible once the fix lands; 4 documents materialise in the link graph)
5. Items 2, 4 (two-direction fixes; require a maintainer call to pick the direction)

## Explicitly out of scope for the umbrella (file separately if wanted)

- [ ] A `rivet docs check` extension that specifically hunts the drift class this umbrella surfaces (documented behaviour not backed by validation or test). Substantial enough for its own REQ; would close the meta-pattern rather than each instance.
- [ ] The wider "surfaces disagree about the same fact" umbrella (#956) — sibling lens; some overlap on the `check_conditional_consistency` meta-check but different scope.
- [ ] The wider "gates that pass vacuously" umbrella (#957) — sibling lens; item 1's PASS-with-illegal-link is a vacuous-green story too, but the fix lives in the schema-driven validation path, not in gate wiring.

Three coordination items worth a maintainer call before AC is locked:

• Verification of the agent-reported items. The body flags most items as agent-reported and lists which the maintainer verified (sql/--qualification-mode in rivet sql help says "read-only" and "writes are a planned follow-up slice" — it writes to disk by default, and --qualification-mode does not stop it #952). Doing a verification pass on items 1, 3, 5, 6 before pinning per-item AC (or as its own explicit slice) would avoid the "fix a misread" failure mode. Items 2 and 4 have empirical evidence embedded already.
• Item 2 direction. (A) makes the docs true; (B) makes the code match the docs. (A) is more useful (users get config.js, inline CSS, versioned nav); (B) is a much smaller diff. Pinning direction before implementation matters — reviewer expectations differ substantially.
• check_conditional_consistency scope. Item 5's second bullet (meta-check on when.equals against allowed-values) closes a wider class than just ai-generated-needs-review — every conditional rule in every schema benefits. If it's landed carefully, it might surface other dead rules in the corpus. Doing the meta-check first (as its own PR) and then fixing item 5's specific equals: active value would produce a legible diagnostics-driven fix rather than "we noticed a dead rule and fixed it".

Adjacent state worth flagging:

• rivet sql help says "read-only" and "writes are a planned follow-up slice" — it writes to disk by default, and --qualification-mode does not stop it #952 (the headline instance for sql + --qualification-mode) is filed and needs its own AC block per the triage sent there separately.
• Audit umbrella: surfaces that disagree about the same fact (coverage %, validate verdict, export vs serve) #956 (equivalence umbrella) and Audit umbrella: gates that pass vacuously (traceability, cargo-vet, conformance suites, Playwright) #957 (weak-green umbrella) are the sibling lenses. Cross-linking the three so a reviewer sees the audit as one lens rather than three would help.
• The meta-observation that rivet docs check PASSes on all 64 files is itself a candidate for a follow-up: what class of drift would a docs check --strict catch that today's linter doesn't? Worth capturing as a separate research issue if a fix isn't scoped soon.

Process side-rail: https://pulseengine.eu/blog/ is up today (200 OK). Fetched at the start of this run — no post speaks to per-issue AC shape or claim-vs-code convention, so the kill-criterion posture used across adjacent triage threads remains the working default.

Not implemented this run — no PR opened; the umbrella-level AC + per-item AC on the follow-up issues are the gating inputs.

---

Generated by Claude Code