Audit umbrella: gates that pass vacuously (traceability, cargo-vet, conformance suites, Playwright)

[avrabe]

Umbrella from the inconsistency audit, weak-green lens. Headline instances filed separately: #953 (compliance discards validate's exit code), #954 (check verification-evidence ok-on-empty).

Ground truth: gh api repos/pulseengine/rivet/branches/main/protection → required_status_checks.contexts == ["CI Gate"], enforce_admins: true. Severity below is measured against that.

Verification status: agent-reported, several with constructed negative controls. I verified #953 and #954 myself; the rest need confirming before fixing.

1. The traceability gate cannot fail on a traceability regression — negative control constructed

ci.yml:304 runs rivet validate. On the real corpus: Result: PASS (931 warnings), exit 0. Every traceability finding is Warning or Info — 286 prose-mention-without-typed-link, 240/87 requirement-verification, 214/73 requirement-coverage, 119 orphan-artifact.

Control: appending one implemented requirement with no links, no design, no test still gives PASS, exit 0 (935 warnings). A PR merging an untraced implemented requirement passes the gate whose section header reads "The project that BUILDS the traceability tool must gate its own PRs on traceability."

The step comment is honest about it. But rivet-cli/src/docs.rs:1468,1498 publish to users the recipe rivet validate --strict-cited-sources --fail-on warning. rivet ships a stricter gate to its customers than it runs on itself. The ready-made ratchet — --new-since <ref> --fail-on … at main.rs:5445 — is wired into no workflow.

2. Supply Chain (cargo-vet) audits zero dependencies

ci.yml:1281-1313:

if [ ! -d supply-chain ]; then cargo vet init; ... fi
cargo vet --locked

ls supply-chain/ → No such file or directory. git log -- supply-chain → empty; it has never existed and is not gitignored. So init runs every time, and cargo vet init records every current dependency as an exemption precisely so the initial state passes. Observed green in run 35052742650. Also outside CI Gate's needs.

3. Miri and Proptest are outside CI Gate and outside its documented exclusion list

ci.yml:1520-1524 enumerates exclusions as "Kani, Mutation Testing, Coverage, Playwright" — presenting itself as complete. Also absent from needs and from that list: Miri (safety surface), Proptest (extended), Security Audit, Supply Chain, VS Code Extension, Zola export smoke, Traceability (hosted fallback).

Miri is the only gate over the std::mem::transmute of raw SyntaxKind in sexpr.rs/yaml_cst.rs — the sole unsafe in rivet-core. A Miri red cannot block a merge. (Whether that's intended is the decision; the list being wrong is the bug.)

4. yaml_test_suite.rs — 68 conformance cases blind to total loss of parse structure

Module doc promises three properties; the third ("graceful Error recovery") is never asserted — fn parse_has_errors at :93 has zero call sites. No test inspects tree shape. If yaml_cst::parse degenerated so every document lexed into one flat Scalar, root.text() == input would still hold (rowan is lossless by construction) and all 68 cases stay green against a CST with no mappings or sequences.

5. sexpr_fuzz::roundtrip_equivalence — the failure mode is prop_assume!d away

sexpr_fuzz.rs:375: prop_assume!(reparsed.is_ok(), "pretty-print must re-parse"). prop_assume! rejects the sample rather than failing. The one outcome the property exists to detect is routed to "skip". Partial breakage never surfaces.

6. unknown-field is Severity::Info

validate.rs:1187. Control: renaming priority→prioriti, category→categorie, adding relase: on REQ-001 gives PASS, exit 0 with three INFO lines. A rename silently detaches schema-declared fields; release-scoping queries go empty; the gate stays green.

7. Runner Liveness — the liveness branch is unreachable (opposite failure from REQ-354)

runner-liveness.yml:114-121 queries repo-scope runners. The workflow's own issue body at :264 says "runners are registered at the ORG. Repo scope returns 0 on a perfectly healthy pool and can never say otherwise." So total is always 0, the -gt 0 guard fails, and the "All N offline" branch is unreachable. Scenario: pool dies on a quiet Sunday with nothing queued → probe sees total=0 and oldest_age=0 → posts "✅ Runner pool healthy again" and closes the alert.

8. || true on the release path

• scripts/build-wasm.sh:27-28 — success is echoed before the check, and the ls verifying both wasm filenames has stderr and exit code discarded. The only artifact assertion on the release path. (See Two spar revisions ship in one binary: native pinned to v0.10.0, wasm cloned from HEAD, and the drift guard is dead code #951.)
• ci.yml:652 — cargo install cargo-deny --version 0.20.2 || true silently falls back to a preinstalled 0.16.4 whose unmaintained default differs. This one is in CI Gate's needs, so it ranks highest of this group. REQ-347's own text calls that "exactly how a gate passes vacuously".
• ci.yml:1499 — cp target/nextest/ci/junit.xml … || true; the release tarball ships without test results, green.

9. Playwright specs that assert nothing

• documents.spec.ts:32,53,84 select a[hx-get^='/documents/'], but render/documents.rs:53,81 emits href-only links. All three loops iterate an empty list; :84 then logs "No .doc-ref links found" while the corpus holds 234 [[…]] refs.
• rendering-invariants.spec.ts:239 — comment says "Strict inequality: scoping must yield fewer nodes", assertion is toBeGreaterThanOrEqual. graph_view could ignore variant entirely and pass. The sibling at :243 gets it right.
• audit-regression.spec.ts:104 — XSS regression test whose only assertion is body?.length > 0, inside an unguarded if (isVisible()) with no else/test.skip().
• coverage-view.spec.ts:58, source-view.spec.ts:45,81 — assert row counts / body.length > 100; the nav shell alone clears them.

10. Smaller, verified

• verify_archive_size.sh — cap with no floor; an empty report tars to ~45 bytes and passes (see Release path: the compliance action discards rivet validate's exit code, so a release can ship on a failing validate #953).
• action.yml:308 — grep 'name:' rivet.yaml | head -1 | awk … in bash -e without pipefail; a missing file gives PROJECT="" and a misnamed archive that looks intentional.
• zola-export-smoke.sh:93 computes PAGES after the verdict; an empty content dir prints OK: … no absolute artifact-link leaks. Its leak regex requires exactly one path segment, so href="/artifacts/req-001/" evades it. (Advisory — continue-on-error: true.)
• check-schema-version-bump.sh:30-33 — git diff | grep -v … || true swallows a failing git diff; STRICT defaults to 0 and ci.yml:257 never sets it.
• install-hooks.sh:68 — … || echo "0"; a JSON-shape change or a validate panic yields errors=0 and the hook commits.
• rivet-core/tests/integration.rs:1269 — strictdoc_reqif_import gated on /tmp/zephyr-reqif/… which no script creates. Dead on every machine.

The inverse, and the cheapest thing here

rivet check bidirectional exits 1 on the live corpus and is run by no workflow (grep -rn 'rivet check' .github/ matches only verification-evidence). A working oracle wired into nothing.

[avrabe]

Triage note (scheduled routine, first pass on this issue — filed 2026-09-16, zero prior comments).

Umbrella at the "weak-green" lens — CI Gate is the only required-status check on main per branch protection, so a gate outside its needs: cannot block a merge, and the ten items measure that surface against its own claims. Highest-consequence items are those in CI Gate's needs that pass vacuously (item 1 traceability regression can't fail the gate; item 8 cargo install cargo-deny … || true silently falls back to an older version whose defaults differ), and Runner Liveness (item 7) whose branch is unreachable — the opposite failure from REQ-354. Item 5 (prop_assume! routing the one failure mode the property exists to detect to "skip") is a subtle-but-total defeat; item 4 (yaml_test_suite blind to structural collapse) is the same shape at a different surface.

Verification status: some items marked as agent-reported. #953 (compliance discards validate's exit code — already has PR #961) and #954 (check verification-evidence ok-on-empty) are the maintainer-verified headline instances filed separately.

There's no ## Acceptance Criteria block on the body a routine pass can gate on. Sketch matching the umbrella-triage pattern used across adjacent threads (#956, #958):

## Acceptance Criteria — umbrella closure

- [ ] Each of items 1–10 has a follow-up issue filed with its own testable AC block. This umbrella tracks the class; the individual issues carry the fixes.
- [ ] Every fix ships with a **negative control** — a synthetic input crafted to reproduce the vacuously-green failure mode, asserted to redden `CI Gate` on a scratch branch (or exit non-zero locally where the item is not in `CI Gate`'s needs). "Fix passes green CI" isn't sufficient; the pre-fix vacuous-green must be shown to have been possible.
- [ ] Umbrella is closed only when items 1–10's follow-ups are closed AND a repository-level convention lands: gates in `CI Gate`'s `needs:` are cross-checked against the workflow's own documented-exclusion list on every CI run (item 3's shape — the list being wrong is the bug).

## Item-level AC — item 1 (traceability gate cannot fail on a traceability regression)

- [ ] The `rivet validate` invocation in `ci.yml:304` grows either `--strict-cited-sources` + `--fail-on warning` (matching what `rivet-cli/src/docs.rs:1468,1498` publishes to users) OR the ratchet form `--new-since <base-ref> --fail-on warning` (matches `main.rs:5445`, admits the 931 existing warnings, catches new ones).
- [ ] Negative control (the maintainer-constructed one): an untraced `implemented` requirement reddens the gate. Assertion: the same input on `main` today produces `Result: PASS, exit 0` (measured); post-fix it exits non-zero.
- [ ] Rationale note in a `decision` artifact: which of `--fail-on warning` (absolute) or `--new-since --fail-on warning` (ratchet) rivet picks, and why. Ratchet is more likely to actually ship; absolute is what rivet publishes to consumers.
- [ ] Commit trailers: `Fixes: REQ-<N>` + `Refs: #957`.

## Item-level AC — item 8 (|| true on the release path)

- [ ] `ci.yml:652` `cargo install cargo-deny --version 0.20.2 || true` is replaced with a hard install (no `|| true`, no fallback). Version pinning is preserved so `unmaintained` default doesn't drift silently. Highest-priority of the three sub-items — this one is in `CI Gate`'s needs.
- [ ] `scripts/build-wasm.sh:27-28` — the `ls` verifying both wasm filenames captures its exit code and fails the script on nonzero. The "success is echoed before the check" ordering is reversed. Ties into #951.
- [ ] `ci.yml:1499` `cp target/nextest/ci/junit.xml … || true` — either the missing junit.xml is a hard error (fixing the underlying "why does it sometimes not exist"), or the release tarball's contents-manifest asserts junit.xml is present before shipping.
- [ ] Regression fixture (repo-level): a pre-commit hook or a `test-ci-vacuous-green` job scans `.github/workflows/*.yml` for `|| true` on lines that carry `cargo install`, `cp target/`, or `mv target/` and hard-fails on new instances. Convention lives in a `decision` artifact.
- [ ] Commit trailers: `Fixes: REQ-347` (or a new REQ if that one is scoped elsewhere) + `Refs: #957`.

## Item-level AC — item 7 (Runner Liveness posts "healthy" on a dead pool)

- [ ] `runner-liveness.yml:114-121` queries **org-scope** runners instead of repo-scope (matching what the workflow's own issue body at `:264` says).
- [ ] Regression fixture: the workflow's `--dry-run` mode (or a scratch scenario) exercises "0 runners registered at org scope" and asserts the `-gt 0` guard fires and posts the "N offline" branch. Currently that branch is unreachable.
- [ ] Ties into the REQ-354 fix (opposite failure — that one measured latency and had no reachable right answer; this one measures the wrong scope and has no reachable wrong answer).
- [ ] Commit trailers: `Fixes: REQ-<N>` + `Refs: #957, REQ-354`.

## Suggested per-item filing order (weighted by CI Gate membership)

1. Item 8's `cargo install cargo-deny` sub-item (in CI Gate's needs, one-line fix, silent policy drift on unmaintained defaults)
2. Item 1 (in CI Gate's needs, highest-consequence — the traceability gate)
3. Item 6 (unknown-field is Info — trivial severity bump, high leverage on rename-detection)
4. Item 7 (Runner Liveness — orthogonal to CI Gate, but the wrong-answer-on-quiet-Sunday scenario is what triggered REQ-354 in the first place)
5. Items 3, 4, 5 (structural coverage gaps — each independently useful, none in CI Gate today)
6. Items 2, 9, 10 (isolated to workflows outside CI Gate's needs; useful but not merge-blockers)

## Explicitly out of scope for the umbrella (file separately if wanted)

- [ ] Adding new gates to `CI Gate`'s needs (Miri, Proptest, Security Audit, etc.) — that's a policy call, not a defect. Belongs in a separate design decision.
- [ ] Rewriting Playwright specs from scratch (item 9 documents four assertion-free specs; fixing them is per-spec work, not umbrella-shaped).
- [ ] The `Kani Proofs` failure rate tracked separately in #839 — that's a different lens (a gate that fails infrastructurally on 58% of pushes vs one that passes vacuously).

Three coordination items worth a maintainer call before AC is locked:

• The convention items 6 and 3 imply. Item 6 (unknown-field is Info) and item 3 (documented exclusion list is wrong) both argue for a repo-level rule: severity of a violation must match the review posture that catches it. If unknown-field bumps to Warning, validate --strict picks it up automatically once item 1 is fixed. That coupling is worth pinning in the AC so the two land as one narrative rather than sequential surprises.
• --new-since vs --fail-on warning. Item 1's AC includes both options because either would close the class — but they're operationally different (ratchet accepts today's 931 warnings; absolute demands zero). Ratchet is what the maintainer will actually merge; absolute is what rivet publishes to consumers. Pinning which one before implementation matters.
• The cargo deny check advisories gap. Item 8 overlaps with Extract rivet's lossless YAML stack as its own crate — the org has one, it is just not reachable #930's REQ-347 (advisories check not run in CI). Threading the two so the deny.toml cleanup ships in one review — not two overlapping ones — avoids merge friction.

Adjacent state worth flagging:

• Release path: the compliance action discards rivet validate's exit code, so a release can ship on a failing validate #953 (headline compliance-exit-code instance) has PR ci(compliance): gate the release on rivet validate's exit code (#953) #961 open addressing it — verified this run. rivet check verification-evidence returns ok=true on an empty scan — the anti-rot gate runs twice per PR and verifies nothing #954 (headline check verification-evidence ok-on-empty) landed in fix(check): verification-evidence returned ok on empty scan (#954) #959 (per the git log seen earlier this session).
• Audit umbrella: surfaces that disagree about the same fact (coverage %, validate verdict, export vs serve) #956 (equivalence umbrella) and Audit umbrella: documented behaviour the code does not have (schema link restrictions, export flags, doc recursion) #958 (documented-behaviour-not-in-code umbrella) are the sibling lenses. Cross-linking the three umbrellas so a reviewer sees the audit as one lens rather than three would help.

Process side-rail: https://pulseengine.eu/blog/ is up today (200 OK). Fetched at the start of this run — no post speaks to per-issue AC shape, CI-Gate-membership convention, or vacuous-green negative-control policy, so the kill-criterion posture used across adjacent triage threads remains the working default.

Not implemented this run — no PR opened; the umbrella-level AC + per-item AC on the follow-up issues are the gating inputs.

---

Generated by Claude Code