Skip to content

feat(certify): machine-checkable review certification (v0.4.0) - #14

Merged
macblackstuff merged 7 commits into
mainfrom
feat/review-certification-gate
Sep 30, 2026
Merged

macblackstuff merged 7 commits into
mainfrom
feat/review-certification-gate

Conversation

@macblackstuff

Copy link
Copy Markdown
Owner

What changes

A matrix report can now prove it was reviewed. --certify LEDGER validates an identity-keyed review ledger against the input and writes a hash-bound certification record shipped with the report (report + record + input + ledger, plus the source file under --source) — so a consumer holding the files can re-run certification, and a report without its record is a draft. Reviews that skip the loop, on any model or harness, no longer produce an artifact that reads as finished.

Release v0.4.0. Sibling PR follows in system-adoption-pipeline re-vendoring this tag and requiring certification in its pass-3 done-check.

Why

The review loop was prose-only ("human review is mandatory"), so one-shot runs shipped unresolved-gap reports indistinguishable from reviewed ones (SKILL.md:145-166 at base). The gate machine-checks what the prose asked for: every ledger finding — candidates, gaps, boundary findings, unstated pairs, uncited spans — dispositioned (open-parked gaps and accepted candidates ride as visible advisories), none drifted; the stamped record is a verified replay anchor (input/source sha drift re-opens the review; flags replay exactly); feedback loops, self-dependencies and the class-rule audit stay human-reviewed by design. Review becomes separation of duties: an independent reviewer of record, distinct from the input's drafter — a model reviewer only via the new optional, experimental model pins.

Design decisions

  • Certification certifies review-completeness, not gap-zero — gaps may legitimately remain open; disposition, not closure, is what exit 0 demands.
  • Two-phase gate — generation behavior and exit codes 0/1/2 are byte-identical to v0.3.0 (the shipped example's output is unchanged); exit 3 (refusal, blockers named) and new exit-1 classes exist only under --certify.
  • The refusal record doubles as the review worksheet — certify once against a header-only ledger and every finding returns with its paste-able fingerprint.
  • Model pins are metadata-only, default off — experimental keys (decision/thinker/reviewer/judge) under frontmatter metadata:; the Python reads no configuration.

Checklist

  • Tests pass, with and without -O (125 tests; red-first: 14 then 22 then 11 observed failures before each implementation pass)
  • A test fails without this change, or the change is documentation-only (10 regression tests pin the validated review findings)
  • CHANGELOG.md has a line under Unreleased (dated 0.4.0 heading per release convention)
  • SKILL.md / references/ updated if behaviour the agent sees changed
  • Commits use a GitHub noreply email

Validation

  • python3 scripts/test_interface_matrix.py → 125 tests OK; same under -O; skills-ref validate → valid.
  • Generation regression: v0.3.0 examples/example-output.md reproduced byte-identically; a 180-run differential fuzz over rules/classes/--sample/--source variants found zero mismatches.
  • Certified example shipped: examples/example-ledger.md + .cert.md — certify exits 0 with 4 visible advisories (accepted candidate, parked gap, 2 boundary acceptances); re-run is idempotent.
  • Multi-agent code review (6 lenses + independent validation) run on this branch: verdict Ready with fixes; all 8 validator-confirmed findings — including the stamped-record-not-verified, citing-input-skips-spans, overclaim, and silent-candidate holes — fixed on-branch with tests.

Session-settled decisions carried from planning: enforcement via script gates AND prose (user-directed, over either alone); model-agnostic default with optional explicit pinning (user-directed, over built-in model requirements); both existing repos, this one as source of truth (user-directed, over a new repo).

Unapplied review findings

  • P2 — skills/interface-matrix/scripts/interface_matrix.py:984 — Derivation wiring duplicated between report() and certify(): extract derive(built, rules) returning the shared bundle, called by both. Deferred: behavior-neutral refactor after a validated release.
  • P2 (report-only) — certification layer (~320 lines) pushes the script past 1000 lines (897 → ~1200): extract scripts/certify.py + scripts/test_certification.py; decide together with the next vendored sync in system-adoption-pipeline so pinned sha256s and docs move in one change.
  • Residual risks accepted at review: the flags pin and record stamp live inside the trusted ledger (same trust domain as dispositions — the gate verifies presence, not honesty); each run overwrites <ledger>.cert.md (consumer-side sha diffing before any re-run, documented in RUNBOOK); settle() returns a 6-tuple indexed [1] by certify() until derive() lands.

Review run 20260930-030735-c0aa86ef; artifact retained on the author's machine.


Compound Engineering

macblackstuff and others added 5 commits September 30, 2026 01:32
--certify LEDGER validates a review ledger against the input:
exit 0 + record when complete, exit 3 naming blockers, exit 1 on
bad ledger rows. Generation behavior and exit codes 0/1/2 unchanged.
15 new tests (99 total, green).

Co-Authored-By: Claude <noreply@anthropic.com>
Claude-Session: sess_c59696de-2f36-4d7c-99cd-ad4102b8cc4e
Ledger entries keyed by finding-scoped identity (interface rows
producer->consumer:flows, pairs, boundary names, spans by source
sha+range) with disposition/reviewer/date/fingerprint columns.
Certification record emitted as <ledger>.cert.md binding input and
report sha256, gate result, effective flags (replayed exactly on
certify); source sha under --source. Drift re-opens exactly the
touched rows; duplicate input identities exit 1. 30 certification
tests (115 total, green); v0.3.0 example output byte-identical.

Co-Authored-By: Claude <noreply@anthropic.com>
Claude-Session: sess_c59696de-2f36-4d7c-99cd-ad4102b8cc4e
…s (U3)

Steps 3-5 become the review loop: review = writing the identity-keyed
ledger (refusal record doubles as worksheet), gaps may be parked open,
step 5 ends with --certify exit 0. Deliverable = report, record, input,
ledger together; uncertified report is a draft. Anti-delegation rule
becomes separation of duties (independent review by default, model
reviewer only via explicit pin; arXiv 2312.04134 kept as rationale).
Optional experimental model-pin keys documented under metadata.

Co-Authored-By: Claude <noreply@anthropic.com>
Claude-Session: sess_c59696de-2f36-4d7c-99cd-ad4102b8cc4e
README: certify usage, four-file deliverable, exit-code table (exit 3;
exit 2 shared with argparse documented), test count 115. RUNBOOK:
certify procedure, drift and flag-mismatch playbooks, updated
escalation and health checks. CHANGELOG 0.4.0. New certified example
ledger + record (exit 0, one parked-gap advisory). Version 0.4.0.

Co-Authored-By: Claude <noreply@anthropic.com>
Claude-Session: sess_c59696de-2f36-4d7c-99cd-ad4102b8cc4e
…ss holes

P1: stamped record is now a verified replay anchor (input/source sha
mismatch on re-run = drift blocker, R5); citing input certified
without --source exits 1 (R3 spans); SKILL/docstring scope the
certification claim to the five ledger kinds (loops, self-deps,
rules audit are human-reviewed, not gated); dispositioned candidates
and boundary findings list as record advisories so what ships is
visible. P2/P3: record-section swallow dies; blank-flows identities
round-trip ('?' placeholder); delimiter component names rejected
under certify; five-file deliverable under --source documented;
record-lifecycle docs corrected; CRLF-safe writers; ledger rows
retire by deletion (no Status column); example regenerated from
repo-root invocation (no source citations). Suite 115 → 125.

Review run 20260930-030735-c0aa86ef: verdict Ready with fixes,
8/8 validator-confirmed findings addressed; #16 derive() deferred.

Co-Authored-By: Claude <noreply@anthropic.com>
Claude-Session: sess_c59696de-2f36-4d7c-99cd-ad4102b8cc4e
macblackstuff and others added 2 commits September 30, 2026 04:48
The script reconfigures stdout to UTF-8, but the test helpers decoded
with the locale default; on a US-Windows runner (cp437) the em-dashes
in certification record lines mojibake and four advisory/record
assertions fail. Decode the captured streams explicitly.

Co-Authored-By: Claude <noreply@anthropic.com>
Claude-Session: sess_c59696de-2f36-4d7c-99cd-ad4102b8cc4e
SAP's CI greps ^(import|from) over the vendored script; a wrapped
docstring sentence beginning 'from the input as…' read as a
third-party import. Reworded; no code change.

Co-Authored-By: Claude <noreply@anthropic.com>
Claude-Session: sess_c59696de-2f36-4d7c-99cd-ad4102b8cc4e
@macblackstuff
macblackstuff merged commit 99e4362 into main Sep 30, 2026
8 checks passed
@macblackstuff
macblackstuff deleted the feat/review-certification-gate branch September 30, 2026 13:44
macblackstuff added a commit to macblackstuff/system-adoption-pipeline that referenced this pull request Sep 30, 2026
## What changes

A pass-3 matrix now has to certify, not just run: the pipeline's
done-check requires `--certify` to exit 0 against a review ledger, and
the finished deliverable carries the report, its certification record,
the input, the ledger (plus the source file under `--source`) — enough
for any consumer to re-run certification. Release v0.4.0, re-vendoring
interface-matrix v0.4.0.

Lead-in: macblackstuff/interface-matrix#14 (the gate itself — this PR
completes that program on the consumer side).

## Why

The vendored `interface_matrix.py` gains the certification gate
(`--certify LEDGER`: exit 0 + record when review is complete, exit 3
naming `drifted:`/`unreviewed:` blockers, exit 1 on bad rows/duplicate
identities/flag mismatch) and its 125-test self-check; sha256 pins in
`scripts/interface_matrix.UPSTREAM` updated in the same change (the CI
pin job verifies the local pairs; the upstream `v0.4.0` tag is pushed
when that PR merges). Pass-3 review becomes separation of duties — an
independent reviewer of record recorded in the ledger, distinct from the
input's drafter; a model reviewer only via the new optional,
experimental model pins (decision/thinker/reviewer/judge under
frontmatter `metadata:`, mapped to passes 3/4, 5, 6, 7). MATRIX-INPUT
documents the ledger format and certification duties; the RUNBOOK gains
the certify procedure, drift/flag-mismatch playbooks, and exit-3
escalation. Certification gates the five ledger kinds (candidates, gaps,
boundary findings, unstated pairs, uncited spans); feedback loops,
self-dependencies and the class-rule audit stay human-reviewed.

## Checklist

- [x] Tests pass, with and without `-O` (vendored matrix self-check 125
+ `check_plan` 65; both green)
- [x] A test fails without this change, or the change is
documentation-only (upstream's 10 regression tests pin the validated
findings)
- [x] `CHANGELOG.md` has a line under `Unreleased` (dated 0.4.0 heading
per release convention)
- [x] `SKILL.md` / `references/` updated if behaviour the agent sees
changed
- [x] Commits use a GitHub noreply email

## Validation

- Vendored files byte-identical to upstream `v0.4.0` final head (sha256
`98e58564…` / `91581976…`, pins recomputed and matching).
- CI pin-job logic replicated locally: 2 pins parsed, no shared
basenames, pinned == actual. Stdlib allow-list extended with
`hashlib|json` (CI + runbook hand-check in lockstep).
- `skills-ref validate` → valid; `check_plan` suite unchanged-green
(done-check semantics preserved where not certification-related).

## Unapplied review findings

Carried from the upstream review (macblackstuff/interface-matrix#14, run
`20260930-030735-c0aa86ef`) — both deferred there and applying here only
with the next vendored sync:

- [ ] P2 — `derive(built, rules)` extraction collapsing the
report/certify derivation wiring.
- [ ] P2 — certification-layer split into `scripts/certify.py` +
`scripts/test_certification.py`; the vendored file layout and pins must
move in the same change as upstream's.

---

[![Compound
Engineering](https://img.shields.io/badge/Built_with-Compound_Engineering-6366f1)](https://github.com/EveryInc/compound-engineering-plugin)

---------

Co-authored-by: Claude <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant