feat(certify): machine-checkable review certification (v0.4.0) - #14
Merged
Merged
Conversation
--certify LEDGER validates a review ledger against the input: exit 0 + record when complete, exit 3 naming blockers, exit 1 on bad ledger rows. Generation behavior and exit codes 0/1/2 unchanged. 15 new tests (99 total, green). Co-Authored-By: Claude <noreply@anthropic.com> Claude-Session: sess_c59696de-2f36-4d7c-99cd-ad4102b8cc4e
Ledger entries keyed by finding-scoped identity (interface rows producer->consumer:flows, pairs, boundary names, spans by source sha+range) with disposition/reviewer/date/fingerprint columns. Certification record emitted as <ledger>.cert.md binding input and report sha256, gate result, effective flags (replayed exactly on certify); source sha under --source. Drift re-opens exactly the touched rows; duplicate input identities exit 1. 30 certification tests (115 total, green); v0.3.0 example output byte-identical. Co-Authored-By: Claude <noreply@anthropic.com> Claude-Session: sess_c59696de-2f36-4d7c-99cd-ad4102b8cc4e
…s (U3) Steps 3-5 become the review loop: review = writing the identity-keyed ledger (refusal record doubles as worksheet), gaps may be parked open, step 5 ends with --certify exit 0. Deliverable = report, record, input, ledger together; uncertified report is a draft. Anti-delegation rule becomes separation of duties (independent review by default, model reviewer only via explicit pin; arXiv 2312.04134 kept as rationale). Optional experimental model-pin keys documented under metadata. Co-Authored-By: Claude <noreply@anthropic.com> Claude-Session: sess_c59696de-2f36-4d7c-99cd-ad4102b8cc4e
README: certify usage, four-file deliverable, exit-code table (exit 3; exit 2 shared with argparse documented), test count 115. RUNBOOK: certify procedure, drift and flag-mismatch playbooks, updated escalation and health checks. CHANGELOG 0.4.0. New certified example ledger + record (exit 0, one parked-gap advisory). Version 0.4.0. Co-Authored-By: Claude <noreply@anthropic.com> Claude-Session: sess_c59696de-2f36-4d7c-99cd-ad4102b8cc4e
…ss holes
P1: stamped record is now a verified replay anchor (input/source sha
mismatch on re-run = drift blocker, R5); citing input certified
without --source exits 1 (R3 spans); SKILL/docstring scope the
certification claim to the five ledger kinds (loops, self-deps,
rules audit are human-reviewed, not gated); dispositioned candidates
and boundary findings list as record advisories so what ships is
visible. P2/P3: record-section swallow dies; blank-flows identities
round-trip ('?' placeholder); delimiter component names rejected
under certify; five-file deliverable under --source documented;
record-lifecycle docs corrected; CRLF-safe writers; ledger rows
retire by deletion (no Status column); example regenerated from
repo-root invocation (no source citations). Suite 115 → 125.
Review run 20260930-030735-c0aa86ef: verdict Ready with fixes,
8/8 validator-confirmed findings addressed; #16 derive() deferred.
Co-Authored-By: Claude <noreply@anthropic.com>
Claude-Session: sess_c59696de-2f36-4d7c-99cd-ad4102b8cc4e
5 of 7 tasks
The script reconfigures stdout to UTF-8, but the test helpers decoded with the locale default; on a US-Windows runner (cp437) the em-dashes in certification record lines mojibake and four advisory/record assertions fail. Decode the captured streams explicitly. Co-Authored-By: Claude <noreply@anthropic.com> Claude-Session: sess_c59696de-2f36-4d7c-99cd-ad4102b8cc4e
SAP's CI greps ^(import|from) over the vendored script; a wrapped docstring sentence beginning 'from the input as…' read as a third-party import. Reworded; no code change. Co-Authored-By: Claude <noreply@anthropic.com> Claude-Session: sess_c59696de-2f36-4d7c-99cd-ad4102b8cc4e
macblackstuff
added a commit
to macblackstuff/system-adoption-pipeline
that referenced
this pull request
Sep 30, 2026
## What changes A pass-3 matrix now has to certify, not just run: the pipeline's done-check requires `--certify` to exit 0 against a review ledger, and the finished deliverable carries the report, its certification record, the input, the ledger (plus the source file under `--source`) — enough for any consumer to re-run certification. Release v0.4.0, re-vendoring interface-matrix v0.4.0. Lead-in: macblackstuff/interface-matrix#14 (the gate itself — this PR completes that program on the consumer side). ## Why The vendored `interface_matrix.py` gains the certification gate (`--certify LEDGER`: exit 0 + record when review is complete, exit 3 naming `drifted:`/`unreviewed:` blockers, exit 1 on bad rows/duplicate identities/flag mismatch) and its 125-test self-check; sha256 pins in `scripts/interface_matrix.UPSTREAM` updated in the same change (the CI pin job verifies the local pairs; the upstream `v0.4.0` tag is pushed when that PR merges). Pass-3 review becomes separation of duties — an independent reviewer of record recorded in the ledger, distinct from the input's drafter; a model reviewer only via the new optional, experimental model pins (decision/thinker/reviewer/judge under frontmatter `metadata:`, mapped to passes 3/4, 5, 6, 7). MATRIX-INPUT documents the ledger format and certification duties; the RUNBOOK gains the certify procedure, drift/flag-mismatch playbooks, and exit-3 escalation. Certification gates the five ledger kinds (candidates, gaps, boundary findings, unstated pairs, uncited spans); feedback loops, self-dependencies and the class-rule audit stay human-reviewed. ## Checklist - [x] Tests pass, with and without `-O` (vendored matrix self-check 125 + `check_plan` 65; both green) - [x] A test fails without this change, or the change is documentation-only (upstream's 10 regression tests pin the validated findings) - [x] `CHANGELOG.md` has a line under `Unreleased` (dated 0.4.0 heading per release convention) - [x] `SKILL.md` / `references/` updated if behaviour the agent sees changed - [x] Commits use a GitHub noreply email ## Validation - Vendored files byte-identical to upstream `v0.4.0` final head (sha256 `98e58564…` / `91581976…`, pins recomputed and matching). - CI pin-job logic replicated locally: 2 pins parsed, no shared basenames, pinned == actual. Stdlib allow-list extended with `hashlib|json` (CI + runbook hand-check in lockstep). - `skills-ref validate` → valid; `check_plan` suite unchanged-green (done-check semantics preserved where not certification-related). ## Unapplied review findings Carried from the upstream review (macblackstuff/interface-matrix#14, run `20260930-030735-c0aa86ef`) — both deferred there and applying here only with the next vendored sync: - [ ] P2 — `derive(built, rules)` extraction collapsing the report/certify derivation wiring. - [ ] P2 — certification-layer split into `scripts/certify.py` + `scripts/test_certification.py`; the vendored file layout and pins must move in the same change as upstream's. --- [](https://github.com/EveryInc/compound-engineering-plugin) --------- Co-authored-by: Claude <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes
A matrix report can now prove it was reviewed.
--certify LEDGERvalidates an identity-keyed review ledger against the input and writes a hash-bound certification record shipped with the report (report + record + input + ledger, plus the source file under--source) — so a consumer holding the files can re-run certification, and a report without its record is a draft. Reviews that skip the loop, on any model or harness, no longer produce an artifact that reads as finished.Release v0.4.0. Sibling PR follows in
system-adoption-pipelinere-vendoring this tag and requiring certification in its pass-3 done-check.Why
The review loop was prose-only ("human review is mandatory"), so one-shot runs shipped unresolved-gap reports indistinguishable from reviewed ones (SKILL.md:145-166 at base). The gate machine-checks what the prose asked for: every ledger finding — candidates, gaps, boundary findings, unstated pairs, uncited spans — dispositioned (open-parked gaps and accepted candidates ride as visible advisories), none drifted; the stamped record is a verified replay anchor (input/source sha drift re-opens the review; flags replay exactly); feedback loops, self-dependencies and the class-rule audit stay human-reviewed by design. Review becomes separation of duties: an independent reviewer of record, distinct from the input's drafter — a model reviewer only via the new optional, experimental model pins.
Design decisions
--certify.metadata:; the Python reads no configuration.Checklist
-O(125 tests; red-first: 14 then 22 then 11 observed failures before each implementation pass)CHANGELOG.mdhas a line underUnreleased(dated 0.4.0 heading per release convention)SKILL.md/references/updated if behaviour the agent sees changedValidation
python3 scripts/test_interface_matrix.py→ 125 tests OK; same under-O;skills-ref validate→ valid.examples/example-output.mdreproduced byte-identically; a 180-run differential fuzz over rules/classes/--sample/--sourcevariants found zero mismatches.examples/example-ledger.md+.cert.md— certify exits 0 with 4 visible advisories (accepted candidate, parked gap, 2 boundary acceptances); re-run is idempotent.Session-settled decisions carried from planning: enforcement via script gates AND prose (user-directed, over either alone); model-agnostic default with optional explicit pinning (user-directed, over built-in model requirements); both existing repos, this one as source of truth (user-directed, over a new repo).
Unapplied review findings
skills/interface-matrix/scripts/interface_matrix.py:984— Derivation wiring duplicated betweenreport()andcertify(): extractderive(built, rules)returning the shared bundle, called by both. Deferred: behavior-neutral refactor after a validated release.scripts/certify.py+scripts/test_certification.py; decide together with the next vendored sync in system-adoption-pipeline so pinned sha256s and docs move in one change.<ledger>.cert.md(consumer-side sha diffing before any re-run, documented in RUNBOOK);settle()returns a 6-tuple indexed[1]bycertify()untilderive()lands.Review run
20260930-030735-c0aa86ef; artifact retained on the author's machine.