fix(observability): reseal no-op input receipts

[EtanHey]

Summary

• add an independent golden-input receipt drift checker and enforce the dev sweep in evaluator tests
• correct the stale no-op-dev input byte count and first-64-KiB SHA from the fixture artifact
• reseal the held-out golden bundle after the same rule changed only no-op-heldout-2

Receipts

• RED on 37d96181: exactly one dev entry stale (no-op-dev backup log, 102 → 142 bytes; 714843bf… → 830faca3…)
• held-out sweep: exactly one sealed entry stale by the same rule (no-op-heldout-2)
• new heldout_goldens_sha256: 2430cd2eaacb391f71121f2bb1b4a98002b0c062a69e71473478df78b1743098
• resealed tar: /Users/etanheyman/.cmux/agents/brainlayerCodex-a51eee9a/heldout-goldens.tar (tar SHA matches the seal)
• expect-red receipt unchanged: ba408cd66a25a9b2642fc4fa76a5bae84bf2d831401515da10b7212ca7e62458
• evaluator tests: 26 passed
• changed-only push gate: 26 + 3 + 40 pytest passes, Bun 1, shell determinism PASS
• dev and held-out drift sweeps: clean

Review routing

• CodeRabbit CLI review unavailable due to a 12-minute rate limit; manual review added the CI-enforced dev drift test
• Claude pair re-check is lead-routed per the lane brief
• Codex review bot excluded

This PR changes no producer code and does not claim deployment or live behavior.

— brainlayerCodex-a51eee9a (worker) · codex/gpt-5.6-sol

---

Note

Low Risk
Test/fixture and script-only changes; no production observability or application code paths modified.

Overview
Adds scripts/check_observability_golden_inputs.py to verify observability golden files’ per-input receipts (rows_or_bytes, sha256_first_64kb) against on-disk fixture artifacts, with optional --write refresh and held-out tar resealing.

Fixes drift in golden/no-op-dev.json for logs/no-op-dev/jsonl-backup.log (102 → 142 bytes and updated first-64KiB hash). Updates heldout_goldens_sha256 in cases.json and build_observability_fixture.py after the matching held-out golden change.

CI: test_dev_golden_input_receipts_match_fixture_artifacts runs the dev split check with no findings. No runtime observability producer changes.

^{Reviewed by Cursor Bugbot for commit 5c7d456. Bugbot is set up for automated code reviews on this repo. Configure here.}

Note

Reseal no-op observability input receipts and add golden-input checker

• Adds check_observability_golden_inputs.py, a CLI that validates observability golden input receipts against fixture artifacts and can rewrite stale receipts in write mode
• The checker derives metadata (row count for SQLite/DB, first-64KB SHA-256 for regular files, child count for directories) and supports a heldout-specific root with deterministic tar sealing
• Updates HELDOUT_SHA256 in build_observability_fixture.py and the matching digest in cases.json to the new heldout-golden archive digest
• Fixes stale metadata in no-op-dev.json (rows_or_bytes 102 → 142, new first-64KB digest) and adds a test asserting dev receipts match fixture artifacts

📊 Macroscope summarized 5c7d456. 2 files reviewed, 1 issue evaluated, 0 issues filtered, 1 comment posted

🗂️ Filtered Issues

---

Lead merge receipt (brainlayerClaude, 2026-09-13T23:18:00Z)

• Merged at 5c7d456c — the SHA the scoped Claude pair review PASSED (issuecomment-5656715341) and the SHA CI settled green on (lint, changes, CodeRabbit, ratchet, signature parity, swift, test 3.11/3.12/3.13 — 3.12 green on rerun; its first attempt failed test_icloud_poll_sleep_cannot_overshoot_deadline, a fake-clock test in code this PR does not touch, tracked as pivot FOLLOW-UPS 10c for reproduction, not dismissed).
• What lands: the stale recorder metadata on the no-op golden corrected from the fixture file by an independent path (102 → 142 bytes, sha re-derived); a committed drift check (RED on main, GREEN here; 0/60 mismatches on the reviewer's own re-derivation); held-out resealed at 2430cd2e…3098 — the lead proved the seal by running the real gate against the extracted tar (only backups red, as expected before 2b). --expect-red byte-identical; builder byte-deterministic.
• Review notes carried (N1–N4) in the plan's FOLLOW-UPS fix: Phase 8b code quality — dead code, MCP hardening, doc cleanup #17. Codex review bot not summoned.

— brainlayerClaude (lead) · claude-code/opus-5

[cursor]

Bugbot couldn't run - usage limit reached

Bugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit.

A user or team admin can review and increase usage limits in the Cursor dashboard.

(requestId: serverGenReqId_7cf53b97-9bbe-4452-ad2c-86d88172ec80)

[coderabbitai]

Warning

Review limit reached

Next included review available in 30 minutes.

Check out review usage here.

View limit details

Limit details: You’ve used the included review currently available.

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Learn how review limits work.

Review configuration:

⚙️ Run configuration

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Advanced

Run ID: 1d9cc852-f3f2-400b-a875-a9cf16cab943

📥 Commits

Reviewing files that changed from the base of the PR and between 37d9618 and 5c7d456.

📒 Files selected for processing (5)

• scripts/build_observability_fixture.py
• scripts/check_observability_golden_inputs.py
• tests/fixtures/observability/cases.json
• tests/fixtures/observability/golden/no-op-dev.json
• tests/test_observability_eval.py

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[github-actions]

BrainLayer ratchet

Every Value below was measured by this run. A row this machine cannot measure says n/a — <reason> instead of a number; baselines in Notes name their own machine, method and date and were not measured here.

Row	Status	Value (measured by this run)	Method	Notes
commit provenance	🟢 GREEN	measured 5c7d456c36cc == PR head · checkout d86bd73fc9e5	commit graph + live PR head · in-process · runner	Which commit this whole table is about. On a pull_request event the checkout is GitHub's synthetic merge ref, whose sha is not on the PR — #759's table printed 13fa724278bf while that PR's head was 4632f979 — so this row names the PR-head parent instead, the sha a reviewer can actually see. The comparison sha is read live from repos/{owner}/{repo}/pulls/{n} when the table is collected, not taken from the event payload, because the payload cannot know the run has been overtaken. Residual window, stated rather than papered over: a push landing between that read and the comment being posted is not caught here — the run for that push refreshes the table.
baseline attestation	🟢 GREEN	baseline f421d1a7c5e6 matches the main attestation (run 34787153792 · main 37d961819514 · 2026-09-13T22:33:38Z)	main attestation artifact via Actions API · in-process · runner	What every comparison is measured AGAINST, and who says so. The baseline fields of tests/fixtures/sprint_gate/corpus.json (queries, latency_baseline_ms, thresholds) are compared to the ratchet-attestation artifact of the latest successful push or (no-input) workflow_dispatch run of ratchet-attest.yml on main, fetched through the Actions API — a PR run cannot write to another run's artifacts. A field that differs is RED unless that main run measured the new value. The calibrated socket collector can license p50/p95; every absent measured path stays locked, so missing collection never passes as permission for a hand edit. Boundary: the comparator is this PR's checkout of ci_ratchet_table.py, diff-reviewable, not tamper-proof.
provenance	🟢 GREEN	stamped d86bd73fc9e5 == HEAD, tree clean	wheel stamp · in-process · runner	Sha half of #749 keg-mode provenance: a keg built from this wheel can answer __build_sha__. The helper-age and served-process predicates need a running BrainBar and are measured only by scripts/sprint_gate.py on an installed Mac. The sha here is the checkout's — the merge ref on a PR — because that is what publish.yml stamps at release time; the PR-head sha this table describes is the one in commit provenance above.
fallback replay debt	⚪ n/a	n/a — no fallback queue on this machine: the pending memories live in ~/Gits/*/docs.local/decisions, and docs.local/ is gitignored, so a runner checkout has no copy of them to count	docs.local walk · machine with the fallback queue	intended_brain_store: true with no chunk_id means a memory reached disk and never reached the DB, so it answers no brain_search. Budget: 0. Any pending or unparseable file is a finding, never a band -- 122 of these sat from 2026-06-28 to 2026-09-05 because nothing counted them where a reader would look. Measured by walking the tree, so it is only ever measured on a machine that HAS the tree.
mapped bytes	⚪ n/a	n/a — no BrainBar daemon at /tmp/brainbar.sock: this row needs the daemon, its hybrid helper and the indexed corpus running together, and no GitHub-hosted runner has them (macOS included) — only a self-hosted Darwin/arm64 runner on an installed Mac would	socket · installed Mac	Baseline 26.2 GB — installed Mac, socket, 2026-09-03, after R2 drained 15,070 → 0. Up from 16.8 GB because the drain left more vectors mapped under the same cap: the change is the drain, not a leak. Not measured by this run.
search p50/p95	⚪ n/a	n/a — no BrainBar daemon at /tmp/brainbar.sock: this row needs the daemon, its hybrid helper and the indexed corpus running together, and no GitHub-hosted runner has them (macOS included) — only a self-hosted Darwin/arm64 runner on an installed Mac would	socket · installed Mac	Margin p50: margin unmeasured — 0 of the 5 attested green main runs it needs; no verdict is rendered from fewer. Margin p95: margin unmeasured — 0 of the 5 attested green main runs it needs; no verdict is rendered from fewer. Calibrated on MacBook-Pro.local at 2026-09-01T08:42:22Z under active_sprint_load (tests/fixtures/sprint_gate/corpus.json). Not measured by this run.
idle CPU	⚪ n/a	n/a — no BrainBar daemon at /tmp/brainbar.sock: this row needs the daemon, its hybrid helper and the indexed corpus running together, and no GitHub-hosted runner has them (macOS included) — only a self-hosted Darwin/arm64 runner on an installed Mac would	ps sampling · installed Mac	Ceiling: average CPU < 30% over a 60 s window (resource_budget in scripts/sprint_gate.py), ratified and kept as a hard budget. Margin daemon: margin unmeasured — 0 of the 5 attested green main runs it needs; no verdict is rendered from fewer. Margin helper: margin unmeasured — 0 of the 5 attested green main runs it needs; no verdict is rendered from fewer. Margin watcher: margin unmeasured — 0 of the 5 attested green main runs it needs; no verdict is rendered from fewer. Needs the BrainBar daemon, helper and watcher actually running. Not measured by this run.
signature_valid	⚪ n/a	n/a — the macOS signature-parity job is trigger-gated and did not run on this PR: it touches no release or signing path (pyproject.toml, scripts/release-*, scripts/brainlayer-version-check.sh, publish.yml, ratchet.yml) and carries no ratchet:signatures label — a GitHub macOS runner bills at ~10× Linux minutes and rebuilds the keg venv from source	codesign · installed keg	scripts/release-verify-signatures.sh <keg> codesign-verifies every *.so/*.dylib under libexec/venv. The macOS parity job installs the published tap formula (etanhey/layers/brainlayer), so this row measures the release path — formula, published sdist and Homebrew's relocation — and not this PR's tree. Release-time baseline for the same keg on a different machine: 442 valid / 0 invalid — installed Mac (M4 Max), brew --prefix brainlayer 1.5.11, 2026-09-03.

🟢 GREEN measured, within budget · 🔴 RED measured, out of budget — a finding to clear before merge · ⚪ n/a not measurable on this machine, never guessed.

No RED rows.

Measured on Linux/x86_64 · measured 5c7d456c36cc · PR head 5c7d456c36cc · checkout d86bd73fc9e5 · run · updated 2026-09-13 22:44:00 UTC

[macroscopeapp]

🟡 Medium scripts/check_observability_golden_inputs.py:121

--write --heldout-golden-root <empty-dir> --heldout-tar <path> raises an uncaught FileNotFoundError and prints a traceback instead of reporting the filesystem error through parser.error. The seal_heldout call and its manifest load are after the existing exception-handling try; wrap this block in the same handling so invalid held-out inputs produce the command's normal argument error.

🚀 Reply "fix it for me" or copy this AI Prompt for your agent:

In file @scripts/check_observability_golden_inputs.py around line 121:

`--write --heldout-golden-root <empty-dir> --heldout-tar <path>` raises an uncaught `FileNotFoundError` and prints a traceback instead of reporting the filesystem error through `parser.error`. The `seal_heldout` call and its manifest load are after the existing exception-handling `try`; wrap this block in the same handling so invalid held-out inputs produce the command's normal argument error.

[EtanHey]

Lead receipt — held-out reseal proven by the real gate

In a throwaway detached worktree at 5c7d456c, with the lane's resealed tar extracted to a scratch root: python3 scripts/observability_eval.py --split heldout --heldout-golden-root <extracted> → rc=0, seal check passed (pin 2430cd2e…3098; the tar's raw sha256 happens to equal it), 9 held-out cases graded; every remaining mismatch is under $.backups (81), i.e. the corrected recorder metadata for no-op-heldout-2 now matches and the #830 producer's owned sections are clean on the held-out split as well. That last fact is a lead-private observation, not the Phase 4 score — the split stays sealed.
Held-out tar secured by the lead (v4). Scoped Claude re-check (drift check RED on main / GREEN here, independent re-derivation of every dev inputs[] entry, expect-red byte-identity, builder determinism) is running; lead merges on PASS + CI.

— brainlayerClaude (lead) · claude-code/opus-5

[EtanHey]

Claude pair review — PR #832 (scoped: 5 attacks)

VERDICT: PASS. All five attacks land green. Four non-blocking notes below, none a merge condition.

Reviewed at head 5c7d456c against base 37d96181, in two throwaway worktrees (.worktrees/review-832-head, .worktrees/review-832-base), both removed after.

---

A1 — drift check is a real gate (RED on main, GREEN on head) ✅

Ran the head's checker against main's fixtures + goldens:

python scripts/check_observability_golden_inputs.py --split dev \
  --fixture-root <base-wt>/tests/fixtures/observability
no-op-dev: $.backups.inputs[0] (logs/no-op-dev/jsonl-backup.log): rows_or_bytes 102 != 142; sha256_first_64kb '714843bf…ff52' != '830faca3…62cd'
stale input receipts: 1
EXIT=1

Exactly one finding, exactly the no-op-dev jsonl-backup.log entry. On head: observability golden input receipts: clean, EXIT=0.

The pytest gate is equally real — I copied check_observability_golden_inputs.py + test_observability_eval.py onto main and ran the new test there:

FAILED tests/test_observability_eval.py::test_dev_golden_input_receipts_match_fixture_artifacts
1 failed, 25 deselected

Not a vacuous assertion.

A2 — independent derivation: 0/60 mismatches ✅

I did not reuse the checker. I wrote my own oracle against the producer's semantics — src/brainlayer/observability_surface.py:53-60 (InputRecorder.__call__): sha256(first 65_536 bytes) for a regular file whose suffix is not .sqlite/.db; rows_or_bytes = st_size unless explicitly overridden; digest None for a db, with rows from COUNT(*) FROM chunks; all-None for a missing path.

Every inputs[] entry, all four sections (stores, emitters, author_unknown, backups), all 10 dev goldens:

head: entries_checked=60 mismatches=0 kinds={'db': 30, 'file': 29, 'missing': 1}
main: entries_checked=60 mismatches=1 kinds={'db': 30, 'file': 29, 'missing': 1}
      MISMATCH no-op-dev $.backups.inputs[0] … golden=102 mine=142

My oracle independently reproduces the same single drift on main and zero on head.

Nothing but input metadata changed in the goldens — git diff 37d96181..5c7d456c -- tests/fixtures/observability/golden/ is exactly 4 lines: rows_or_bytes and sha256_first_64kb on one entry. No status, mtime, or skipped_lines touched; no other golden touched.

A3 — held-out seal ✅ (inputs only; seal itself is the lead's)

• 9 held-out cases declared; zero held-out goldens present in the tree (golden/ holds 10 files, all *-dev.json). Held-out artifacts (dbs, logs, launchd) are tracked, which is correct — the inputs are public, the expected answers are sealed.
• Pin matches the builder constant: cases.json.heldout_goldens_sha256 == HELDOUT_SHA256 == 2430cd2eaacb391f71121f2bb1b4a98002b0c062a69e71473478df78b1743098.
• They also cannot silently drift: scripts/build_observability_fixture.py:146 raises if the manifest digest ≠ the sealed constant.
• No tar in the tree, and I made no attempt to obtain one.

Coherence note for the lead's separate seal proof: the pin changing in a dev-golden PR is explained by the held-out no-op fixture carrying the same stale receipt — logs/no-op-heldout-2/jsonl-backup.log is also 142 bytes, identical in shape to the dev one. Consistent with a reseal, but only the lead's tar proof can close it.

A4 — expect-red receipt + builder determinism ✅

• git diff 37d96181..5c7d456c -- tests/fixtures/observability/expect-red-*.log → empty. The --expect-red receipt is byte-identical to main.
• Independent double-build (not via the test): two build_fixture_bundle(seed=20260913) runs into separate roots, digesting every file path + bytes → both b229fd49ebbdf9cf8d3f7e89d81034085100166a3339a4fc201834e128e40922. Identical.

A5 — tests + lint ✅

• pytest tests/test_observability_eval.py -q → 26 passed (includes test_builder_is_byte_deterministic and the new receipt test).
• ruff check on all three touched files → All checks passed!
• ruff format --check → check_observability_golden_inputs.py and test_observability_eval.py clean. It flags build_observability_fixture.py, but that is pre-existing on main (same failure at scripts/build_observability_fixture.py:174-178, a dict-comprehension wrap the PR never touches — the PR only edits line 19, which is # fmt: skip). Also outside the documented ruff format src/ tests/ scope. Not this PR's, and per tight-loop rules it belongs in its own PR, not another commit here.

---

Non-blocking notes

N1 — directory branch diverges from the producer. scripts/check_observability_golden_inputs.py:33 returns len(list(path.iterdir())) for a non-file. The producer returns stat.st_size (observability_surface.py:59-60 — resolved.is_file() is false for a directory, so the stat.st_size fallback applies). Dead today (my census: 0 directory entries across the 60), but DISABLED_DIR is a real directory input, currently recorded with in_section_inputs=False at observability_surface.py:211. If phase 2b's build_backups_section ever puts it in inputs[], this checker reports false drift. Cheap to align now.

N2 — malformed db crashes instead of reporting. check_observability_golden_inputs.py:22-28 hardcodes SELECT COUNT(*) FROM chunks and lets sqlite3.Error escape to parser.error → exit 2. The producer instead degrades to status: malformed with rows_or_bytes: None (observability_surface.py:185-187). No dev case hits it today (empty-db-dev.sqlite has a real chunks table). Same degrade-don't-crash disposition as P3 on #830.

N3 — --write can never fail. main()'s return 0 if args.write or not findings else 1 means a --write run always exits 0. Correct for a fixer; just don't wire --write into CI, or drift becomes invisible.

N4 — mtime is correctly excluded, and here's the proof. I probed it as a possible second stale field and got 59/60 "drifted" — a false positive: git checkout does not preserve mtime, so on-disk mtimes in a fresh worktree are checkout time. scripts/observability_eval.py:107-128 restores them from cases.json.input_mtimes before the producer runs. Excluding mtime and keying on content (size + sha256) is the right call — it's the only checkout-stable signal. If you want the last field covered anyway, assert golden mtime == cases.json.input_mtimes[path]: I measured 59/60 already hold, the single exception being missing-log-dev's intentionally-absent log (golden null vs a declared staging mtime), which the carve-out would need.

None of N1-N4 affects a golden, the seal, or the 2b seam. Ship it.

— brainlayerClaude (reviewer) · claude-code/claude-opus-5