refactor(audit)!: replace chained journals with shared JSONL streams - #1560
Conversation
|
@codex review |
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 9467174eed
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
4820bb3 to
109975d
Compare
9467174 to
159b6ce
Compare
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 159b6ceaae
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
registry-stack/crates/registry-evidence/src/audit.rs
Lines 1223 to 1227 in eb751cd
When operation A has an access-attempt entry but no terminal entry and a later concurrent operation B completes, last_operation points to B and this branch returns B's completed view while silently leaving A in self.pending. Consequently evidencectl audit show --last-operation succeeds even though the retained stream contains an unmatched earlier access attempt, contrary to its completeness check; after selecting the last operation, reject any remaining pending operations other than the intentionally displayed access-only last operation.
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
|
Codex review-body finding (Evidence |
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: a9ad47ca18
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
a9ad47c to
503b8cf
Compare
There was a problem hiding this comment.
💡 Codex Review
registry-stack/crates/registry-evidence/src/audit.rs
Lines 1159 to 1164 in 503b8cf
When a local Evidence operation uses a multi-stage acquisition such as search-then-fetch, the runtime appends one AccessAttempt for each physical source call under the same operation correlation. The collector treats the second legitimate access entry as reuse of a pending operation and rejects the entire retained stream, so evidencectl audit show --last-operation becomes unusable after such a request. Retain all access entries for an operation and validate them against its single terminal entry instead of requiring exactly one.
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
|
Codex review-body finding (multi-stage Evidence operations break |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 58537313a9
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: e464b9d80f
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
e464b9d to
4a688ff
Compare
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: f40f30e46e
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: e94b878564
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 58a897b92c
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
…concerns by its pseudonym Signed-off-by: Jeremi Joslin <jeremi@joslin.fr>
… lost before recording it Signed-off-by: Jeremi Joslin <jeremi@joslin.fr>
Folds in the follow-ups: - fix(audit): state the parent-segment refusal in the runtime schemas Signed-off-by: Jeremi Joslin <jeremi@joslin.fr>
A replay whose commit acknowledgement was lost and whose reset cannot be read back may have committed, so recording it as refused and dead-lettered could contradict the database. It is now answered with `replay_unfinished` under the pending disposition, and `replay_refused` stays for a reset that read back as rolled back or changed no row. Folds in the follow-ups: - docs(hooks): state the audit pairing contract as at least one answer Signed-off-by: Jeremi Joslin <jeremi@joslin.fr>
…nfinished A records swap whose commit acknowledgment was lost may have taken effect, so `records apply` recorded a refusal the database could contradict. The store now reads the swap's commit back like a capacity commit, and a swap whose outcome cannot be read is answered `unfinished` with reason `records.replace-unacknowledged`. Signed-off-by: Jeremi Joslin <jeremi@joslin.fr>
A client disconnect can drop a Relay handler after its attempt entry was accepted and before its terminal entry, leaving the request unpaired. The attempt now begins a platform audit request whose guard the handler holds until its terminal returns; a dropped guard writes a terminal record with the added unfinished outcome. Signed-off-by: Jeremi Joslin <jeremi@joslin.fr>
`Path` drops a trailing `/` or `.`, so `/a/b/` passed the preflight as the file `b` in `/a` and then failed to open at startup. The file destination, Relay's shape check, and the runtime schema patterns now require the path to end in a file name. Folds in the follow-ups: - chore(identifiers): refresh the catalog digests of the regenerated runtime schemas Signed-off-by: Jeremi Joslin <jeremi@joslin.fr>
A worker aborted after the attempt entry was accepted and before the lease committed rolled the lease back, leaving that attempt with no lease for expiry recovery to answer. The claim now runs to its commit in a task of its own, as the replay already does. Signed-off-by: Jeremi Joslin <jeremi@joslin.fr>
…s commit Reading the transaction id for a lost-acknowledgment read-back put a database round trip between the grant expiry re-check and COMMIT, so a grant could lapse in that gap. The re-check now runs inside commit_capacity, after the id is read and immediately before COMMIT. Signed-off-by: Jeremi Joslin <jeremi@joslin.fr>
The retention page, the platform changelog, and the Render README said only a crash could leave a request entry unanswered. An exit or runtime shutdown with a write in flight can too, and a delivery attempt whose terminal commit fails waits for lease-expiry recovery. State that a request may be answered more than once, that an unfinished outcome also covers refused and no-op operator commands, and that an ingestion commit error is answered unfinished. Signed-off-by: Jeremi Joslin <jeremi@joslin.fr>
…eled A worker aborted after its terminal disposition committed and before the terminal entry was accepted left a delivered or dead-lettered row with no entry, and a terminal row is never reaped, so nothing answered that attempt. Finalize now runs to its terminal entry in a task of its own, as the claim and the replay already do. Folds in the follow-ups: - docs(hooks): scope the detached-task answer promise to a running process Signed-off-by: Jeremi Joslin <jeremi@joslin.fr>
…esponse An appended response could claim a request whose handle was already writing its own response, so that request ended with two responses and another open request under the same correlation was answered unfinished. Folds in the follow-ups: - docs(audit): describe the detached queue's lock wait without poisoning Signed-off-by: Jeremi Joslin <jeremi@joslin.fr>
…tes in Creating the missing audit directory needs write and search permission on its nearest existing ancestor, so the preflight probes both instead of write alone. Signed-off-by: Jeremi Joslin <jeremi@joslin.fr>
A name the filesystem accepts for the active file could leave no room for the sealed segment's sequence suffix, so the first rotation failed and stopped the writer for good. Startup refuses such a name, and a process role's sibling name, instead. Signed-off-by: Jeremi Joslin <jeremi@joslin.fr>
…r creates A missing audit directory was created without syncing its parent, so a power loss could remove the directory together with entries already reported durable. The preflight also requires read permission on the existing ancestor, since the writer opens it to sync. Signed-off-by: Jeremi Joslin <jeremi@joslin.fr>
…e a shipper - Validate an audit directory level won by a concurrent creator. - Refuse to create a missing audit directory only below a world-writable, non-sticky ancestor, and say to create it owned by the service user, 0700. - Record the next sealed sequence in .seq at open and before each rotation, so numbering continues after a shipper removes every sealed segment. - Keep retention and process roles out of another stream's active file. - Apply retention oldest first, stop at the first unexpired segment, and log a retention failure instead of refusing open or rotation. - Carry on when the sealed segment is removed right after rotation. - Sync the directory on every open, not only when a name was created. - Drop ctime from the active-file identity check; length, mtime and inode still catch truncation, append, in-place rewrite and replacement. - Add AuditError::operator_description and pass it through Evidence, Relay, Casework and Scheduling startup refusals and logs. - Refuse a multi-stage Evidence operation that retention cut short. - Record a delivery terminal whose commit cannot be read back with the Unknown disposition instead of a guessed retry state. - Describe the writer contract in products/platform/AGENTS.md. Signed-off-by: Jeremi Joslin <jeremi@joslin.fr>
…at startup - Answer a committed ingestion transition unfinished when its response entry fails. - Record an attachment verdict only after its commit resolves. - Keep the request scrub from answering the erase-history lifecycle. - Answer an unproven mutation commit unfinished, never refused. - Accept the unknown webhook audit disposition for an interrupted terminal. - Report an audit destination refusal as StartupError::AuditDestination with the writer's recovery text, in the startup log and in bregctl doctor. Signed-off-by: Jeremi Joslin <jeremi@joslin.fr>
…d ones - Leave a trace of background work whose commit is unresolved. - Settle a commit whose caller is canceled after COMMIT is sent. - Pass the audit writer's recovery text through the startup refusal. Signed-off-by: Jeremi Joslin <jeremi@joslin.fr>
…come - Describe Render's server-drawn audit correlation in the retention table. - State how Scheduling's audit-outbox retirement is covered outside the rehearsal. - Document ordered, non-fatal retention, the .seq record, the narrowed ancestor rule, and the recovery text operators see at startup. - Replace the lease-expiry wording with worker_interrupted and unknown. Signed-off-by: Jeremi Joslin <jeremi@joslin.fr>
A child left with umask 0777 writes its coverage profile at exit with mode 000, and llvm-profdata then crashes merging it, failing platform line coverage in the merge queue. Signed-off-by: Jeremi Joslin <jeremi@joslin.fr>
404b7f4 to
db06d67
Compare
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: db06d67131
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| let writer = AuditWriter::open(destination) | ||
| .await | ||
| .map_err(|_| RegistryAuditError::Unavailable)?; |
There was a problem hiding this comment.
Preserve the companion audit recovery reason
When another bregctl maintenance command already holds the companion lock, AuditWriter::open returns a role-specific SinkLocked recovery message telling the operator to wait, but this mapping discards it. The checked history-erasure, rebaseline, reconciliation, and field-encryption callers then report only a generic audit failure—often advising the operator to check the path and permissions—so a valid deployment cannot distinguish lock contention from a broken destination. Preserve operator_description() (or the typed error) through RegistryAuditError so these commands report the actionable recovery.
Useful? React with 👍 / 👎.
| release does not read or verify the chain; it appends its own entries to whatever file sits at | ||
| the configured path, and at startup and on each rotation it deletes any file named with an | ||
| eight-digit suffix beside it that is older than `audit.retainDays`, which includes files | ||
| Casework 0.34.0 sealed. | ||
|
|
||
| ```sh | ||
| install -d -m 0700 /var/lib/registry-casework/audit-archive | ||
| cd /var/lib/registry-casework/audit | ||
| mv casework.ndjson casework.ndjson.[0-9]* /var/lib/registry-casework/audit-archive/ |
There was a problem hiding this comment.
Make the archive command work without sealed segments
When the old Casework audit stream has never rotated, casework.ndjson.[0-9]* matches nothing and an ordinary shell passes that literal pathname to mv. The mv --help multi-source form is SOURCE... DIRECTORY, so the command moves the active file, then reports that the unmatched second source does not exist and exits nonzero; this makes the mandatory upgrade procedure fail in a common small deployment, especially under set -e. Enumerate existing segments conditionally, use find, or enable a null-glob behavior explicitly.
AGENTS.md reference: docs/site/AGENTS.md:L21-L24
Useful? React with 👍 / 👎.
Summary
BReg serialized audit writes through its PostgreSQL chain head, and the products maintained separate chain implementations and outbox publishers. This change moves BReg, Evidence, Relay, Render, Casework, and Scheduling onto one per-process JSONL
AuditWriter.The envelope is
{schema,eventId,time,phase,correlation,record}. File acceptance follows group-commit fsync; stdout is explicitly best effort. Files have exclusive writer locks, private permissions, bounded rotation/retention, and companion CLI paths. Product record minimization and keyed identifier hashes are preserved.#1519 has merged, and this branch is rebased onto current
main. The branch has linear, signed-off history. The detailed security review is retained privately. The response-contract decisions are resolved as described below; the PR stays in draft until the maintainer undrafts it.Review fixes
A staff review of the previous head found three P1 gaps against the approved contract (a request entry accepted before protected I/O, and a response entry accepted before any result is released). Each fix below has PostgreSQL tests that failed before the change and pass after it.
service.unavailable. SCHEDULING-SEC-14 now covers replays.{event, outcome}response (replayedorunchanged) before its result is returned. An operation with neither a domain event nor an outcome is refused before commit. This is registered as CASEWORK-SEC-22.casework_audit_outboxnow injects it through review history in the same transaction.Response-contract decisions, resolved:
service.unavailablewithout the run id, and finds the run by listing open runs for itsinputDigest. Operator reruns of erasure, rebaseline, and reconciliation find the committed state instead of replaying it. This is documented inINGESTION-RUNS.md,DEFINITION-OF-DONE.md, and the operator pages. A scoped recovery contract is tracked in Make BReg ingestion-run creation replayable after a refused response audit entry #1570.HISTORY.mderasure coverage, the BReg API and retention pages, and Casework's runtime configuration and retention pages).destination: stdout(Codex P2). Companion processes (bregctl,caseworkctl,schedulingctl) now write audit to stderr, so a--format jsonreport keeps stdout. Configuration cannot select stderr;for_processderives it fromstdout. The publicbregctl --format json testlifecycle test runs against a TLS PostgreSQL with a stdout destination, failed before the change with trailing characters after the JSON report, and passes after it.Codex review of the rebased head:
check_writableaccepted an existing file the writer cannot open for read and append. This is fixed: the preflight checks read and write access, with a 0400 regression test.symlink_metadatarefuses a file ancestor or an unsearchable ancestor with an error other thanNotFound. A guard test covers both shapes.A later Codex finding, that Evidence
audit show --last-operationignores an earlier unmatched attempt, is pre-existing onmainand tracked in #1569.CI on the rebased head also failed in the BReg tutorial job, because
dev_lifecycle.rsstill called the removedbregctl audit verify, and in Docs checks, because of an unintroduced "BReg" on the API stability page. Both are fixed. The lifecycle test passes locally, and the full docs check passes locally.CI on the rebased head then failed in Casework PostgreSQL transactions. The native-exchange ThunderID test still wrote the retired Evidence
auditStorageblock. The test now writesaudit.path, and both ignored ThunderID exchange tests pass locally the way CI runs them.Review of the audit-pairing follow-up commits
A five-reviewer pass over the follow-up commits found no blockers. The pairing contract is now stated as "every audited request entry gets at least one answer while the process runs": a second answer is tolerated, and a process that stops with a write in flight can leave a request unanswered (docs, CHANGELOGs, and hooks comments are scoped accordingly). Fixed here, each with a test that failed first unless noted:
unfinished, neverrefused, when its commit errors or its response entry fails.Tracked as follow-ups: commit-fate read-backs after a lost COMMIT acknowledgment (#1604), unfinished answers at runtime shutdown (#1605), and stronger pairing assertions across BReg PostgreSQL tests (#1606).
Evidence
Rebased head
These checks ran on the rebased head with the review fixes. Only the two Codex fixes above came after them, and those were rechecked with
cargo fmt --check, clippy (platform-audit, and Casework withpostgres-test), the platform-audit, caseworkctl, and evidencectl tests, the BReg startup and runtime-config targets, and the Casework native-exchange tests.test-postgres.sh --lane all, run with no-fail-fast across 70 binaries: 661 passed, 11 failed, 10 ignored. All 11 failures are HTTPS loopback delivery tests (postgres_webhook_delivery,postgres_hook_proposals, and one request-lifecycle webhook case).postgres_webhook_deliveryfails the same way on currentmainon this Mac, and CI's BReg PostgreSQL contracts job passes on this branch. I treat this as a local environment fault, not a regression, but the cause is not yet identified.relayctllaunched through a wrapper that setsDYLD_FALLBACK_LIBRARY_PATHfor the aws-lc FIPS library, because SIP stripsDYLD_*through/usr/bin/env.Previous head
Evidence below is for the previous head
9467174eedd039cba5b773f0d47092b69c5da927, before the rebase and review fixes. Checks below used the pinned toolchain, locked dependencies, and disabled incremental compilation. No third-party dependency versions changed.cargo fmt --checkandcargo clippy --locked --workspace --all-targets -- -D warningspassed.cargo test --locked -p registry-cli-docs -p registry-language-serverpassed, including doctests.cargo deny checkpassed all four policy classes.cargo test --locked --workspace --no-fail-fastrun completed with 6,978 passed and eight failed. All failures are the existing BReg (three) and Casework (five) macOS installer fixtures, unchanged from9a05d3688: their v9.8.7 fixtures supply raw binary assets while the v0.33+ installers require FIPS tar archives. They fail at asset lookup before the intended installer fault injection. An isolated rerun reproduced the BReg failures, and independent source review confirmed the shared cause. All remaining executed tests and doctests passed; earlier concurrent-run rustdoc artifact errors did not recur.products/breg/scripts/test-postgres.shpassed 648 tests across 68 test binaries. Casework's documented PostgreSQL targets passed 145 tests across 13 targets; Scheduling's three PostgreSQL targets passed 82. Disposable databases were provided; these were executed tests, not missing-database skips. BReg's ten explicitly opt-in cases remain listed as ignored; the separately requested S3 journey passed.products/identifiers/scripts/check.shpassed 18 tests and canonical artifact reproduction;generate.py --check-referencespassed.caseworkreview-authority binding. Source comparison with9a05d3688confirmed this baseline omission; this is a failed gate, not execution proof for that journey.sync-registry-client-node.py --checkpassed. macOS tests used the maintained Cargo runtime-library helper; exact Node test scripts were invoked directly to retain its FIPS library path. No binding source or rpath was changed.npm testpassed 614 tests;npm run checkpassed, including 207,148 link/asset checks. Final prose corrections passed 604 evidence anchors, 1,884 paths, and 1,727 symbols; architecture checks passed all 12 models.gitleaks git --config .gitleaks.toml --log-opts=9a05d3688..HEAD --redactscanned all 19 commits with no leaks. All commits are signed off and history is linear.The parent #1519 advanced to
575bc4db9with an unrelated harness fix after this branch was based on9a05d3688; a merge-tree check confirmed a clean combination.The workspace reported 41 intentionally ignored tests: opt-in benchmarks, public demos, issuer/container and installed-binary lifecycles, fixture regeneration, and externally fetched schema checks. Their reasons remain in the test output. The requested real S3 and Evidence sustained-load tests were executed separately and passed; ignored entries are not counted as verification of the remaining journeys.
Directional measurement
The paired BReg runs use 100,000 seeded records, seed 20260902, the same workload, and interleave before-1/after-1/before-2/after-2. Before is
9a05d3688; after is8821b3a67, including the writer cancellation fix. The later final-tail startup check does not change normal append behavior.The before environment retained historical measurement mutations; after was freshly seeded. Nominal seed settings and identifier pools matched, but the accumulated database state was not reset identically. This is an additional limit on interpreting the measured ratios.
Counters are the primary evidence. Transaction counts include monitoring and background activity; the denominator is completed workload operations, including failed operations. Achieved operations/second is not successful-request throughput.
The observed transaction ratio was 41.8–47.4% lower across pairs. Different failure and work-completion counts limit interpretation of the magnitude. All eight steady runs saturated and failed the unchanged harness SLOs. Completed rates were 44.933–45.220 operations/s, with 2,938–3,929 HTTP 504 responses per run and p99 near 10 seconds. This does not establish successful-throughput improvement.
The ignored release-mode Evidence sustained-load test passed at 5,461 requests/s, 10,923 audit appends/s, p50/p95/p99 20.07/44.82/64.75 ms, and zero failures. It used the actual temporary file-backed group-commit writer, not an injected sink. The source baseline was 130,722 requests/s, 23.9 times the gateway rate. Audit cardinality assertions passed.
Host: shared Apple M5 Max, 18 logical cores, macOS 26.4.1, Docker PostgreSQL. BReg start/end one-minute load ranged 5.59–36.05. Evidence load moved from 31.11/28.39/23.68 to 45.46/32.14/25.17 (1/5/15 minutes), with 17 and 8 unrelated rustc processes at its endpoints. Full tables and reproduction commands are in the BReg loadtest README and Evidence operator contract.
All timings are directional observations from a shared development machine, not capacity claims or absolute pass/fail gates. Host load and build overlap were recorded for every run. The optional sweep was omitted; the paired steady repeats are the comparison.
Discarded timing comparisons:
before-1/after-1andbefore-2/after-2both had lower after rates while builds overlapped. Each pair was repeated once asbefore-1-retry/after-1-retryandbefore-2-retry/after-2-retry; both retries also overlapped unrelated compilation, so their timing evidence remains inconclusive and is excluded from improvement claims. All raw observations are retained. The BReg sampler counts build-command matches, including wrappers, rather than exact compiler processes; separate executable checks confirmed real compiler overlap. No runtime setting or threshold was tuned from these runs.Notes
Ordering and recovery
A crash or audit refusal after a mutation commits can leave an unmatched request entry. An audit refusal can return unavailable while the effect remains committed. Recovery must inspect the receipt/history and use the operation's supported idempotency key. BReg ingestion-run creation, cancellation, hook proposals, and maintenance commands are not uniformly covered by idempotent replay; blindly resubmitting is unsafe. A refused writer needs repair and process restart.
Security review note, commit resolution (review round of 2026-09-27):
unfinished, never refused. Once a BReg mutation, batch, request action, immediate action, hook proposal, or evidence action reaches its commit, a commit error or a failed post-commit audit append drops the attempt guard, which recordsunfinished. Previously a lost COMMIT acknowledgement was journaled as a refusal for an effect that could be durable. This is deliberately conservative: a commit that clearly rolled back is also recordedunfinished. Callers still seeunavailable.unfinished, instead of rolling the verdict back.request-scrubresponse under the lifecycle correlation; scrub counts appear only in the terminal entry, so a failure after the scrub commits journalsunfinishedrather than a committed lifecycle.unfinishedresponse entries for its events, with no preceding request entry.Security review note, writer contract (staff review and final review round, 2026-09-27):
/var/logis accepted), and each refusal names the fix.<path>.<sequence>.<path>.seqrecords the next sequence before every rotation, so numbering stays continuous across restarts, including after a shipper removes every sealed segment.<path>.lock,<path>.seq, and<path>.seq.tmpare reserved;for_processrefuses a role whose sibling would land on one (ProcessRoleOverlapsStream)..lockcompanion, and a retention failure is logged instead of stopping the writer (it costs disk, never an entry).StartupError::AuditDestination, reported bybregctl doctorasstartup.audit.refused. Evidence's audit startup error kinds are Configuration, Secret, Storage, and File.unknowndisposition (BReg and Scheduling), never asretry_pendingfor a row that may be terminal.Follow-ups tracked instead of held on this PR: #1614 (torn last line recovery), #1615 (shared segment naming), #1616 (descriptor-held audit directory, including retention through a swapped ancestor), #1617 (handle-based answers), #1619 (durable prefix when a grouped write fails during rotation), #1620 (
evidencectl doctorand relativeaudit.path).The former Relay claim that terminal audit cryptographically bound exact response bytes was inaccurate: the old implementation ignored
_exact_response_bytes. The corrected claim is that terminal acceptance gates release of the exact held bytes. This is a documentation/test-name correction, not removal of a prior binding behavior.Compatibility and migration
This is a breaking audit-format and configuration change. In-product chain verification, key-epoch detection, and SQL audit querying are removed. Operators must ship logs to append-only storage with separate deletion authority when tamper evidence and retained completeness are required.
registry_auditandregistry_audit_headdata before package application removes them. Rebuild, sign, and apply the successor package for the changed catalog fingerprint. History-erasure coverage and encryption-erasure progress remain database state independent of audit retention.audit.pathcontract. Render moves toaudit/render.jsonl.bregctl audit verify/export/prune,evidence verify-audit, andregistry-render audit-verify. CLI publication metadata, product changelogs, operational guidance, and architecture models follow the new contract.Minimization remains at parity. In particular, Casework's published allowlist already excluded raw task/request/result identifiers, decision, and target fields before this change. Known-answer tests preserve the production identifier-hashing derivation.
Platform inventory
AuditWriter,AuditEntry,AuditPhase,AuditDestination,FileDestination, destination/unavailability errors, rotation/retention bounds,for_process, andfrom_line_sinktest support.AuditProfile,AuditKeyHasher, identifier-key derivation, redaction helpers, authorization events,require_audit_under, andassert_json_absent_strings/AuditJsonLeakError.AuditSink,JsonlFileSink,JsonlStdoutSink,SyslogSink,AuditEnvelope,DurableSegmentedJsonlSink,DurableSegmentedAuditLog, andSegmentedAuditSummary.ChainState,AuditChainHasher,AuditChainProfile, chain-key derivation,AuditProfile::chain_hasher/bootstrap_or_start_empty,verify_chain, JSONL/segmented-chain verification and visiting helpers, quarantine/recovery, chain-break records, chain verification errors, rotation/syslog helpers, andassert_chain_integrity/ChainAssertionError.async-trait,registry-platform-canonical-json,subtle,ulid, and developmenttracing-subscriber; platform-testing also dropsasync-trait.Cargo.lockchanges only the workspace packages' dependency edges, with no third-party version updates. The writer uses the existingschemarsanduuidpackages;evidencectladds the shared audit crate for destination checks. Relay and Render shed obsolete chain-related dependencies.Issues
This change removes the hash chains, the segmented-JSONL sink, and the BReg audit head those issues are about:
Closes #1447
Closes #1504
Closes #1125
Closes #1133
DCO
Signed-off-bytrailer.