Skip to content

docs(backlog): park differential closeout on its measured cost - #235

Open
iperev wants to merge 1 commit into
chain/r1-go-test-countfrom
chain/r2-backlog-gate-diff
Open

iperev wants to merge 1 commit into
chain/r1-go-test-countfrom
chain/r2-backlog-gate-diff

Conversation

@iperev

@iperev iperev commented Oct 24, 2026

Copy link
Copy Markdown
Contributor

docs(backlog): park differential closeout on its measured cost

Part of the UI-DESIGN-01 chain, stacked on it; the change itself is a backlog decision, not UI work. Base branch: chain/r1-go-test-count (#234, "fix(gate): run Go tests without replaying cached verdicts"). This PR is one commit on that branch's head; merge it after #234. Only BACKLOG.md changes.

Why

GATE-DIFF-01 asked for a closeout gate that inherits the verdicts of unchanged steps from a passing parent tree. Its premises were measured against this repository's recent commits and local full gates, and its rules save almost nothing:

  • Saving. The window is the 40 first-parent commits from 6d19004 to 834f6ae, inclusive. The last seven, from ae283b5 on, were chain commits not yet on main when measured.
    • Over that window the rules would have saved a mean of at most 2.1 s per commit, 0.2 % of a 22.7-minute hosted gate. That already assumes an admitted traced receipt for every parent and a free trace check.
    • 34 of the 40 commits change a gate-definition file or an input of self:receipt or release:closeout, so they run the full gate. Without the release:closeout inputs, which no selector declares yet, the count is 32 and the mean saving 2.2 s.
    • The six commits that save anything are documentation, backlog and test-only commits.
  • Cost. The trace check is required on every gate, full or differential. Its cheapest Linux mode that sees reads, enumerations and failed lookups is strace --seccomp-bpf -f. In micro-benchmarks it multiplied median wall time by 1.5 for file reads and by 11 for process spawns. Any overhead above 0.2 % exceeds the saving. The overhead on a real gate was not measured.
  • False premises:
    • browser:check, the ./... steps and the CLI builds of three node contract steps read the whole worktree.
    • browser:check and boundary-contract:check gave both verdicts on one unchanged tree.
    • On macOS, sandbox-exec hangs Firefox unless Firefox's own sandboxes are disabled.
    • release:closeout has no selectors for the full-gate trigger to match.
    • The toolchain list omitted git, Python, shellcheck and pyflakes.
    • The full gate replayed cached go:test verdicts, which the base PR stops.

The cost that did show was same-tree verdict flips: full gates that failed on trees that later passed unchanged. Inheritance cannot recover that cost, because a step that flips is never inherited.

What changes

GATE-DIFF-01, NEXT to DEFERRED.

  • Scope: the unsoundness statement stays. Added: the measurement, the premises found false, the fixed causes of the boundary-contract:check flips, and what the browser:check flips were.

  • Completion condition: it starts with the reopen condition. All of these must hold:

    • Twenty clean gates in a load envelope. 20 consecutive full gates of one unchanged tree pass browser:check and boundary-contract:check with zero retries.
      • Load per CPU is the 1-minute load average divided by the logical CPU count, sampled at least once a minute during those two steps and kept with the gate's record.
      • Inside the envelope, its median is at most 6 and its maximum at most 12. At least five of the 20 gates must have a median of at least 3.
      • A failure inside the envelope restarts the count. A gate that leaves the envelope is recorded with its verdict and samples, and neither counts nor restarts the count.
      • No timeout is raised, no retry is added and no CSP, console or security-policy check is weakened.
    • Fewer triggers. Fewer than half of the last 40 first-parent commits hit a full-gate trigger.
    • Cheap enough tracing. A trace overhead measured on the real hosted gate is below the mean saving then measured.

    A reopened row first traces every step's file reads, directory enumerations and failed lookups on its own tree. It then recomputes the saving over its own recent first-parent commits and resolves each false premise in its scope.

  • Requirement text: kept, with two corrections:

    • the toolchain list also names Python, git, shellcheck and pyflakes, which tracing showed the steps use;
    • the release:closeout trigger names the witness command, with selectors, that release:closeout must first get.

No new row for browser:check.

  • With fixed 30-second timeouts, any test fails under enough contention, so a verdict can be a function of the tree only inside a stated load envelope.
  • None of the recorded failures is known to lie inside one.
  • The reopen condition is the only consumer of that claim, so it states the envelope itself.
  • The envelope's bounds stay below every failing condition on record. In the isolated Firefox A/B reruns of one evidence test recorded for docs(browser): keep Firefox process isolation and block the SDK successor #220, the lowest-load block that failed had a load median of 66 and a maximum of 103 on 8 CPUs, 8.3 and 12.9 per CPU. Blocks at medians of 53 and 59 passed, and so did one at 81 with a maximum of 111.

Premises re-checked against the gate receipts

As of 2026-10-24 04:25Z, all 54 receipts of the local full gates were read again and every failing log was classified. The gates ran on macOS arm64 with 8 CPUs and zero retries.

browser:check gave both verdicts on three trees. Every failure ended at the 30-second test timeout.

Tree Failing gate: start, load 1/5/15 min Failures Browser phase Load during the run, from notes kept while it ran Passing gate: start, load Browser phase
3f0f398f 2026-10-13 15:03Z, 47.0 / 42.1 / 34.0 WebKit workspace.spec.mjs:524 (download response cannot certify a later failed document): navigateWorkspace never saw its target document 22.6 min 47 to 230 2026-10-14 14:51Z, 7.0 / 15.2 / 21.6 8.8 min
0c8d17eb 2026-10-13 15:40Z, 76.2 / 78.1 / 70.6 Firefox: a focus assertion (workspace-evidence-layout.spec.mjs:73), a page evaluation and two clicks (workspace-evidence.spec.mjs:141, :179, :206). Chromium: one click (lookup.spec.mjs:111), its page closed by the timeout 50.6 min 76 to 310 2026-10-14 15:14Z, 5.1 / 8.7 / 17.6 11.1 min
ac69eea2 2026-10-14 21:34Z, 2.5 / 15.4 / 62.7 Firefox: one click (workspace-graph.spec.mjs:88). WebKit: the same workspace.spec.mjs:524 navigation 33.5 min rose to about 258 2026-10-15 01:50Z, 6.6 / 9.5 / 11.9 8.4 min
  • Time lost: the three failed gates took 152 minutes.
  • How much slower: their browser tests ran 2.6 to 4.6 times as long as on the passing reruns.
  • Start load does not separate the verdicts:
    • every later gate that completed browser:check passed it (561/561), at recorded start loads up to 27.3;
    • earlier gates passed it at start loads of 47.4 and 68.8;
    • the third failing gate started at 2.5.
  • No Firefox navigation failure: no gate log records a Firefox navigation-completion failure (Playwright's "navigation to finish"). All gates ran with Firefox site-origin isolation, which BROWSER-TOOLCHAIN-01 keeps. The Firefox failures are a focus assertion, a page evaluation and clicks that did not complete.

boundary-contract:check gave both verdicts on five trees. All have known causes, and both causes are fixed:

  • 3f0f398f and 0c8d17eb (2026-10-14 09:17Z; the two gates ran concurrently): spawnSync … EPIPE in an early-refusal oracle. Fixed by the shared CLI transport assertion, which accepts EPIPE only from a child that exited non-zero.
  • 48e3c2f1, ac69eea2 and be64b1ef: spawnSync … ETIMEDOUT of one CLI call. That is the lost wake-up of a reader on the macOS Unix socket that spawnSync's input option creates. Fixed by the test-transport change, which gives spawned CLI children their input as a file.

On trees with the test-transport change, five gates have passed boundary-contract:check so far. Three of them finished with passing receipts; the other two logs reached later steps.

Compatibility

Documentation only. BACKLOG.md is not in the npm package. No rule in AGENTS.md and no product surface changes.

Proof

Proof: A local npm run check sequence on darwin/arm64 passed (exit 0) on tree 99c57d7635e5 with Node 26.11.1, npm 12.2.0, CPython 3.15.0 and Go 1.27.3, with no Go test verdict replayed from the test cache: 561/561 runtime browser cases in Chromium, Firefox and WebKit with zero retries. One recorded deviation: the host firewall blocks Go-built binaries from the network, so go:vulncheck ran against a vulnerability-database snapshot fetched with curl (modified 2026-10-21T00:27:29Z): no vulnerabilities.

Narrow evidence: git diff --check, npm run text-policy (passed, 1421 text files) and npm run source-hygiene pass. BACKLOG.md is not shipped, but the added lines still match none of the shipped-markdown rules of packageverify (version tokens, package coordinates, run and tarball URLs, raw hashes, source refs).

Non-claims

  • Figures. The saving and trigger figures were measured once, on the 40 commits named above; the overhead factors come from micro-benchmarks. They are not claims about other repositories, other commit mixes, or the trace overhead on a real gate.
  • Causes. The causes of the browser:check flips are not identified beyond host contention. Whether any browser test also has a logic race is not shown.
  • Envelope. Its bounds were chosen below every failing condition on record. That no failure occurs inside them is what the reopen condition tests; it is not a result.
  • Boundary fix. That the boundary-contract:check flips stop is shown only by the test-transport change's own falsifiers and by the gates above, not by 20 gates in the envelope.
  • Other failures. Two failures had no same-tree pass and are not covered here: go test package timeouts after 600 s at start load 69, and one timing test of releasepreflight at start load 9.

GATE-DIFF-01 asked for a closeout gate that inherits the verdicts of
unchanged steps from a passing parent tree. Measured on this repository's
recent commits and gates, its rules save a mean of at most 0.2 % of a
hosted gate, and the trace check they need on every gate would cost more
than that: any overhead above 0.2 % does, and Linux micro-benchmarks of
its cheapest suitable mode showed factors of 1.5 and 11. The row is
deferred with its measurement, the premises found false and a reopen
condition.

- GATE-DIFF-01 moves from NEXT to DEFERRED. Its scope keeps the
  unsoundness statement and adds the measurement: over the 40
  first-parent commits from 6d19004 to 834f6ae, the last seven of them
  not yet on main when measured, the rules would have saved a mean of at
  most 2.1 s per commit, 0.2 % of a 22.7-minute hosted gate, because 34
  of the 40 commits hit a full-gate trigger and the costliest steps
  cannot be inherited. It lists the premises found false: whole-worktree
  reads by browser:check, by the ./... steps and by the CLI builds of
  three node contract steps; same-tree verdict flips of browser:check and
  boundary-contract:check; Firefox under sandbox-exec on macOS;
  release:closeout without selectors; an incomplete toolchain list; and
  go:test verdicts replayed from Go's test cache, which the parent commit
  fixes.
- The scope records that the known causes of the boundary-contract:check
  flips are fixed: the early-refusal EPIPE race by the shared CLI
  transport assertion, and the lost macOS Unix-socket wake-up by giving
  spawned CLI children their input as a file. It also records the
  browser:check flips: two WebKit navigations that did not reach their
  target document and six Firefox and Chromium page operations that did
  not complete, all at the 30-second test timeout, in local full gates
  whose browser tests took 2.6 to 4.6 times as long as on the passing
  reruns, with no load sampled during them.
- The completion condition starts with the reopen condition. Its first
  clause asks for 20 consecutive full gates of one unchanged tree that
  pass browser:check and boundary-contract:check with zero retries inside
  a load envelope: the 1-minute load average per CPU, sampled at least
  once a minute during those steps, has a median of at most 6 and a
  maximum of at most 12, and at least five of the gates have a median of
  at least 3. A gate outside the envelope is recorded but neither counts
  nor restarts the count, and no timeout is raised, no retry added and no
  CSP, console or security-policy check weakened. The other clauses stay:
  fewer than half of the last 40 first-parent commits on a full-gate
  trigger, and a trace overhead measured on the real hosted gate below
  the mean saving then measured. A reopened row first traces every
  step's reads on its own tree, recomputes the saving and resolves each
  false premise.
- The requirement text stays, except that its toolchain list also names
  Python, git, shellcheck and pyflakes, and its release:closeout trigger
  names the witness command that release:closeout must first get.

No row is added for browser:check. With fixed 30-second timeouts any
test fails under enough contention, so its verdict can be a function of
the tree only inside a stated load envelope. None of the recorded
failures is known to lie inside one, and the reopen condition above is
the only consumer of that claim.

AGENTS.md and every product surface are unchanged.

Proof on this tree:
A local npm run check sequence on darwin/arm64 passed (exit 0) on tree
99c57d7 with Node 26.11.1, npm 12.2.0, CPython 3.15.0 and Go 1.27.3,
with no Go test verdict replayed from the test cache: 561/561 runtime
browser cases in Chromium, Firefox and WebKit with zero retries. One
recorded deviation: the host firewall blocks Go-built binaries from the
network, so go:vulncheck ran against a vulnerability-database snapshot
fetched with curl (modified 2026-10-21T00:27:29Z): no vulnerabilities.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Development

Successfully merging this pull request may close these issues.

1 participant