diff --git a/.dev-loop/INGEST_REPORT.md b/.dev-loop/INGEST_REPORT.md index 3a965e5..598be41 100644 --- a/.dev-loop/INGEST_REPORT.md +++ b/.dev-loop/INGEST_REPORT.md @@ -1,237 +1,52 @@ -# Knowledge flush — 13 insight(s) ingested (21 claimed, 2 dropped, 6 released) +# Knowledge flush — 6 insight(s) -Cross-Check: 1× independent adversarial `claude` CLI headless pass over the 5 new pages — it refuted the changed-files gate page's "prettier exits 0 on an empty match set" claim; re-measured against Prettier 3.7.4, confirmed the reviewer was right (unmatched operand exits **2**), and rewrote the page, report rows 3/5 and `log.md`. Other 5 claim groups verdicted sound. Limits: the reviewer's sandbox blocked repo reads, so source-quote-supports-directive and self-contradiction dimensions went unaudited (details in `## Cross-Check`). - -Queue drained under run id `20260827-125731-38371` (this session is the detached -`hooks/auto-flush.sh` run; its step-0 acquire resolved re-entrantly to -`already-owned`, not to a competing holder). 21 rows were claimed; 13 are -ingested below, 2 are retired as out-of-layer, and 6 are released back to -`pending` for a later flush because each needs its own page rather than a row, -and rushing six more pages in one pass would have lowered the bar on all of them. +Run: auto-flush headless child (run id `20260910-161926-53458`), queue drained from 3 session files (6 claimed rows, none previously processed). Outcome: 6 new pages, 0 merges, 0 folds, 0 drops. Lint after ingest: `wiki-structure-checks.js` 282 pages / 13 indexes / 0 findings; `wiki-lint-prohibitions.js` directives 75 / violations 0 (count unchanged, so the bats pin needs no bump); every new page ≤ 120 body lines (56–75). Six research agents vetted one candidate each against primary sources; local reproductions were run where the claim was runnable, and every cited URL was fetched or search-confirmed live this run. ## Verified best-practice -Every external claim below was live-fetched this session and quoted in the page's -`Sources` block. Field evidence carries the repo, date, and the measured numbers. - -| # | Claim | Sources checked | How verified | Confidence | -|---|-------|-----------------|--------------|------------| -| 1 | `now()` is `transaction_timestamp()` (fixed at transaction start) while `clock_timestamp()` "changes even within a single SQL statement"; `RETURNING` yields computed defaults "without needing a separate database query" | postgresql.org `functions-datetime`, `dml-returning`, `transaction-iso` | Fetched; both key sentences quoted verbatim into the page | verified | -| 2 | A boundary recomputed in a follow-up step is a *second, later* `now`, widening a `<= boundary` set | Field: `rtb-unified` `packages/orpc/src/routers/batch.ts` — codifies "one `now` per decision" and passes `now` into the boundary helper; its result type omits the boundary, which is the shape that invites recomputation | Read the invariant and the signature in the cited file | field-tested | -| 3 | **[CORRECTED BY CROSS-CHECK]** The vacuous-pass shapes for `prettier --check` are: no operands (rc **0**), all operands ignore-filtered (rc **0**), and unsupported extensions with `--ignore-unknown` (rc **0**). A pattern/operand matching nothing exits **2** — it prints the success sentence *and* an unmatched-pattern error | prettier.io CLI + ignore docs; local measurement, Prettier 3.7.4 | The first draft generalised "empty match set ⇒ exit 0" from a field log where both messages appeared together. The independent reviewer flagged it; I then ran all seven cases against a real binary and rewrote the page around the measured table | verified (re-measured) | -| 4 | zsh does not word-split unquoted parameter expansions by default, so `cmd $FILES` arrives as **one** operand | zsh FAQ ch. 3 (`SH_WORD_SPLIT`) | Fetched; quoted ("By default, zsh does not have that behaviour: the variable remains intact") | verified | -| 5 | The zsh word-split operand exits **2**, but its log still carries the success sentence — so the log misleads even though the exit code does not | Field 2026-08-24 (`rtb-unified`, zsh) + local measurement 2026-08-27 | Field log showed both messages together; the local run reproduced it as `rc=2`. The page now says explicitly that this row fails loudly *unless* `--no-error-on-unmatched-pattern` is set. Probe placement re-confirmed: `.claude/tmp/` is `.gitignore`d, so a probe there passes at rc 0 | verified (re-measured) | -| 6 | TypeScript applies excess-property/contextual typing to fresh object literals, so a value of a type can be constructed with the type's name absent from the text | typescriptlang.org handbook, *Object Types* | Fetched; confirmed the check follows from the contextual type, not from a written annotation | verified | -| 7 | `tsc`'s program is `files` ∪ `include` ∪ transitive imports; `exclude` "only changes which files are included as a result of the `include` setting" and does not stop an imported file entering the program | typescriptlang.org TSConfig `#include`, `#exclude` | Fetched; the `exclude` sentence quoted (it sharpens the rule to "in the program", not "in `include`") | verified | -| 8 | Consequence of 6+7 measured | Field 2026-08-24/25 (`rtb-unified`): `grep "DealViewer"` reported 3 construction sites, actual 8 — the missed set included production wiring `routers/deal.ts:38`; `ContractScopeActor` 7→~22. Separately, `packages/orpc/tsconfig.json` `include: ["src/**/*"]` produced 3 production + 13 api-test errors and **zero** for `__tests__/routers/deal.test.ts`, whose 6 sites appeared only as 6 failing tests | Counts recorded from the cited runs | verified | -| 9 | cgroup v2: `memory.peak` is max usage since creation/reset; at `memory.max` "the OOM killer is invoked in the cgroup"; in `memory.events`, `max` counts times usage "was about to go over the max boundary" — **distinct** from `oom_kill` | docs.kernel.org cgroup-v2 admin guide | Fetched; all four quoted. This corrected the candidate, which had read a non-zero `max` as a kill; the page now states the distinction explicitly | verified | -| 10 | An `exec`'d process joins the container's cgroup and is invisible to the application's own semaphore | kubernetes.io `manage-resources-containers`, `assign-memory-resource`, `kubectl exec` reference + field 2026-08-26 (review-bot pod, `limits.memory: 3Gi`): `memory.current` 2.54 GiB, `memory.peak` 3.0 GiB (at the limit), `memory.events: max 5`, while `maxConcurrentAgents: 20` reported free slots | Docs fetched; pod numbers from the cited measurement | verified | -| 11 | Basename-keyed mutation backups collide across directories and restore cross-writes; an untracked file's `git diff` is empty whether restored or destroyed | Field 2026-08-21 (`rtb-unified`, NEWRTB-2936): restore wrote `schemas/deal.ts` into `routers/deal.ts` → `Cannot find module './common.js'`, `grep -c dealRouter` = 0; **both files were 154 lines**, so a line-count check passed; after re-keying, M9/M10 flipped SURVIVED→KILLED. Plus stryker mutant-states / pitest for the verdict vocabulary | Reproduced end to end in the cited run | field-tested | -| 12 | A negative assertion is vacuous when the fixture never supplies the triggering input | Field 2026-08-25 (`rtb-unified`): with `staleQueuedJobIds: []` the code early-returned; the widening the assertion claimed to catch survived 116/116 green | Mutation applied and observed | field-tested | -| 13 | A body-level (non-inline) review finding cites no file, so rebutting from an assumed file rejects real defects | Field 2026-08-19 (PR #327 r16): quote matched `report.py:393/416/425`, not the assumed `fill_plan.py:307` — sibling modules, one already fixed | Grep resolved the quote to the real site | field-tested | -| 14 | Unifying two duplicate allowlists defaults to the union and silently widens each side | Field 2026-08-25 (`rtb-unified` PR #965): folding `DISPLAYABLE_ERROR_CODES` into `USER_FACING_ERROR_CODES` would have added `UNAUTHORIZED` + `VALIDATION_ERROR`, exposing raw server messages as inline UI errors; caught only by computing the difference first | Difference computed before the merge | field-tested | - -Not upgraded: nothing was marked `verified` on field evidence alone. Two pages -carry `confidence: field-tested` (`mutation-harness-file-custody`, -plus the pre-existing `evaluating-review-feedback`), and no candidate was -recorded as `verified` without a fetched primary source. +1. `infrastructure-agent-orchestration-gate-evidence-exit-code-class` (16f22ec2) — **verified**. GNU Bash manual "Exit Status" ("If a command is not found … returns a status of 127. If a command is found but is not executable, the return status is 126"); POSIX XCU §2.6.2 Parameter Expansion (an unset `${parameter}` with no modifier substitutes nothing, so `${CLAUDE_PLUGIN_ROOT}/skills/…` becomes `/skills/…`); code.claude.com plugins reference (`${CLAUDE_PLUGIN_ROOT}` is substituted only in skill/agent content, hook/monitor commands, MCP and LSP server fields — no plain-shell export documented). Source read of dev-loop v1.21.0: `templates/plan-gates.md` leaves the variable live for `gate-check.sh`'s `bash -c`; `gate-check.sh` records `exit=: ` with no 126/127 special-casing; `plan-gate.sh` documents its own codes 0/2/3/4. Field: the candidate's ledger went met=1 unmet=4 (all exit=127) → met=5 unmet=0 with the variable exported and no file edited. +2. `platforms-tools-jq-dot-rebinding-in-predicates` (3564214d) — **verified**. jqlang.org manual (pipe rebinds `.`: "`.a | . | .b` is the same as `.a.b`"; `index(s)` on an array input is a subsequence search; `IN(s)`; `any(gen; cond)`), identical wording in the 1.6 manual; Stack Overflow 48898983 (scalar `index` is the one-element special case of subsequence search). Local reproduction on jq-1.7.1-apple: `["a","b"] as $arr | ["b","zzz"] | any(.[]; $arr | index(.) != null)` → `true` even for an input sharing nothing with `$arr`; `. as $x | $arr | index($x) != null` and `any(.[]; IN($arr[]))` discriminate correctly; `["a","b"] | index(["a","b"])` → `0`. Field site confirmed by source read: dev-loop `skills/orchestrate/scripts/graph-drop.sh:67`. +3. `infrastructure-agent-orchestration-worker-reported-plan-contradiction` (d7a0cae5) — **field-tested**. Supporting sources: Python `sqlite3` docs (`IntegrityError` and `OperationalError` are both subclasses of `DatabaseError` ⊂ `Error`, so `except sqlite3.Error` swallows an `IntegrityError` a test asserts propagates — verified); Anthropic "Building multi-agent systems" (verification subagent must "run the complete test suite before marking as passed") and "How we built our multi-agent research system" (lead agent synthesizes subagent results as evidence) support the general resolve-by-running principle; principles-wiki DRY entry (a single source of truth is what stops representations diverging). No primary source names the exact design-doc-vs-step-file case, so the page stays field-tested; the field run is quoted (`2 failed, 242 passed`, both failures `DatabaseError` wrapping `IntegrityError`). +4. `debugging-performance-attributing-a-benchmark-speedup` (048faa67) — **verified**. Brendan Gregg "Active Benchmarking" ("you benchmark A, but actually measure B, and conclude you've measured C"); Gregg "Evaluating the Evaluation: A Benchmarking Checklist" ("what is the limiting factor?"; a firewall timeout measured as server latency); Gernot Heiser "Systems Benchmarking Crimes" (a measured delta does not follow from the change believed to cause it; compare against the real, correctly configured baseline). Field: `bench_pipeline.py:354` built the baseline with `delay_seconds=0.0` while `local_scraper.py:126` read `default_config.DELAY_SECONDS` (3.0) — the credited removed sleep cost nothing in the measurement. +5. `testing-quality-sequential-dispatch-assumption-under-concurrency` (28c88c5f) — **verified**. Python `concurrent.futures` docs (`submit()` schedules and returns a Future; default `max_workers = min(32, cpu+4)`; `shutdown(cancel_futures=True)` cancels only not-yet-started futures); Fowler "Eradicating Non-Determinism in Tests"; Google Testing Blog "Where do our flaky tests come from" (concurrency changes as a measured cause). Local reproduction, CPython 3.14.6: 3 tasks, `max_workers=4`, task 1 sets an Event checked at each task's top → `ran: [0, 1, 2]`; `max_workers=1` → `ran: [0, 1]`. Field: `test_shutdown_stops_after_current_company` deterministically red (call_count 3 vs 1) after the ThreadPoolExecutor migration, 265/266 on repeated runs. +6. `testing-quality-proving-a-critical-section-is-lock-protected` (5d88d302) — **verified**. Python FAQ "What kinds of global value mutation are thread-safe?" (`D[x] = D[x] + 1` is not atomic); `sys.setswitchinterval` docs; Google Testing Blog "Mutation Testing" (a test proves detection only if it fails when the guarded behavior is removed); Python free-threading HOWTO for the 3.13+ caveat. Local reproduction, CPython 3.14.6 with GIL: plain dict `+=`, 16 threads × 10, no lock → 160 in 20/20 trials (0 races); `__setitem__`-sleeping dict subclass, no lock → 10 in 10/10 trials; with `with lock:` → 160 in 10/10. The candidate arrived as a prohibition ("don't rely on raw stress") and was rewritten as the positive directive with the prohibition kept only as an `Instead of` row. ## Existing-layer check -Method: routed via `INDEX.md` → domain `index.md`; then built a full id+title -index of all 265 pre-existing pages and probed it with concept greps -(`clock_timestamp|clock skew`, `changed[- ]files|--ignore-unknown`, `tsconfig`, -`contextual typ|excess property`, `set difference|allowlist`, `cgroup`, -`basename|backup.*restore`, `2>&1`, `delta|baseline`) before deciding new vs merge. +Routing went INDEX.md → domain `index.md` → every page whose "load when" overlapped; a second domain index was read wherever the queue tag was doubtful (d7a0cae5 tagged backend → infrastructure/agent-orchestration, the domain that owns the coordinator/worker process; 048faa67 tagged qa → debugging/performance, which already hosts the sibling Gregg-sourced benchmark page; 28c88c5f/5d88d302 → testing/quality rather than testing/async or testing/flaky, since both concern a test's ability to fail, not async code or intermittent failures). -Pages read: testing-quality-source-text-wiring-assertions, testing-quality-tests-that-cannot-fail, backend-common-change-impact-call-site-enumeration, backend-common-change-impact-widening-a-closed-value-table, qa-process-evaluating-review-feedback, infrastructure-containers-host-cgroup-visibility, testing-quality-behavior-not-implementation +Pages read: testing-quality-checks-that-cannot-pass, debugging-signals-reading-error-messages, platforms-environment-path-resolution, platforms-shells-command-text-inspected-before-execution, infrastructure-config-path-valued-config, infrastructure-agent-orchestration-session-completion-gates, infrastructure-agent-orchestration-control-signals-vs-primary-artifacts, platforms-tools-plugin-mcp-server-registration, platforms-shells-portable-shell-scripts, platforms-shells-escapes-in-shell-string-literals, platforms-processes-parsing-cli-structured-output, testing-quality-tests-that-cannot-fail, testing-quality-guard-shape-vs-consequence, infrastructure-agent-orchestration-autonomous-decision-rulings, infrastructure-agent-orchestration-unattended-worker-questions, infrastructure-agent-orchestration-shared-run-state, qa-process-evaluating-review-feedback, backend-common-llm-binding-instructions-for-agents, debugging-methodology-hypothesis-testing, qa-deliverables-quantitative-claims-in-a-published-document, qa-process-completion-claims, debugging-performance-profile-before-optimizing, debugging-methodology-probe-path-vs-operation-path, testing-quality-harness-reverse-controls, testing-async-async-testing, testing-flaky-diagnosing-flaky-tests, testing-quality-behavior-not-implementation, backend-common-concurrency-shared-state-and-pools, backend-python-concurrency-gil-and-concurrency-model, backend-node-runtime-graceful-shutdown, debugging-concurrency-intermittent-failures, testing-quality-mutation-harness-file-custody, testing-quality-write-path-assertions, backend-common-concurrency-distributed-locks -Findings: +Read from open-PR branches only (not on main): testing-quality-gate-parsing-vs-command-execution (#183), infrastructure-agent-orchestration-checkable-claims-in-an-adopted-plan (#181), infrastructure-agent-orchestration-verify-command-in-a-worker-brief and -semantic-conflicts-after-parallel-merge (#179), infrastructure-agent-orchestration-inbound-validation-ownership-in-task-decomposition (#182), infrastructure-agent-orchestration-forward-references-in-a-numbered-protocol (#180), backend-common-llm-vendor-benchmark-claims-for-an-llm-tool (#182), testing-quality-narration-based-ordering-assertions and testing-quality-cross-task-stub-assertions (#188). -- **Zero coverage** (→ new pages): changed-files-only gates, tsconfig/contextual - typing, allowlist set-difference, app-clock-vs-DB-timestamp, exec-into-a-running-container. - The concept greps returned no hits for these; the clock hits were incidental - (offline sync, token handling) and none compared an app clock to a DB column. -- **Already covered — one candidate all but retired.** The comment-stripping - insight is `source-text-wiring-assertions` step 2 verbatim ("Make the - assertion's subject the file with comments removed"), and its false-RED and - negative/count false-GREEN shapes are already edge rows. Only the *empty-slice* - consequence was new, so that alone was merged. -- **Line-cap conflict handled without breaking the invariant.** - `source-text-wiring-assertions` sits at exactly **120** body lines (the - documented cap). Rather than add a row and violate maintenance invariant 5, the - new nuance and the new field evidence were merged **in place** into an existing - edge row and an existing source bullet. Body count re-measured after editing: - still 120. -- **No conflicts found.** Nothing ingested contradicts an existing directive. - The one correction made was to a *candidate*, not to the wiki (item 9: the - `memory.events` `max` counter is approaches-to-limit, not kills). -- **Related links added both ways**: `tests-that-cannot-fail` ↔ - `mutation-harness-file-custody`; `widening-a-closed-value-table` ↔ - `compiler-as-call-site-inventory` (+ `errors-diagnostics-from-a-shared-code-path`); - `host-cgroup-visibility` → `exec-added-processes-and-the-memory-budget` - (from its existing self-monitoring row). -- **Indexes/log updated**: 4 domain indexes (+5 "load when" rows), `log.md` - appended. Root `INDEX.md` unchanged — no new domain. - -Gates run (the exact CI commands from `.github/workflows/test.yml`): -`node scripts/wiki-structure-checks.js wiki` → **pages: 270, indexes: 13, -findings: 0**; `node scripts/wiki-lint-prohibitions.js wiki` → **directives 72, -compliant 72, violations 0** (the 1 `info` is pre-existing in -`config/keys-ahead-of-their-consumer.md`, untouched); `bash scripts/check-versions.sh` -→ `ok: dev-loop 1.11.2`. The `bats tests/` job was **not** run — bats is not -installed on this machine, and this change touches only wiki markdown (no -scripts or hooks), so that suite's subject is unchanged. +Overlaps and outcomes: +- **Created (new trigger, nothing covering it):** all six. Closest neighbours were cross-linked, not merged: checks-that-cannot-pass (scoped to a check never yet observed passing; the gate here had passed elsewhere and failed only environmentally); escapes-in-shell-string-literals and parsing-cli-structured-output (shell-quoting and field-path layers, not jq's own pipe semantics — a wiki-wide grep for `jq`/`index(.)`/`rebind` found no existing guidance); autonomous-decision-rulings (how to record a ruling, not how to decide a testable contradiction); profile-before-optimizing (pre-optimization profiling, not post-hoc attribution) and harness-reverse-controls (mutation-harness construct validity, not benchmark arms); async-testing (async code, not dispatch-model migration); intermittent-failures (widens a race to diagnose an observed flake — the new page authors a test before any failure is observed and cites it). +- **Merged:** none. +- **Conflicts flagged:** none. The jq page contrasts, without contradicting, plugin-mcp-server-registration's note that an unresolved `${VAR}` in MCP config stays literal while the shell expands it to empty. +- **Related links added both ways** on 16 existing pages (frontmatter `related:` only, appended at the end of each list to minimise textual conflict with open PRs that touch the same files). The two testing pages link each other. Deferred until the owning PRs land, because the target ids do not resolve on main: gate-evidence-exit-code-class ↔ gate-parsing-vs-command-execution (#183); worker-reported-plan-contradiction ↔ checkable-claims-in-an-adopted-plan (#181; the page's inline reference to it was rewritten as plain text); attributing-a-benchmark-speedup ↔ vendor-benchmark-claims-for-an-llm-tool (#182). The reverse link from tests-that-cannot-fail to the two new testing pages and the jq page is also deferred: three open PRs (#179, #186, #188) already edit that file, so a fourth edit was avoided; the forward links from the new pages are in place. ## Open-PR check -`gh pr list --repo choiyounggi/dev-loop --state open --search "head:knowledge/"` -returned **no open PRs**, and a second unfiltered `gh pr list --state open` -returned none either — the repository has zero open PRs at flush time. There -were therefore no in-flight sibling branches to diff against, and no -`git fetch origin ` / `git diff origin/main origin/ -- wiki/` -comparisons to run. +Open `knowledge/*` heads listed via `gh pr list --search "head:knowledge/"` and fetched: #188 (`knowledge/choiyounggi-20260908-154412`), #187 (`…-20260906-213635`), #186 (`…-20260906-013856`), #185 (`…-20260906-003745`), #183 (`…-20260904-133717`), #182 (`…-20260903-214027`), #181 (`…-20260903-203836`), #180 (`…-20260903-184706`), #179 (`…-20260903-172728`). Each candidate was diffed against every head's `wiki/` changes (`git diff origin/main origin/ -- wiki/`, grepped for the candidate's key terms, overlapping pages read via `git show origin/:`). -Per-candidate verdict: **all 21 = `new`.** No `fold`, no `drop-as-pending-duplicate`. -(The 2 drops recorded below are out-of-layer drops, not pending-duplicate drops.) +Per-candidate verdicts: +- **new** — 16f22ec2 (gate exit-code class). #183 adds gate-parsing-vs-command-execution (parser/executor split, command-injection framing of the same script) and #179 touches reading-error-messages, path-valued-config, path-resolution with related-links and an unrelated argparse edge case; no head mentions exit 127/126, `CLAUDE_PLUGIN_ROOT`, or "command not found". +- **new** — 3564214d (jq `.` rebinding). #180 adds jq content to portable-shell-scripts about the *shell* re-splitting `jq -r` output in a `for` loop (same graph-drop script family, different root cause; its own suggested fix uses `--arg` binding and never mentions the pipe-rebinding trap). Not folded: one case per page. +- **new** — d7a0cae5 (worker-reported contradiction). #181's checkable-claims-in-an-adopted-plan is the closest relative but covers the worker-side "check the plan's claims before implementing, escalate and wait" half; this page is the coordinator-side resolution method after the escalation. #179's semantic-conflicts-after-parallel-merge and verify-command-in-a-worker-brief, #182's inbound-validation-ownership, #180's forward-references are adjacent process pages with different triggers. +- **new** — 048faa67 (benchmark attribution). #182's vendor-benchmark-claims-for-an-llm-tool evaluates a third party's published claim; #185's benchmark-relative-signal-classification and #181's synthetic-corpus-measurement-floor are different construct-validity failures; #181/#188 touch quantitative-claims-in-a-published-document only for unrelated content. +- **new** — 28c88c5f (sequential-dispatch assumption). async-testing is modified by #183, #185 and #188, so a new page plus back-links was chosen over a fourth edit; no head touches ThreadPoolExecutor/`submit`/dispatch semantics. #188's narration-based-ordering-assertions is a different failure mode. +- **new** — 5d88d302 (lock-protection test). tests-that-cannot-fail is modified by #179, #186 and #188 (unrelated additions), so a new page was chosen; #185's async-testing row about mutation-validating a fake-timer lockout is single-threaded and not a duplicate. +- **fold** — none. **drop** — none. ## Routing decision -**New pages (5)** - -| Page | Domain/category | From | Why not an existing page | -|------|-----------------|------|--------------------------| -| `application-clock-vs-database-timestamps` | databases / transactions | `2b27d15d` + `bea92fdd` | No page compares an app clock to a DB column. `transactions` chosen over `schema-design` because the decisive content is transaction-time semantics (`now()` = transaction start ⇒ stamp order ≠ commit order) and the fix is a lock/isolation choice | -| `changed-files-only-gates` | infrastructure / ci-cd | `ff041061` + `4b9af3a0` | Zero grep hits. Both candidates are the same defect (a gate green with an empty subject) from two directions, so they became one page rather than two | -| `compiler-as-call-site-inventory` | backend / common / change-impact | `702dcf4e` + `94d55f2f` | `call-site-enumeration` is the sibling case (callers of a changed signature, Python positional-vs-keyword) and is at 80 body lines; the TS mechanism is *constructors of a type* with its own workflow, so per "one case per page" it is a separate page, cross-linked | -| `mutation-harness-file-custody` | testing / quality | `6a9de235` + `41fa1c87` | `harness-reverse-controls` covers scoring a harness; nothing covers the harness's custody of the tree. Both candidates are that one case (keying, and the read window) | -| `exec-added-processes-and-the-memory-budget` | infrastructure / containers | `7b9e8788` | `host-cgroup-visibility` is cross-pod read mechanics and explicitly routes self-monitoring elsewhere; `resource-limits-and-probes` is manifest authoring. This is a runtime preflight before adding load | - -No new category was created — all five landed in existing categories. - -**Merged into existing pages (5 candidates)** - -| Candidate | Merged into | Shape | -|-----------|-------------|-------| -| `91ef5d53` | `testing-quality-tests-that-cannot-fail` | +1 never-fails row, +1 Instead-of row, +1 source | -| `f189f423` | `testing-quality-source-text-wiring-assertions` | In-place extension of 1 edge row + 1 source bullet (page at the 120-line cap) | -| `bb6d8539` | `backend-common-change-impact-widening-a-closed-value-table` | +Do-this 6 & 7 (incl. a set-difference ruling table), +1 Instead-of row, +1 source | -| `60a817ee` | `qa-process-evaluating-review-feedback` | +2 edge rows, +1 Instead-of row, +1 source | -| `7b9e8788` | `infrastructure-containers-host-cgroup-visibility` | Cross-link from its self-monitoring row to the new page | - -**Dropped — out of layer (2, retired)** - -- `094dedf3` — a Figma MCP `inspect_node` → `get_dev_ready` children-fetch - workaround. The server is a private, org-internal MCP plugin; the behavior is - not publicly verifiable and the directive does not transfer to any other reader. -- `e165a365` — an `/rtb:review` remote-fallback runbook naming - `~/.claude/tools/rtb-remote-review.sh` and an internal pod. The transferable - kernel ("a two-provider review gate degraded to one provider is not a passed - gate") is already the subject of `qa-process-llm-review-pipelines`; what remains - is machine-specific paths. - -**Released back to `pending` (6)** — each needs its own page, not a row, and is -better served by a dedicated pass than by being appended here: -`81dc1f98` (naming the carrier field/type when a plan says "wire A to B"), -`b9ae304a` (`VAR="$(cmd 2>&1)"` mixing stderr into a value used as a path), -`fdd0b3c6` (monitor markers anchored at line start; delta rather than absolute -state; first cycle records a baseline), -`c2adb2be` (positional-order assertions on rendered SQL predicates), -`815e8cb9` (grep only *active* `DATABASE_URL` assignments, and confirm which -dotenv file the tool loads, before a destructive DB command), -`f1146adb` (CI ticket-key extraction scoped by changed-file intersection rather -than by mention). - -## Decision Log - -**Intent.** Drain the harvested `★ Insight` queue into reviewable wiki knowledge -without lowering the wiki's evidence bar. The queue held 21 rows accumulated over -several days; the goal was correct routing and real verification, not a high -ingest count. - -**Alternatives considered and rejected.** - -- *Ingest all 21 in this pass.* Rejected: six of them each need their own page, - and writing six more pages in one pass would have produced thin, weakly-sourced - entries. They are released to `pending`, not dropped, so the next flush takes - them with a full budget. -- *Append the two TypeScript candidates to `call-site-enumeration`.* Rejected: - that page is the sibling case (callers of a changed signature, Python - positional-vs-keyword). AGENTS.md requires one case per page, so the - constructor-enumeration case became its own page, cross-linked both ways. -- *Add a row to `source-text-wiring-assertions` for the empty-slice nuance.* - Rejected: that page is at exactly the documented 120-line body cap, so adding a - line would violate maintenance invariant 5. The nuance was merged **in place** - into an existing edge row instead; body re-measured at 120. -- *Drop the comment-stripping candidate entirely as a duplicate.* Rejected: its - directive is already the page's step 2, but the empty-slice consequence - (vacuous **green**, not the documented noisy red) was genuinely absent. -- *Claim a cross-check exemption because this PR cannot merge itself.* Rejected — - see below; the check found a real error, which is the argument against exempting. -- *Push to `origin`* as the skill's snippet does. Not available: this contributor - has no write access to `choiyounggi/dev-loop` (403). Used the pre-existing - `fork` remote, which is how every prior knowledge branch here was published. -- *Branch name from `git config user.name`.* The skill's ASCII sanitisation of a - Korean name yields an empty string → `anon`, defeating the attribution the - branch name exists for. Used the gh login, matching existing branch names. - -**Where reviewers should look hardest.** - -1. `infrastructure/ci-cd/changed-files-only-gates.md` — rewritten after the - cross-check. The measured table is the load-bearing part; please sanity-check - it against your own Prettier version, since the exit codes are version-visible - behaviour rather than a documented contract. -2. `databases/transactions/application-clock-vs-database-timestamps.md` step 5–6 — - the claim that timestamp order is not commit order, and that the remedy is a - lock/isolation level rather than finer clock resolution. `[추정]` on the MySQL - `NOW()`/`SYSDATE()` row: taken from general MySQL semantics, not fetched this - session like the PostgreSQL pages were. -3. `widening-a-closed-value-table.md` Do-this 6–7 — this inserts a security-shaped - concern (allowlist widening) into a page whose original subject was value - tables. If that reads as two cases, it should be split. -4. The 2 dropped candidates — if you consider private-tooling runbooks in scope - for this wiki, they should be restored rather than retired. - -## Cross-Check - -Independent adversarial pass via `claude` CLI headless (separate process, no -shared context), prompted to refute rather than confirm, over the five new pages' -technical claims. - -**It found a real error, and the page was rewritten because of it.** The reviewer -challenged the claim that `prettier --check` exits 0 on an empty match set, -arguing an unmatched pattern errors by default and that exit-0 belongs to the -ignore-filtered case. I resolved it by measurement rather than by argument — -running all seven cases against Prettier 3.7.4 — and the reviewer was right: -an unmatched operand exits **2** (while still printing the success sentence), -whereas the genuine silent vacuous passes are no-operands, all-ignore-filtered, -and `--ignore-unknown`-with-unsupported-extensions. The page, this report's -rows 3 and 5, and the `log.md` entry were all corrected. - -Verdicts on the other five claim groups: **sound** (PostgreSQL clock semantics — -noted as if anything *understated*; zsh word-splitting; TS contextual typing; -`tsc` program membership incl. `exclude`-does-not-stop-imports; cgroup v2 -`max` vs `oom_kill` and `kubectl exec` cgroup placement). - -Stated limits of the check: the reviewer's sandbox denied it read access to -`~/.dev-loop/repo/wiki`, so it adjudicated the six claims as quoted in its prompt -and could **not** audit (b) whether each `Sources` quote supports the directive it -is cited for, or (c) whether any page contradicts its own edge-case rows. Those -two dimensions remain unreviewed by an independent party and are the residual -risk in this PR. A first attempt also returned only the session's Stop-hook -output rather than a verdict; that run was discarded rather than read as -"no findings". - -## Review notes +No new category or domain was created; every candidate fit an existing category. -- PR-only, as required: no merge, no push to `main`. -- Commit is under the contributor's own ambient git identity - (`최영기 `, gh `dch0202-rsquare`); no assistant identity - and no `Co-Authored-By` trailer. The branch uses the gh login because - sanitizing the Korean `user.name` to ASCII yields an empty string, which the - skill's snippet would have turned into `anon` — that would have defeated the - attribution the branch name exists for. -- Scope purity: only `wiki/**`, four domain indexes, `log.md`, and this report. - Two untracked leftovers from earlier flushes - (`.dev-loop/CROSSCHECK_FINDINGS.md`, `.dev-loop/fold-note-73.md`) were left - untouched and unstaged. +| Candidate | Target | +|---|---| +| 16f22ec2 | infrastructure/agent-orchestration/gate-evidence-exit-code-class.md (new; sits beside session-completion-gates and control-signals-vs-primary-artifacts, the category's other gate/signal-interpretation pages) | +| 3564214d | platforms/tools/jq-dot-rebinding-in-predicates.md (new; `tools` per the bsd-vs-gnu-cli precedent for CLI-tool semantics — not `shells`, which owns quoting/portability) | +| d7a0cae5 | infrastructure/agent-orchestration/worker-reported-plan-contradiction.md (new; queue tag backend overridden — the directive is coordinator/worker process, the sqlite3 fact is illustrative evidence) | +| 048faa67 | debugging/performance/attributing-a-benchmark-speedup.md (new; queue tag qa overridden — qa/deliverables recomputes static counts, this diagnoses why a measured number moved) | +| 28c88c5f | testing/quality/sequential-dispatch-assumption-under-concurrency.md (new; testing/quality over testing/async because the case is a test's validity under a dispatch-model change, and over testing/flaky because the failure is deterministic) | +| 5d88d302 | testing/quality/proving-a-critical-section-is-lock-protected.md (new; same reasoning; debugging/concurrency's intermittent-failures is linked as the diagnose-side counterpart) | diff --git a/log.md b/log.md index e2ee9ca..3745aeb 100644 --- a/log.md +++ b/log.md @@ -104,3 +104,4 @@ Append-only. Format: `## [YYYY-MM-DD] : ` for any non-zero, non-timeout + result; it does not itself separate "the checker never ran" from "the + checker ran and found a defect" — that classification is the reader's job. +2. **Classify by exit code before acting**: + +| Exit code | Meaning | What it tells you | +|-----------|---------|--------------------| +| 127 | command not found | the shell never found/started the program named in `CHECK:` — a broken invocation, not a verdict on the plan | +| 126 | found but not executable | same class as 127 — a permission/exec-bit problem on the invoked file, not a verdict | +| 124 / 142 | timeout (`GATE_CHECK_TIMEOUT`, default 120s, via `gate-check.sh`'s own perl-alarm wrapper) | the command ran but exceeded its budget — not this page's case | +| Any other code the checked script defines (e.g. dev-loop's `plan-gate.sh`: 3 = content defect, 4 = target file/section missing) | the checker ran to completion and returned its own documented failure | this is the only class that means "edit the plan" | + +3. **When exit is 127/126, look for an unresolved variable in the `CHECK:` + line itself before opening the artifact.** dev-loop's + `templates/plan-gates.md` leaves `${CLAUDE_PLUGIN_ROOT}` as a live shell + variable that `gate-check.sh` expands via `bash -c` at execution time. In + a plain, non-Claude-Code shell it is unset, and an unset `${VAR}` with no + `:-`/`:=` default expands to the empty string (POSIX §2.6.2), so + `${CLAUDE_PLUGIN_ROOT}/skills/wiki-plan/scripts/plan-gate.sh` becomes + `/skills/wiki-plan/scripts/plan-gate.sh` — a path that never exists. +4. **Export `CLAUDE_PLUGIN_ROOT` to the plugin's installed root before + re-running outside a Claude Code session**, then re-run + `gate-check.sh --run` unchanged. `CLAUDE_PLUGIN_ROOT` is exported by + Claude Code only into plugin skill/agent content, hook and monitor + commands, and MCP/LSP server fields — not into an arbitrary shell, so a + ledger produced for one context and re-run by hand in another needs the + variable supplied explicitly. +5. **Only exit 1/3 (or whatever the checker's own documented failure code + is) is evidence of a real content defect** — edit the plan/artifact only + then. +6. **Use a mixed result within one ledger as its own diagnostic.** A gate + whose `CHECK:` needs no plugin-rooted script (e.g. dev-loop's + `baseline-tests-ran`, whose command is the plan's own recorded Baseline + command) passing while sibling gates in the same run fail with + `exit=127` isolates the fault to the plugin-path expansion, not to plan + content — read it as confirmation, not as partial progress. + +## Edge cases + +| Case | Then | +|------|------| +| Re-running with `CLAUDE_PLUGIN_ROOT` exported flips every UNMET gate to MET, with zero file edits | The gate identified an invocation bug, not a plan defect — this confirms the exit-127 reading and closes the investigation | +| Only some `exit=127` gates flip after exporting `CLAUDE_PLUGIN_ROOT` | Diff the still-UNMET gates' `CHECK:` lines for a second broken path, or a genuine content defect now visible as exit 3/4 | +| The EVIDENCE line reads `exit=124` or `exit=142` | Timeout, not command-not-found — raise `GATE_CHECK_TIMEOUT` or investigate why the checked command is slow; this page does not cover that case | +| Hand-testing `plan-gate.sh` / `gate-check.sh` directly in a terminal, outside any dev-loop skill invocation | `CLAUDE_PLUGIN_ROOT` will never be auto-set there; export it yourself (pointed at the plugin's cache directory) or invoke the scripts by absolute path instead of relying on the `CHECK:` line's `${CLAUDE_PLUGIN_ROOT}` form | + +## Instead of + +| If you are about to | Do this instead | Why | +|---------------------|-----------------|-----| +| Treat every UNMET gate as a defect in `analysis.md`/`design.md` and start editing | Read the EVIDENCE exit code first; edit content only on the checker's own documented failure code | `exit=127`/`126` means the shell could not even start the checker — editing content it never evaluated does nothing and hides the real (environment) cause | +| Assume a `CHECK:` line's `${CLAUDE_PLUGIN_ROOT}` resolves the same in a plain shell as inside a Claude Code session | Export `CLAUDE_PLUGIN_ROOT` to the plugin's root before running `gate-check.sh`/`plan-gate.sh` outside Claude Code | Claude Code exports `CLAUDE_PLUGIN_ROOT` only into plugin hook/skill/MCP/LSP contexts; a bare `sh`/`bash` invocation never receives it, so the variable silently expands to empty rather than erroring | + +## Sources + +- https://www.gnu.org/software/bash/manual/html_node/Exit-Status.html — "If a command is not found, the child process created to execute it returns a status of 127. If a command is found but is not executable, the return status is 126." +- https://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html — POSIX Shell & Utilities §2.6.2 Parameter Expansion: `${parameter}` substitutes the value of `parameter`, if any; with no `:-`/`:=`/`:?`/`:+` modifier, an unset parameter substitutes nothing +- https://code.claude.com/docs/en/plugins-reference — `${CLAUDE_PLUGIN_ROOT}` is substituted/exported inline in skill and agent content, hook and monitor commands, MCP `stdio`/`http`/`sse`/`ws` server fields, and LSP server fields; no plain-shell or interactive-session export is documented +- Local reproduction 2026-09-10, dev-loop plugin `groundwork/dev-loop` v1.21.0 (`~/.claude/plugins/cache/groundwork/dev-loop/1.21.0`): `templates/plan-gates.md` states `${CLAUDE_PLUGIN_ROOT}` is "left as a live shell variable — gate-check.sh runs CHECK via `bash -c`, which expands it at execution time"; `skills/loop-implement/scripts/gate-check.sh`'s `run_check()` executes `CHECK:` via `/bin/bash -c` and records EVIDENCE as `exit=: ` for any non-zero, non-timeout exit, with no special-casing of 126/127; `skills/wiki-plan/scripts/plan-gate.sh` documents its own exit codes (0 ok, 2 usage, 3 check failed/content defect, 4 target file/section missing) — one layer below the shell-level 126/127 a broken `${CLAUDE_PLUGIN_ROOT}` produces +- Field reproduction (a dev-loop wiki-plan run, 2026-09-09): `.dev-loop/gates/plan-A-t1-normalize.md`, first `gate-check.sh --run` → `met=1 unmet=4`, all four UNMET `EVIDENCE:` lines reading `exit=127: sh: /skills/...: No such file or directory`; identical file re-run with `CLAUDE_PLUGIN_ROOT` exported to the plugin root → `met=5 unmet=0`, no file edited diff --git a/wiki/infrastructure/agent-orchestration/session-completion-gates.md b/wiki/infrastructure/agent-orchestration/session-completion-gates.md index 137c5bc..a24697d 100644 --- a/wiki/infrastructure/agent-orchestration/session-completion-gates.md +++ b/wiki/infrastructure/agent-orchestration/session-completion-gates.md @@ -8,7 +8,7 @@ sources: - https://code.claude.com/docs/en/hooks - https://csf.tools/reference/nist-sp-800-53/r5/ac/ac-5/ last_verified: 2026-08-13 -related: [infrastructure-agent-orchestration-pane-delivery-confirmation, infrastructure-agent-orchestration-worktree-isolated-workers, platforms-processes-tool-diagnostics-without-a-failing-exit-code, infrastructure-agent-orchestration-dispatching-after-a-completion-report] +related: [infrastructure-agent-orchestration-pane-delivery-confirmation, infrastructure-agent-orchestration-worktree-isolated-workers, platforms-processes-tool-diagnostics-without-a-failing-exit-code, infrastructure-agent-orchestration-dispatching-after-a-completion-report, infrastructure-agent-orchestration-gate-evidence-exit-code-class] --- # A Gate That Blocks a Worker Session from Ending Mid-Workflow diff --git a/wiki/infrastructure/agent-orchestration/worker-reported-plan-contradiction.md b/wiki/infrastructure/agent-orchestration/worker-reported-plan-contradiction.md new file mode 100644 index 0000000..1bf3a32 --- /dev/null +++ b/wiki/infrastructure/agent-orchestration/worker-reported-plan-contradiction.md @@ -0,0 +1,77 @@ +--- +id: infrastructure-agent-orchestration-worker-reported-plan-contradiction +domain: infrastructure +category: agent-orchestration +applies_to: [general] +confidence: field-tested +sources: + - https://docs.python.org/3/library/sqlite3.html#exceptions + - https://claude.com/blog/building-multi-agent-systems-when-and-how-to-use-them + - https://www.anthropic.com/engineering/multi-agent-research-system + - http://principles-wiki.net/principles:don_t_repeat_yourself +last_verified: 2026-09-10 +related: [infrastructure-agent-orchestration-autonomous-decision-rulings, infrastructure-agent-orchestration-shared-run-state, debugging-methodology-hypothesis-testing, qa-process-evaluating-review-feedback] +--- + +# A Worker Reports the Design Doc and Step Files Disagree + +## When this applies + +A coordinator authored both a design doc and per-task step files for the same +piece of work, and a worker implementing a step reports that the two disagree +on a concrete, checkable fact — a value, an exception type, an API shape, a +threshold — not a matter of taste. + +## Do this + +1. Treat the worker's contradiction report as a hypothesis to test, not a + deviation to correct by re-asserting the design doc. That the design doc + was written by the stronger model, or written first, is not evidence about + which document matches the working code. +2. Make the change the disputed reading requires, in the worker's own + worktree, and run the affected test suite. The suite is ground truth + neither document can substitute for. +3. Read any failures for what they assert about the running system, not about + either document — a test that pins specific behavior (e.g. a bare + exception type propagating) tells you which document's claim the codebase + actually depends on. +4. Patch whichever document is wrong once confirmed, and record which + document changed and why in the run's ledger + ([infrastructure-agent-orchestration-autonomous-decision-rulings]), so the + correction is visible to every other task still reading the stale copy. +5. When the worker misread a document rather than the documents actually + disagreeing, correct the worker's step instead — the test run decided it, + not who asserted first or loudest. + +| Signal | Do | +|--------|----| +| A concrete, testable disagreement (a value, a type, an exception class, a threshold) | Run the disputed change and the suite before ruling either way | +| The dispute is over widening or narrowing a catch clause, condition, or table | Check the language's own exception/type hierarchy first — a broadened catch can silently swallow a narrower failure a test already asserts | +| The suite passes under both readings | The documents are underspecified, not contradictory in a way code can arbitrate — add a test that pins the intended behavior, then patch the wrong doc | +| The disagreement is architectural or subjective and the plan never claimed to test it | Escalate it as a plan defect for the requester to rule on, with both readings quoted, instead of forcing it through the suite — the suite cannot arbitrate a claim the plan never made testable | + +## Edge cases + +| Case | Then | +|------|------| +| The worker is the one who is wrong (misread a document, ran the wrong test target) | Correct the worker's step, cite the passing baseline that proves it, and leave both documents alone | +| Both documents are wrong — neither matches the codebase's actual constraint | Patch both, and record one ruling listing both edits so a reader of either document sees the correction | +| No test covers the disputed behavior either way | Write the missing test first instead of guessing, from behavior the codebase already depends on elsewhere, then apply steps 2-4 | +| The contradiction is about an external library's behavior the codebase merely calls (which exceptions a driver raises, a default a client library sets) | Confirm against that library's own documentation too — a test run shows what the current version does, not the documented contract a future upgrade must keep honoring | +| The step file and design doc are the same physical document, with no split | This page does not apply — the drift the trigger describes cannot occur here | + +## Instead of + +| If you are about to | Do this instead | Why | +|---------------------|-----------------|-----| +| Re-assert the design doc because the coordinator wrote it on the stronger model | Run the disputed change and the test suite before ruling | A plan split across two documents drifts between them, and the worker executing both is the first reader to hit the seam; model strength is not evidence about which artifact matches the code | +| Tell the worker to "just follow the step file" or "just follow the design doc" without checking | Test both readings against the suite and patch whichever document is wrong | Overriding without verification leaves the same drift live for the next worker who reads the other document | +| Widen a catch clause because the design doc's exception table looks incomplete | Check the language's exception hierarchy for what the broader class already includes | `except sqlite3.Error` also catches `IntegrityError` and `OperationalError` — both are subclasses of `DatabaseError`, itself a subclass of `Error` — so a catch written to cover one silently absorbs a failure a test was written against the other | + +## Sources + +- https://docs.python.org/3/library/sqlite3.html#exceptions — `sqlite3.IntegrityError` and `sqlite3.OperationalError` are both subclasses of `sqlite3.DatabaseError`, itself a subclass of `sqlite3.Error`; `except sqlite3.Error` catches all of them +- https://claude.com/blog/building-multi-agent-systems-when-and-how-to-use-them — verification-subagent pattern: "You MUST run the complete test suite before marking as passed... Specify 'Run the full test suite and report all failures' rather than 'make sure it works'"; resolves disputes through executable validation rather than trusting either side's assertion +- https://www.anthropic.com/engineering/multi-agent-research-system — orchestrator-worker pattern: the lead agent "synthesizes these results and decides whether more research is needed," treating a subagent's report as new evidence to act on rather than trusting its own prior plan as final +- http://principles-wiki.net/principles:don_t_repeat_yourself — DRY as single source of truth: "if there is a single source of truth, there is only one place where changes have to be applied. Then the representations cannot diverge" — a design doc and step files stating the same fact twice is exactly the setup this principle warns drifts apart +- Field evidence (a dev-loop orchestrate run, one worker task, 2026-09-09): a step file said `except sqlite3.Error` while the design doc's table said `OperationalError` only. Running the widened catch in the worker's worktree produced `2 failed, 242 passed` — `test_insert_company_rejects_normalized_duplicate_names` and `test_init_db_preserves_existing_duplicate_rows` both failed with `DatabaseError` wrapping `IntegrityError` — confirming the design doc's table, not the step file, was the document to patch diff --git a/wiki/infrastructure/config/path-valued-config.md b/wiki/infrastructure/config/path-valued-config.md index 8f322fb..5bf594a 100644 --- a/wiki/infrastructure/config/path-valued-config.md +++ b/wiki/infrastructure/config/path-valued-config.md @@ -9,7 +9,7 @@ sources: - https://developer.apple.com/library/archive/documentation/MacOSX/Conceptual/BPSystemStartup/Chapters/CreatingLaunchdJobs.html - https://12factor.net/config last_verified: 2026-08-04 -related: [infrastructure-config-environment-config, platforms-processes-background-services, platforms-environment-path-resolution, backend-python-boundaries-runtime-validation, backend-node-boundaries-runtime-validation] +related: [infrastructure-config-environment-config, platforms-processes-background-services, platforms-environment-path-resolution, backend-python-boundaries-runtime-validation, backend-node-boundaries-runtime-validation, infrastructure-agent-orchestration-gate-evidence-exit-code-class] --- # A Config Value That Is a Filesystem Path diff --git a/wiki/infrastructure/index.md b/wiki/infrastructure/index.md index 86c2ac3..7ca3c4b 100644 --- a/wiki/infrastructure/index.md +++ b/wiki/infrastructure/index.md @@ -16,11 +16,13 @@ Match your situation to a "load when" line; load only matching pages. | [shared-run-state](agent-orchestration/shared-run-state.md) | Several agent/worker sessions coordinate through files in one repository (status directory, briefs, escalations, claim files); choosing the path layout for that state; starting an orchestration in a repo that may already have one running; a watcher woke on a task id it did not create; the default branch moved during a run; a coordinator is about to reset a task's status file while re-delivering a prompt | | [pane-delivery-confirmation](agent-orchestration/pane-delivery-confirmation.md) | An orchestrator drives another program through a terminal multiplexer (`tmux send-keys` + `capture-pane`) and must decide whether the input was consumed, retry, or escalate; a pane diff is being used as delivery evidence; the target echoes but never runs the input; deciding *where* in a pane capture to search for a collapsed paste marker whose position depends on payload size, or what to report when the pane's input-box chrome cannot be located at all | | [session-completion-gates](agent-orchestration/session-completion-gates.md) | Writing a Stop/completion hook that blocks a worker session from ending while its phase is non-terminal; the gate fires on a worker that followed its own prompt; deciding the terminal phase set, the unknown-phase default, and how the gate bounds its own repetition; you are the worker the gate repeats on at an instructed pause and are deciding whether to advance your phase to silence it | +| [gate-evidence-exit-code-class](agent-orchestration/gate-evidence-exit-code-class.md) | A dev-loop gate-check run (`gate-check.sh --run`, `plan-gate.sh check`) reports gates UNMET with an EVIDENCE line reading `exit=127`/`exit=126`; deciding whether to edit the checked plan/artifact or the gate's own invocation before re-running; a `CHECK:` line embeds `${CLAUDE_PLUGIN_ROOT}` run outside a Claude Code plugin context where that variable is unset; one gate needing no plugin-rooted script passes while sibling gates in the same ledger fail with exit=127 | | [dispatching-after-a-completion-report](agent-orchestration/dispatching-after-a-completion-report.md) | A worker reported completion and the orchestrator wants to hand that same terminal or runtime slot its next task; a start/dispatch call fails with a runtime-unavailable-class error moments after a completion report; a task reached a terminal `failed` status with no worker having worked on it; deciding a settled dispatch's next owner (transfer, release, or retain) and how to retry a failed start without spending the task's attempt budget | | [unattended-worker-questions](agent-orchestration/unattended-worker-questions.md) | A worker agent raises a question through its own interactive UI (a numbered chooser, a confirmation/trust/re-auth screen) with no human at that terminal; a worker is flagged stalled with a live terminal and no task-level error; a worker reports a decision it assumed rather than asked; designing the channel a worker uses to ask its coordinator for a decision | | [usage-limit-paused-workers](agent-orchestration/usage-limit-paused-workers.md) | Several workers billed to one account go quiet within minutes of each other while every liveness check passes; a worker's terminal shows a `You've hit your session/weekly/Opus limit · resets …` notice; deciding whether to restart, replace, or wait on a worker with no task-level error; writing the prompt that resumes a worker after a usage window resets | | [worktree-isolated-workers](agent-orchestration/worktree-isolated-workers.md) | Authoring the brief/output contract for parallel workers each confined to its own git worktree; workers stall at the same phase with no task-level error; deciding where shared or produced artifacts live and which direction (read vs write) a worktree guardrail stops; a guardrail escalates on read-only access to another worktree; the isolation guard is a Bash-command hook while workers also edit files with native Edit/Write tools | | [autonomous-decision-rulings](agent-orchestration/autonomous-decision-rulings.md) | An unattended agent hits a decision its plan does not answer and must choose between stopping to ask and proceeding; a run stalls on questions no human needed to see; deciding which decision categories require a human; recording autonomous decisions for audit; resuming after interruption/compaction without re-dispatching completed work | +| [worker-reported-plan-contradiction](agent-orchestration/worker-reported-plan-contradiction.md) | A coordinator's design doc and per-task step files disagree on a concrete, checkable fact and a worker reports it; deciding whether to re-assert the design doc or verify empirically; a widened/narrowed catch clause, condition, or table is the disputed change; the worker may have misread a document instead of the documents actually disagreeing | | [session-context-token-budget](agent-orchestration/session-context-token-budget.md) | Planning or running long-lived coordinator/worker agent sessions and deciding when to compact or clear context; a run's cost is dominated by cache reads; screenshots or large file reads are entering a long-lived session; choosing slot counts / per-phase token budgets for an orchestrated run | | [code-graph-as-orientation-layer](agent-orchestration/code-graph-as-orientation-layer.md) | A repository carries a locally built code knowledge graph (graphify `graphify-out/graph.json` or similar) and an agent is about to plan, decompose, or estimate the blast radius of a change; an orchestrator needs each parallel task's file set before dispatch; deciding whether a graph hit can stand as plan evidence; checking whether the graph is fresh enough to use | diff --git a/wiki/platforms/environment/path-resolution.md b/wiki/platforms/environment/path-resolution.md index 2bb8bc2..8e966ab 100644 --- a/wiki/platforms/environment/path-resolution.md +++ b/wiki/platforms/environment/path-resolution.md @@ -12,7 +12,7 @@ sources: - https://www.sudo.ws/docs/man/sudoers.man/ - https://docs.brew.sh/FAQ last_verified: 2026-08-04 -related: [platforms-toolchains-version-management, platforms-processes-background-services, platforms-shells-env-var-off-switches, platforms-toolchains-compiler-sysroot-on-macos] +related: [platforms-toolchains-version-management, platforms-processes-background-services, platforms-shells-env-var-off-switches, platforms-toolchains-compiler-sysroot-on-macos, infrastructure-agent-orchestration-gate-evidence-exit-code-class] --- # The Wrong Binary (or None) Resolving From PATH diff --git a/wiki/platforms/index.md b/wiki/platforms/index.md index 8200fae..3d41cfd 100644 --- a/wiki/platforms/index.md +++ b/wiki/platforms/index.md @@ -29,6 +29,7 @@ Match your situation to a "load when" line; load only matching pages. | [agent-permission-classifier-denials](tools/agent-permission-classifier-denials.md) | A model-based permission gate (Claude Code auto mode) denied an action that is correct for the task — a push, a config write, a deploy command; writing the permission or `autoMode` config itself is refused as self-modification; choosing the settings scope a classifier actually reads; designing an automation whose steps a classifier will judge, and where the consent sentence has to sit | | [bsd-vs-gnu-cli](tools/bsd-vs-gnu-cli.md) | A command works on Linux but fails on macOS or vice versa (`date`, `sed -i`, `timeout`, `seq`, `grep -P`, `readlink`, `stat`); writing a script or CI step that must run on both userlands; deciding whether to install GNU coreutils on macOS or write POSIX-only | | [harness-mediated-tool-results](tools/harness-mediated-tool-results.md) | A plugin or hook in your agent harness returned substitute content for a built-in tool (truncated read, redaction, a note telling you to call something else); deciding whether a short result is interception or a genuinely small/empty file; the hook's suggested workaround failed too; briefing spawned worker sessions about a known-degraded tool | +| [jq-dot-rebinding-in-predicates](tools/jq-dot-rebinding-in-predicates.md) | A jq filter reads `$arr | index(.)` (or `contains(.)`) inside `any(gen; cond)`, `map(select(...))`, or any place a pipe sits between a generator and a membership test, and the predicate matches inputs it should reject; writing or reviewing a jq membership/containment test that must bind the generated element before piping it against another array; deciding between `. as $x | $arr | index($x)` and `IN($arr[])` | | [plugin-mcp-server-registration](tools/plugin-mcp-server-registration.md) | A plugin-bundled MCP server is missing from `/mcp` or its tools are uncallable while the plugin's skills work; deciding whether the fault is registration, config shape, or the server itself; about to change a plugin `.mcp.json` (bare map vs `mcpServers` wrapper, `command`/`args`/`env`) to make a server appear; relaying a shell variable into a server with `${VAR}` and the server reads the wrong value; a plugin updated mid-session and its server still runs the old path | | [version-keyed-artifact-cache](tools/version-keyed-artifact-cache.md) | Shipping a code update to a distribution system that caches artifacts by a version string (a Claude Code marketplace plugin, or any tag-pinned cache) and the update runs but the old behavior persists; deciding why `/plugin update` reports "at latest" yet new code never runs; locating and clearing a stale `~/.claude/plugins/cache////` | | [unpacked-extension-source-reload](tools/unpacked-extension-source-reload.md) | You edited a file of an unpacked Chromium extension (content script, service worker, manifest) and are about to judge the change in the browser, or the edit appears to have no effect; deciding which of extension-reload and host-page-reload a given file requires; adding a running-build version marker; automating the `chrome://extensions` reload control from an agent or CDP script | diff --git a/wiki/platforms/shells/escapes-in-shell-string-literals.md b/wiki/platforms/shells/escapes-in-shell-string-literals.md index a3e28fd..07e2922 100644 --- a/wiki/platforms/shells/escapes-in-shell-string-literals.md +++ b/wiki/platforms/shells/escapes-in-shell-string-literals.md @@ -8,7 +8,7 @@ sources: - https://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html - https://www.gnu.org/software/bash/manual/bash.html#Double-Quotes last_verified: 2026-08-04 -related: [platforms-shells-portable-shell-scripts, platforms-shells-command-text-inspected-before-execution] +related: [platforms-shells-portable-shell-scripts, platforms-shells-command-text-inspected-before-execution, platforms-tools-jq-dot-rebinding-in-predicates] --- # Backslash Escapes Inside a Shell String Literal Holding a Regex or Pattern diff --git a/wiki/platforms/shells/portable-shell-scripts.md b/wiki/platforms/shells/portable-shell-scripts.md index 58b9371..9ffe5f2 100644 --- a/wiki/platforms/shells/portable-shell-scripts.md +++ b/wiki/platforms/shells/portable-shell-scripts.md @@ -13,7 +13,7 @@ sources: - https://www.gnu.org/software/bash/manual/html_node/Double-Quotes.html - https://www.gnu.org/software/bash/manual/html_node/Shell-Parameter-Expansion.html last_verified: 2026-08-05 -related: [platforms-tools-bsd-vs-gnu-cli, platforms-toolchains-version-management, platforms-shells-command-text-inspected-before-execution, platforms-shells-escapes-in-shell-string-literals, platforms-shells-env-var-off-switches, platforms-shells-unset-versus-empty-parameters, platforms-shells-option-like-argument-values, platforms-processes-tool-diagnostics-without-a-failing-exit-code, testing-quality-completion-predicates] +related: [platforms-tools-bsd-vs-gnu-cli, platforms-toolchains-version-management, platforms-shells-command-text-inspected-before-execution, platforms-shells-escapes-in-shell-string-literals, platforms-shells-env-var-off-switches, platforms-shells-unset-versus-empty-parameters, platforms-shells-option-like-argument-values, platforms-processes-tool-diagnostics-without-a-failing-exit-code, testing-quality-completion-predicates, platforms-tools-jq-dot-rebinding-in-predicates] --- # Shell Scripts That Must Run on More Than One Machine or Shell diff --git a/wiki/platforms/tools/jq-dot-rebinding-in-predicates.md b/wiki/platforms/tools/jq-dot-rebinding-in-predicates.md new file mode 100644 index 0000000..b7e09b0 --- /dev/null +++ b/wiki/platforms/tools/jq-dot-rebinding-in-predicates.md @@ -0,0 +1,68 @@ +--- +id: platforms-tools-jq-dot-rebinding-in-predicates +domain: platforms +category: tools +applies_to: [general] +confidence: verified +sources: + - https://jqlang.org/manual/ + - https://stackoverflow.com/questions/48898983/jq-index-1-not-working-when-element-is-an-array +last_verified: 2026-09-10 +related: [platforms-shells-portable-shell-scripts, platforms-shells-escapes-in-shell-string-literals, testing-quality-tests-that-cannot-fail] +--- + +# A jq Predicate That Pipes an Array Into `index(.)` Inside a Generator + +## When this applies + +Writing a jq filter that tests "does the current generated element appear in +array `$arr`" — inside `any(gen; cond)`, `map(select(...))`, `reduce`, or any +condition downstream of a generator — and the test is written as +`$arr | index(.)`, `$arr | contains(.)`, or any other `$arr | fn(.)` form where +`.` is meant to still be the generated element. + +## Do this + +1. **Bind the generated element to a variable before piping into the array.** + `. as $x | $arr | index($x) != null`. This is the fix, not a stylistic + preference: once bound, `$x` keeps the generator's element regardless of + what `.` becomes later in the pipeline. +2. **Or skip `index()` entirely and use `IN($arr[])`.** `IN(s)` tests `.` + (unchanged, since no pipe sits between the generator and `IN`) against every + value `s` produces — `any(.[]; IN($arr[]))`. +3. **Know the one rule that causes the bug.** Per the jq manual, "`.` is the + input value at the particular stage in a pipeline" — `.a | . | .b` is the + same as `.a.b`, because each `|` re-evaluates what `.` means for everything + to its right. `$arr | index(.)` pipes `$arr` into `index`, so for the + duration of that call `.` **is** `$arr`, not the outer generator's element — + the filter is actually `$arr | index($arr)`. +4. **Know why that produces a false positive instead of an error.** `index(s)` + on an array input searches for `s` as a **subsequence** (subarray), not + single-element membership — `[0,1,2,3] | index([1,2])` finds the subarray at + its start position. Any non-empty array is trivially a subsequence of + itself at position `0`, so `$arr | index($arr)` returns `0` for every + non-empty `$arr`, and `0 != null` is `true` — the predicate is + unconditionally true, not merely wrong for one case. + +## Edge cases + +| Case | Then | +|------|------| +| `$arr` is empty | `$arr | index($arr)` returns `null`, so `!= null` is `false` — the bug is silent until `$arr` gets its first element, so a test seeded only with an empty array will not catch it | +| You need the position, not just membership | Still bind first: `. as $x | $arr | index($x)` — the position of `$x`, or `null` if absent | +| Testing membership of a whole array (not element) against a list of arrays | `index(.)`'s self-match trap does not apply here — decide the real case (element-in-array vs array-in-list-of-arrays) before choosing `index`/`IN`, since both take different argument shapes | +| Filter is nested two pipes deep from the generator (`$arr | foo | index(.)`) | The same rebinding happens at the last `|` before `index(.)` regardless of how many pipes precede it — bind the element with `as $x` at the generator, not at the final stage | + +## Instead of + +| If you are about to | Do this instead | Why | +|---------------------|-----------------|-----| +| Write `$arr | index(.)` (or `contains(.)`) as a generator's condition | `. as $x | $arr | index($x) != null` | The pipe rebinds `.` to `$arr` itself before `index` runs, so the call becomes a self-comparison, not a membership test against the generated element | +| Trust that a wrong `index(.)` predicate will fail loudly | Reproduce the exact filter standalone against a case where the expected answer is `false` (an element known absent from `$arr`) before trusting it in a guard | The bug returns `true` unconditionally once `$arr` is non-empty, so a guard built on it silently accepts every input it was meant to reject — it never throws or returns an obviously wrong shape | + +## Sources + +- https://jqlang.org/manual/ — "`.` is the input value at the particular stage in a 'pipeline'... `.a | . | .b` is the same as `.a.b`" (pipe rebinding); `index(s)`/`rindex(s)` "Outputs the index of the first/last occurrence of `s` in the input", demonstrated on array input as a subsequence search (`index([1,2])` on `[0,1,2,3,1,4,2,5,1,2,6,7]` → `1`); `IN(s)` "outputs `true` if `.` appears in the given stream"; `any(generator; condition)` applies `condition` to every output of `generator`. Confirmed unchanged between the jq 1.6 and current (1.8) manual pages for `index`/`rindex`. +- https://stackoverflow.com/questions/48898983/jq-index-1-not-working-when-element-is-an-array — "index takes a subsequence of elements to find, with a scalar being a special case equivalent to a one element sequence", explaining why indexing an array by an array differs from indexing by a scalar. +- Local reproduction 2026-09-10 (jq-1.7.1-apple, `jq --version`): `["a","b"] as $arr | ["b","zzz"] | any(.[]; $arr | index(.) != null)` → `true` even though `"zzz"` is not in `$arr` (and `["zzz","yyy"]` alone, with nothing in `$arr`, also → `true`; an empty `$arr` → `false`); `. as $x | $arr | index($x) != null` and `any(.[]; IN($arr[]))` both correctly discriminate (`true` for `"b"`, `false` for `"zzz"`/`"yyy"`); `["a","b"] | index(["a","b"])` → `0` confirms the self-match subsequence mechanics. +- Field context: dev-loop `skills/orchestrate/scripts/graph-drop.sh:67` (plugin v1.21.0) builds a consumers check as `select(any(.consumes[]?; $outs | index(.) != null))` — this exact form, reproduced standalone, named six unrelated tasks as consumers of an output none of them referenced, because the predicate was unconditionally true for any non-empty `$outs`. diff --git a/wiki/testing/flaky/diagnosing-flaky-tests.md b/wiki/testing/flaky/diagnosing-flaky-tests.md index bd2b5c6..6113542 100644 --- a/wiki/testing/flaky/diagnosing-flaky-tests.md +++ b/wiki/testing/flaky/diagnosing-flaky-tests.md @@ -9,7 +9,7 @@ sources: - https://testing.googleblog.com/2016/05/flaky-tests-at-google-and-how-we.html - https://testing.googleblog.com/2017/04/where-do-our-flaky-tests-come-from.html last_verified: 2026-07-10 -related: [testing-data-test-data-and-isolation, testing-mocking-what-to-mock, debugging-concurrency-intermittent-failures] +related: [testing-data-test-data-and-isolation, testing-mocking-what-to-mock, debugging-concurrency-intermittent-failures, testing-quality-sequential-dispatch-assumption-under-concurrency] --- # Fixing a Test That Fails Intermittently diff --git a/wiki/testing/index.md b/wiki/testing/index.md index 4e65ac9..3638ba5 100644 --- a/wiki/testing/index.md +++ b/wiki/testing/index.md @@ -53,6 +53,8 @@ Match your situation to a "load when" line; load only matching pages. | [source-text-wiring-assertions](quality/source-text-wiring-assertions.md) | A test reads a source file as a string and asserts by regex that a call is present (cleanup in every handler, logging after each branch, teardown in each exit path) because the behavior has no reachable seam; such a guard is green while one of the call sites is gone; choosing between a count assertion, an order anchor, and a function-body slice; such a guard (a negative `not.toMatch`, or a token-count assertion) reddens on correct code and a comment is the only thing that changed | | [history-dependent-checks-on-shallow-clones](quality/history-dependent-checks-on-shallow-clones.md) | A test or gate resolves git history (`log --diff-filter`, `merge-base`, `rev-list`) and may run under a shallow CI checkout (actions/checkout default `fetch-depth: 1`); a history query reported every tracked file as added by one commit; deciding between a `--is-shallow-repository` skip-guard and deepening the fetch | | [expectation-sets-with-one-distinct-value](quality/expectation-sets-with-one-distinct-value.md) | Locking that a computed value is really carried into a response/DTO/event and every test case expects the same literal for that field; a constant-substitution mutant survives a green suite and you are about to add more assertions; choosing between a delete probe and a value-substitution probe for wiring; auditing fixtures for one-distinct-value expectation sets | +| [sequential-dispatch-assumption-under-concurrency](quality/sequential-dispatch-assumption-under-concurrency.md) | A previously-sequential per-item dispatch loop is being parallelized (ThreadPoolExecutor or equivalent) and an existing test asserts an exact call count or per-item state after a mid-run stop flag/exception fires inside one item's processing; deciding whether a "stop after item N" test's item count is ≤ the real concurrency cap; verifying a migrated test empirically at the real default cap rather than a synthetic cap=1 | +| [proving-a-critical-section-is-lock-protected](quality/proving-a-critical-section-is-lock-protected.md) | Writing a test meant to prove a shared-mutable-state critical section (e.g. `with lock: shared[key] += n`) is actually lock-protected under concurrent writers; a many-thread/many-iteration stress test passes whether or not the lock is present; choosing between raw stress, an injected-delay test double, and manual mutation to validate such a test | ## data diff --git a/wiki/testing/quality/checks-that-cannot-pass.md b/wiki/testing/quality/checks-that-cannot-pass.md index 2aae3e8..5f851a8 100644 --- a/wiki/testing/quality/checks-that-cannot-pass.md +++ b/wiki/testing/quality/checks-that-cannot-pass.md @@ -10,7 +10,7 @@ sources: - https://docs.semgrep.dev/writing-rules/testing-rules - https://docs.pytest.org/en/stable/reference/exit-codes.html last_verified: 2026-08-06 -related: [testing-quality-tests-that-cannot-fail, testing-quality-minimum-case-set, backend-common-change-impact-call-site-enumeration, platforms-processes-tool-diagnostics-without-a-failing-exit-code, qa-process-scope-purity-checks, testing-quality-history-dependent-checks-on-shallow-clones] +related: [testing-quality-tests-that-cannot-fail, testing-quality-minimum-case-set, backend-common-change-impact-call-site-enumeration, platforms-processes-tool-diagnostics-without-a-failing-exit-code, qa-process-scope-purity-checks, testing-quality-history-dependent-checks-on-shallow-clones, infrastructure-agent-orchestration-gate-evidence-exit-code-class] --- # Validating a Check Whose Target Does Not Exist Yet diff --git a/wiki/testing/quality/harness-reverse-controls.md b/wiki/testing/quality/harness-reverse-controls.md index ec3394d..02e70b8 100644 --- a/wiki/testing/quality/harness-reverse-controls.md +++ b/wiki/testing/quality/harness-reverse-controls.md @@ -13,7 +13,7 @@ sources: - https://testing.googleblog.com/2021/04/mutation-testing.html - https://docs.python.org/3/library/unittest.mock.html last_verified: 2026-08-04 -related: [testing-quality-tests-that-cannot-fail, testing-quality-minimum-case-set, testing-quality-schema-additions-under-a-golden-gate, testing-quality-differential-run-agreement, testing-quality-completion-predicates, testing-quality-unasserted-return-fields, testing-quality-value-preserving-refactor-assertions, testing-quality-stale-artifact-baselines, backend-python-language-bytecode-cache-staleness, qa-exploratory-override-control-pairs, testing-quality-surviving-mutant-equivalence-triage, testing-quality-spec-artifact-checks] +related: [testing-quality-tests-that-cannot-fail, testing-quality-minimum-case-set, testing-quality-schema-additions-under-a-golden-gate, testing-quality-differential-run-agreement, testing-quality-completion-predicates, testing-quality-unasserted-return-fields, testing-quality-value-preserving-refactor-assertions, testing-quality-stale-artifact-baselines, backend-python-language-bytecode-cache-staleness, qa-exploratory-override-control-pairs, testing-quality-surviving-mutant-equivalence-triage, testing-quality-spec-artifact-checks, debugging-performance-attributing-a-benchmark-speedup] --- # Citing a Verification Harness's Own Score diff --git a/wiki/testing/quality/proving-a-critical-section-is-lock-protected.md b/wiki/testing/quality/proving-a-critical-section-is-lock-protected.md new file mode 100644 index 0000000..c4fe673 --- /dev/null +++ b/wiki/testing/quality/proving-a-critical-section-is-lock-protected.md @@ -0,0 +1,85 @@ +--- +id: testing-quality-proving-a-critical-section-is-lock-protected +domain: testing +category: quality +applies_to: [general, python] +confidence: verified +sources: + - https://docs.python.org/3/library/sys.html#sys.setswitchinterval + - https://docs.python.org/3/faq/library.html#what-kinds-of-global-value-mutation-are-thread-safe + - https://testing.googleblog.com/2021/04/mutation-testing.html + - https://docs.python.org/3/howto/free-threading-python.html +last_verified: 2026-09-10 +related: [testing-quality-tests-that-cannot-fail, testing-quality-harness-reverse-controls, debugging-concurrency-intermittent-failures, backend-python-concurrency-gil-and-concurrency-model, backend-common-concurrency-shared-state-and-pools, testing-quality-sequential-dispatch-assumption-under-concurrency] +--- + +# Proving a Lock-Protected Critical Section Actually Needs Its Lock + +## When this applies + +Writing a test meant to prove a shared-mutable-state critical section (e.g. +`with lock: shared[key] += n`) is actually lock-protected under concurrent +writers; a many-thread/many-iteration stress test passes whether or not the +lock is present; auditing whether an existing concurrency test can fail. + +## Do this + +1. **Treat raw thread-count/iteration-count stress as insufficient on its own.** A plain + in-memory read-modify-write (`d[k] += 1`) rarely straddles a GIL thread + switch: reproduced 2026-09-10 (CPython 3.14.6, GIL enabled), 16 threads × + 10 increments with the lock removed reached the correct total of 160 in + 20/20 trials — 0 races detected. The window is real (per + [backend-python-concurrency-gil-and-concurrency-model], the GIL only + guarantees one thread runs bytecode *at a time*, not that a whole + statement is atomic — the Python FAQ lists `D[x] = D[x] + 1` explicitly as + non-atomic) but too narrow for stress to reliably land in it. +2. **Widen the window deterministically with a test double**, not a longer + stress run. Wrap the shared container in a subclass whose mutating method + sleeps inside the read-modify-write span — e.g. a dict subclass whose + `__setitem__` calls `time.sleep(0.001)` before writing. Reproduced same + session: 16 threads × 10 increments through that double, lock removed, + reached only 10 (never 160) in 10/10 trials; with `with lock:` restored, + reached 160 in 10/10 trials. +3. **Validate the test itself by mutation before trusting it** — the same + discipline as [testing-quality-tests-that-cannot-fail] step 1: remove the + lock from the production code, rerun, require red; restore the lock, + rerun, require green. A concurrency test that has not been through this + cycle is unproven regardless of how it reads. +4. Choose the technique by what you are proving: + +| Situation | Do | +|-----------|-----| +| Confirming *some* concurrent-safety exists, order not important | Raw stress (many threads × iterations) is enough — cheap, but cannot prove a missing lock would be caught | +| Proving a specific critical section needs its lock, and the mutation (lock removed) must reliably fail | Injected-delay test double widening the critical section, per step 2 | +| Either technique, before citing the test as evidence | Mutation check (step 3) — a test that cannot go red under a real mutation is not evidence either way | + +5. **Size the sleep against the switch interval and the test budget.** CPython's + default thread-switch interval is 5 ms (`sys.getswitchinterval()`); a + per-write sleep at or above that order of magnitude reliably yields the + GIL mid-critical-section. Keep total added time bounded — + `threads × iterations × sleep`: 16 × 10 × 1 ms ≈ 160 ms per trial above, + fast enough to run on every CI invocation rather than only on demand. + +## Edge cases + +| Case | Then | +|------|------| +| Target runs on a CPython 3.13+ free-threaded (no-GIL) build | The race window is real concurrent execution, not just a switch — the injected-delay double still applies and becomes *more* reliable at finding races, since there is no GIL serializing the interpreter loop at all ([backend-python-concurrency-gil-and-concurrency-model] free-threaded row) | +| The mutated operation is one the Python FAQ already lists as atomic (`L.append(x)`, `D[x] = y`, `x = y`) | Skip the delay double — there is no read-modify-write span to widen, and a lock around an already-atomic op is not what the test needs to catch | +| Sleep is too short relative to thread count/iterations | The window narrows back toward stress-only odds; raise the per-write sleep before adding more threads | +| Sleep is too long relative to the CI time budget | Reduce iterations per thread rather than the per-write sleep — cutting the sleep reopens the gap that made the test unable to fail | +| Critical section already uses a C-level atomic primitive (`itertools.count().__next__`, an `atomics`-style library, a DB's atomic increment) | No Python-level lock or injected delay is the right target — verify atomicity against that primitive's own documentation instead | + +## Instead of + +| If you are about to | Do this instead | Why | +|---------------------|-----------------|-----| +| Rely on raw thread-count/iteration-count stress alone to prove a critical section needs its lock | Inject an artificial delay inside the critical section via a test double (e.g. a dict subclass whose `__setitem__` sleeps) so the read-modify-write span reliably overlaps across threads | A 16-thread/160-increment stress test with no injected delay passed 100% of the time (20/20) even with the lock completely removed from production code; only the injected delay made the same broken code fail deterministically (10 vs 160) | + +## Sources + +- https://docs.python.org/3/library/sys.html#sys.setswitchinterval — the thread switch interval controls how often CPython considers switching threads between bytecode instructions; its default order of magnitude sizes the injected sleep +- https://docs.python.org/3/faq/library.html#what-kinds-of-global-value-mutation-are-thread-safe — `D[x] = D[x] + 1` and `i = i+1` are explicitly listed as non-atomic; `D[x] = y`, `L.append(x)`, `x = y` are atomic and need no widening +- https://testing.googleblog.com/2021/04/mutation-testing.html — a test proves detection only if it fails when the guarded behavior is removed; the mutation-check discipline this page applies specifically to concurrency tests +- https://docs.python.org/3/howto/free-threading-python.html — the 3.13+ free-threaded (GIL-disabled) build changes the underlying concurrency model +- Local reproduction 2026-09-10 (CPython 3.14.6, GIL enabled, `sys._is_gil_enabled()` → True): plain `dict` `+=`, 16 threads × 10 increments, no lock, 20/20 trials landed on 160 (0 races detected); the same shape through a `__setitem__`-sleeping (1 ms) dict subclass, no lock, 10/10 trials landed on 10; with `with lock:` restored, 10/10 trials landed on 160 diff --git a/wiki/testing/quality/sequential-dispatch-assumption-under-concurrency.md b/wiki/testing/quality/sequential-dispatch-assumption-under-concurrency.md new file mode 100644 index 0000000..72eac5a --- /dev/null +++ b/wiki/testing/quality/sequential-dispatch-assumption-under-concurrency.md @@ -0,0 +1,70 @@ +--- +id: testing-quality-sequential-dispatch-assumption-under-concurrency +domain: testing +category: quality +applies_to: [general] +confidence: verified +sources: + - https://docs.python.org/3/library/concurrent.futures.html + - https://martinfowler.com/articles/nonDeterminism.html + - https://testing.googleblog.com/2017/04/where-do-our-flaky-tests-come-from.html +last_verified: 2026-09-10 +related: [testing-quality-tests-that-cannot-fail, testing-flaky-diagnosing-flaky-tests, backend-common-concurrency-shared-state-and-pools, backend-python-concurrency-gil-and-concurrency-model, testing-quality-proving-a-critical-section-is-lock-protected] +--- + +# A Sequential-Dispatch Test's Exact-Count Assertion After the Loop Becomes Concurrent + +## When this applies + +A previously-sequential per-item dispatch loop (`for item in items: process(item)`) +is being parallelized (`ThreadPoolExecutor`/equivalent), and an existing test +sets a flag or raises from inside one item's processing mid-run and then +asserts an exact call count or exact per-item state — a "stop after item N" +or "only N items ran" style assertion. + +## Do this + +1. **Before trusting such a test still passes under the new concurrent code, + run it with the real default (or production-configured) concurrency cap — + not a synthetic `max_workers=1`.** A cap-of-1 test double forces strict + one-at-a-time dispatch, which is exactly the sequential assumption the + migration invalidated; green under cap=1 proves nothing about the + concurrent path and hides the bug. +2. **Flag any such test whose item count is ≤ the real cap as a likely + sequential-dispatch assumption.** `Executor.submit()` schedules the + callable and returns immediately — it does not wait for earlier + submissions to start or finish. When `max_workers` ≥ the item count, every + item's task is created and can pass a start-of-task "check the flag" gate + before any one item's side effect (setting the flag, raising) becomes + visible to it, because submission for later items races ahead of the + earlier item's in-progress work rather than waiting on it. +3. Resolve by item-count-vs-cap, not by tightening the assertion in place: + +| Case | Do | +|------|----| +| Item count ≤ real cap; the test's intent is to verify the stop-check logic itself | Pin that test's own executor to `max_workers=1` deliberately and name it as verifying the guard in isolation from scheduling (e.g. `test_stop_check_logic_sequential`) — this is a legitimate unit test of the predicate, not a claim about concurrent behavior | +| Item count ≤ real cap; the test's intent is to verify behavior under real concurrent dispatch | Rewrite the assertion to what non-blocking submission actually guarantees: all items already submitted before the flag was observably set will run — assert "≤ cap items ran, and the stop is honored only for items submitted in a later batch" (or, if the loop submits every item up front in one batch, assert that all of them ran and the flag only prevents a *second* round of submissions) | +| Item count > real cap | The cap genuinely throttles dispatch — an exact-count assertion can still hold once submission is restructured to check the flag before each `submit()` call, not only inside each task | + +## Edge cases + +| Case | Then | +|------|------| +| Migrating test doubles as coverage for `shutdown(cancel_futures=True)` draining | `cancel_futures=True` cancels only futures the executor has not started running; already-running futures complete regardless. Assert against "started" count, not "submitted" count | +| The stop condition is an exception raised from item processing rather than an explicit flag write | The exception surfaces only when `.result()` is called on that specific future; other already-submitted tasks are unaffected by it. Assert per-future outcomes individually rather than assuming the dispatch loop halts on the first exception | +| The test currently passes only because its fixture hardcodes `max_workers=1` | That configuration is not evidence the migrated code is correct — it forces exactly the sequential order the migration removed; still run once at the real cap per step 1 before merging | + +## Instead of + +| If you are about to | Do this instead | Why | +|---------------------|-----------------|-----| +| Trust a migrated "stop after item N" test because it is green under a synthetic `max_workers=1` fixture | Re-run it under the real default/production cap before merging | cap=1 forces strict one-at-a-time dispatch — the exact sequential assumption the migration invalidated; green there says nothing about the concurrent path | +| Leave a pre-existing exact-call-count test unexamined after swapping a sequential loop for a thread pool | Compare its item count to the real concurrency cap; when count ≤ cap, split or rewrite it per the decision table | Under `max_workers` ≥ item count, `submit()` returns immediately, so every task can pass a start-of-task flag check before an earlier item's side effect lands | + +## Sources + +- https://docs.python.org/3/library/concurrent.futures.html — `Executor.submit()` "schedules the callable... and returns a Future" (non-blocking); `ThreadPoolExecutor` default `max_workers` is `min(32, (os.process_cpu_count() or 1) + 4)` (3.13+); `shutdown(cancel_futures=True)` cancels only futures "the executor has not started running" — running/completed futures are unaffected +- https://martinfowler.com/articles/nonDeterminism.html — tests that encode a hidden ordering/timing assumption pass under the old execution model and fail once that assumption is removed; poll/assert the actual guaranteed condition instead of an incidental one +- https://testing.googleblog.com/2017/04/where-do-our-flaky-tests-come-from.html — threading/concurrency changes are a measured source of tests whose pass/fail depends on an execution-order assumption the test never states +- Field reproduction, 2026-09-10 (CPython 3.14.6, `concurrent.futures.ThreadPoolExecutor`): 3 tasks, `max_workers=4`, task for `i==1` sets a `threading.Event` before running; each task checks the event at its own top and returns early if set. Result: all 3 tasks ran (`ran: [0, 1, 2]`) — the flag never took effect, because all 3 were submitted and passed their gate check before task 1's side effect became visible. The identical harness with `max_workers=1` produced `ran: [0, 1]`, `skipped-2` — strict sequential dispatch is what made the original assertion true, not the stop logic +- Field observation, 2026-09-10 (a Python per-company contact pipeline, `tests/test_resume.py::TestGracefulShutdown::test_shutdown_stops_after_current_company`, 3 companies, default `MAX_COMPANY_WORKERS=4`): passed under the prior sequential-loop dispatch; went deterministically red (`call_count` 3 vs expected 1) the moment per-company dispatch moved to `ThreadPoolExecutor` — reproduced on repeated runs via `sh scripts/run_tests.sh` (265/266 passing, only this test failing), not a flake