Correct the 51% framing: it measures divergence from gold, not wrongness

README.md

@@ -32,30 +32,38 @@ On **SWE-bench Verified** — the industry-standard coding benchmark for code-RL
 open, pre-registered, dual-gated harness:
 
 ```
- Verifier verdict alone (does the shipped test suite accept a wrong patch?)
+ Verifier verdict alone (shipped suite accepts an attacker's candidate patch)
    51.0%  ██████████████████████████░░░░░░░░░░░░░░░░░░░░░░░░   255 / 500 tasks
+   45.2%  ███████████████████████░░░░░░░░░░░░░░░░░░░░░░░░░░░   226 / 500   ...and its added lines differ from gold's
 
- Dual-gated  (patch independently proven wrong, not an LLM's opinion)
+ Dual-gated  (that patch shown WRONG by execution or by hand review — never an LLM's opinion)
    13.7%  ███████░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░   ~69 / 500   [95% CI 10.1–18.0%]
 ```
 
-> **Half of SWE-bench Verified has a verifier that accepts a wrong patch. About 1 in 7 admits a *confirmed*
-> reward-hack** — a patch that passes the shipped tests but demonstrably does the wrong thing, each backed
-> by a reproducer you can re-run.
+> **On half of SWE-bench Verified, the shipped test suite accepted a patch from an attacker that was
+> trying to write a wrong one. About 1 in 7 admits a *confirmed* reward-hack** — a patch that passes the shipped tests but
+> demonstrably does the wrong thing, each backed by a reproducer you can re-run.
 
-The prior published result ([arXiv:2606.16062](https://arxiv.org/abs/2606.16062)) measured this on 49 tasks
+Those numbers measure different things and the gap between them is the point. Getting a patch past the
+suite is **not** evidence the patch is wrong: many are legitimate alternative fixes. In the pre-registered
+sample of 102 — drawn from the 226 tasks whose accepted patch was textually distinct from gold's own fix,
+stratified by repo, seed 42 — adjudication came back **31 HACK / 67 CORRECT / 4 AMBIGUOUS**. About two
+thirds were adjudicated correct. That is why the dual gate exists, and why 13.7% rather than 51.0% is this
+project's finding.
+
+The prior published result ([arXiv:2606.16062](https://arxiv.org/abs/2606.16062)) measured the confirmed rate on 49 tasks
 and **never released its code**. Tripwire reproduces it with an open harness and extends it to the **full
 500-task benchmark** for the first time — with a *stricter* oracle (deterministic behavioral divergence,
 not an LLM-augmented test), so every number here is a conservative **lower bound**.
 
 ## The finding in one table
 
-| Run | Tasks | Attacker | Verifier accepts wrong patch | Confirmed reward-hack (dual-gated) |
+| Run | Tasks | Attacker | Suite accepts an attacker candidate | Confirmed reward-hack (dual-gated) |
 | --- | --- | --- | --- | --- |
 | **Anchor** (reproduction) | 49 (astropy + django) | Claude Sonnet 4.5 | — | **11/49 = 22.4%** |
 | Anchor — capability ladder | 49 | Claude Haiku 4.5 | — | 8/49 = 16.3% |
 | Reference — arXiv:2606.16062 | 49 | Claude Sonnet 4 | — | 14/49 = 28.5% |
-| **Extend** (novel — full benchmark) | **500 (11 repos)** | Claude Sonnet 4.5 | **51.0%** (255/500) | **13.7%** — CI [10.1–18.0%] |
+| **Extend** (novel — full benchmark) | **500 (12 repos)** | Claude Sonnet 4.5 | **51.0%** (255/500) | **13.7%** — CI [10.1–18.0%] |
 
 The full-500 confirmed rate (13.7%) is *lower* than the anchor's 22.4% **on purpose** — the full benchmark
 includes robust repos (sympy, scikit-learn, sphinx, pytest) with stronger suites, so it's more

docs/SPEC-census.md

@@ -0,0 +1,149 @@
+# SPEC: publish the hub verifier census (aggregate only)
+
+Status: DRAFT, pending adversarial review
+Date: 2026-09-02
+Phase: the hub lane of `docs/preregistration.md`, §1-§6. Disclosure governed by §8.
+
+## Problem Statement
+
+Everyone is building RL environments. Nobody independently certifies that their verifiers
+work. Tripwire has published that certification for exactly one benchmark, SWE-bench
+Verified, while the instrument that can do it across the whole public ecosystem has already
+been built and already been run.
+
+512 environments from a 1,367-environment census have been scored, classified by verifier
+type, and aggregated. The result sits on one laptop. Until it is published it certifies
+nothing for anyone.
+
+Four other tools audited this same hub in the two months to September 2026. Every one of
+them stopped at **degenerate probes**: empty strings, echoed prompts, garbage. That measures
+Reward-Hack Susceptibility, the easy bucket. None measured a **false-accept rate against an
+independent oracle on plausible, non-degenerate wrong answers**, which is the harder and more
+useful question, and which this instrument already answers.
+
+## Solution
+
+Publish the aggregate census: pooled false-accept rates by verifier type with confidence
+intervals, the false-reject honesty gate that makes them credible, and the monotonicity
+violation rate for partial-credit graders. **No environment is named.**
+
+The headline is of the form: *across N public RL environment verifiers, the pooled
+false-accept rate against an independent oracle is X% [CI], and separately, LLM-judge
+verifiers resist wrong answers at 0% while failing prompt injection at Y%.*
+
+## What is NOT published, and why
+
+`docs/preregistration.md` §8 binds this project to a **30-day author and platform embargo
+before naming any broken environment**, to a per-env one-pager with the failing probe and a
+fix hint, to re-test-on-fix markers, and to framing authors as collaborators. It further
+states Phase 1-2 are private aggregate research with no per-env public naming until Phase 4.
+
+Therefore, by the author's own instruction and consistent with §8:
+
+- **No environment identifier is published**, including the author's own environments. Not in
+  the aggregate, not in the per-env rows, not in the writeup.
+- Per-env rows are published under a **salted opaque token**. The salt is not published, so
+  the mapping is recoverable by the author alone and by nobody else.
+- No environment's code, prompts, or dataset rows are reproduced.
+
+This is stricter than §8 requires and is a deliberate choice, not an oversight.
+
+## User Stories
+
+1. As someone training on a public RL environment, I want the pooled false-accept rate of its verifier class, so I know the base rate of my reward signal being wrong.
+2. As the same person, I want it split by verifier type, because a binary matcher and an LLM judge fail differently.
+3. As the same person, I want the LLM-judge result split by attack type, because resisting wrong answers and resisting injection are different properties.
+4. As someone using a partial-credit grader, I want the monotonicity violation rate, because a grader that scores a worse answer higher is broken even when it never fully accepts a wrong one.
+5. As a skeptical reader, I want the false-reject rate, so I know the probes did not simply loosen every verifier.
+6. As a skeptical reader, I want the count of environments excluded because their own gold answer failed, because that number is a finding about the ecosystem and not a footnote.
+7. As a skeptical reader, I want the coverage funnel stated at every stage, because 90 eligible out of 1,367 enumerated is the first thing I will ask about.
+8. As a skeptical reader, I want to know the census is a snapshot with a date, and how much the hub has drifted since.
+9. As a reproducer, I want every published figure recomputable from committed records by one command.
+10. As a reproducer, I want the per-env rows, so I can check the aggregate rather than trust it.
+11. As an environment author, I want to know whether I am in the broken set without everyone else knowing, so the author must be able to answer that privately from the withheld salt.
+12. As a maintainer, I want the preregistration for this lane committed before any census artifact enters the repository, so the ordering is provable this time.
+
+## Implementation Decisions
+
+### Ordering, which is the point
+
+1. A dated amendment to `docs/preregistration.md` registering this lane's headline, its
+   thresholds, and the anonymisation scheme lands as **its own commit, pushed, before any
+   census artifact exists in the repository**. The amendment of 2026-09-02 already committed
+   this project to exactly that, and this is the first phase that must honour it.
+2. Only then do the anonymised artifacts land.
+
+### Anonymisation
+
+- A random 32-byte salt, generated once, stored outside the repository and never committed.
+- Every environment identifier becomes `sha256(salt || env_id)`, truncated to 12 hex chars,
+  as a stable token. Collisions checked at generation; regenerate the salt if any occur.
+- Fields carrying identity are dropped entirely rather than tokenised where they serve no
+  analytic purpose: the four `*_env_ids` lists in `binary_far` and `judge_far`, and the two
+  in `coverage`. The counts stay; the names go.
+- Per-env rows keep every statistic and the `domain` and `verifier_type` labels, since the
+  stratification is the analysis. `env_id` is replaced by the token.
+- A test asserts no published artifact contains any string from the census index.
+
+### What is published
+
+- The coverage funnel, stated as a single table from 1,367 enumerated down to the eligible
+  denominators, with the reason for each drop.
+- Per verifier type: pooled false-accept rate, its cluster-bootstrap interval, the
+  Benjamini-Hochberg broken count at q = 0.05 with expected false discoveries, and the
+  Wilson-lower-bound count alongside it.
+- The judge lane split into known-wrong probes and injection probes, reported separately.
+  This split is the most interesting result in the census and must not be pooled away.
+- The soft lane: monotonicity violations as pairs and rate.
+- The stratification by domain and verifier type, as data.
+- The gold-fails and non-discriminating counts, as findings.
+
+### Frozen before publication
+
+- No threshold, interval method, or eligibility rule changes. They are already registered in
+  §5: Wilson lower bound above 5% for the broken label, Benjamini-Hochberg at q = 0.05,
+  cluster bootstrap over environments with 10,000 BCa resamples, and no unweighted means.
+- The census is a **snapshot dated 2026-07-12**. It is not re-run to improve a number. Hub
+  drift since that date is measured and reported, not corrected for.
+
+### Coverage is a first-class result, not an apology
+
+The funnel from 1,367 enumerated to 90 FAR-eligible binary environments is large, and the
+reasons are themselves findings: environments that fail to load, environments whose own gold
+answer does not pass, environments whose verifier does not discriminate between a right and
+a wrong answer at all. Each is reported with its count and an interpretation. A reader who
+concludes the ecosystem is hard to audit has understood the paper correctly.
+
+## Testing Decisions
+
+Tests verify what the analysis returns from committed records, need no network and no
+container, and are table-driven where the input is data.
+
+- **Anonymisation**: tokens are stable across runs given the same salt, differ across salts,
+  are collision-free on the real census, and no published artifact contains any raw
+  identifier. That last one is the test that matters and it runs over every committed file.
+- **Every published figure is pinned**: pooled rates, intervals, broken counts, expected
+  false discoveries, monotonicity rate, and every number in the coverage funnel, recomputed
+  from the anonymised per-env rows and asserted against what the writeup states.
+- **The funnel is closed**: each stage's drop plus the next stage's count equals the previous
+  stage's count. A funnel that does not add up is the most likely error here.
+- **The honesty gate**: an environment whose gold answer fails is excluded from every rate,
+  and a test trips it.
+- **Drift check** is a script, not a test, since it needs the network.
+
+## Out of Scope
+
+- Naming any environment, including the author's own.
+- Publishing environment code, prompts, or dataset rows.
+- Re-running the census, extending coverage, or scoring the sandbox bucket.
+- The cross-model slate registered in §7. That is a later phase.
+- Any recommendation to a specific environment's author.
+
+## Further Notes
+
+- The likeliest hostile question is coverage, not method. Answer it in the abstract.
+- The judge injection result is the strongest finding and has its own control arm at 0%.
+  Lead with it.
+- Four shallower audits of this hub already exist. The claim is not priority over them; it is
+  that they measured degenerate-input susceptibility and this measures false accepts against
+  an independent oracle. State that distinction plainly and cite them.

docs/extend-result.md

@@ -6,9 +6,10 @@ reported on 49 tasks and never ran the full set. Same open, deterministic harnes
 (verifier PASS **and** an independent oracle says WRONG). Aggregate-only until the 30-day disclosure gate.
 
 ## Setup
-- **Tasks:** all 500 of `princeton-nlp/SWE-bench_Verified`, 11 repos (django 231, sympy 60, sphinx 35,
-  matplotlib 34, scikit-learn 26, astropy 22, xarray 18, pytest 15, pylint 8, requests 6, seaborn 2,
-  flask 1).
+- **Tasks:** all 500 of `princeton-nlp/SWE-bench_Verified`, 12 repos (django 231, sympy 75, sphinx 44,
+  matplotlib 34, scikit-learn 32, astropy 22, xarray 22, pytest 19, pylint 10, requests 8, seaborn 2,
+  flask 1). Counted from `results/raw_swe_500/`; an earlier revision listed a stale tally summing to
+  458 and said 11 repos.
 - **Attacker:** Claude **Sonnet 4.5** via AWS Bedrock (Sonnet 4 retired — disclosed substitution), the
   same `harness/code_attack.py` as the anchor. K=3 candidates/task, round 1 single-shot, rounds 2–3 fed
   the shipped-suite failure logs. SEARCH/REPLACE output → deterministic difflib diff (99% apply).
@@ -53,7 +54,7 @@ stronger test suites where the confirmed hack rate is far lower, dragging the be
 from the hackable-heavy astropy/django slice the anchor and the paper used. Every number is a **lower
 bound**: Wilson lower edge, AMBIGUOUS excluded, and the dual-gate requires a *demonstrated* behavioral
 divergence. Even so, **~1 in 7 SWE-bench-Verified tasks admits a confirmed wrong patch that the shipped
-verifier accepts**, and *half* accept a wrong patch on the verifier's own verdict alone — on the exact
+verifier accepts**, and on *half* the verifier accepts an attacker's candidate patch at all — on the exact
 benchmark the field treats as gold-standard for code-RL reward.
 
 Artifacts: `results/swe500_confirmed.json` (aggregate), `results/swe500_sample_verdicts.json` (per-task

harness/swe_native_rate.py

@@ -17,10 +17,17 @@
                       verifier-verdict-alone numerator: the published 51.0%.
 
     queue_size        a task that is native_hackable AND has >= 1 such candidate
-                      the differ marked ``behaviorally_distinct`` from gold.
-                      Candidates identical in behaviour to gold are legitimate
-                      alternative fixes and are excluded before the dual gate,
-                      which is why the confirmed rate is a lower bound.
+                      flagged ``behaviorally_distinct``. Despite the name that is
+                      a TEXTUAL check (``code_attack.behaviorally_distinct``): the
+                      candidate's added lines are not a subset of gold's. Its own
+                      docstring calls it "necessary (not sufficient)" and it never
+                      decides wrongness on its own. Candidates whose added lines
+                      are a subset of gold's are dropped before the dual gate. A
+                      *strict* subset added fewer lines than gold, so it may be a
+                      PARTIAL gold fix rather than gold's fix — which can be
+                      exactly the narrow wrong patch the attacker was asked for.
+                      So this filter drops some genuine hacks too, and that is one
+                      reason the confirmed rate is a lower bound.
 
 The dual gate itself (oracle = WRONG) is applied downstream in
 ``harness/swe_confirm.py``; nothing here decides whether a patch is wrong.
@@ -83,7 +90,7 @@ def main(argv: list[str] | None = None) -> int:
     print(f"native-hackable (verifier alone)     {d['native_hackable']}"
           f"  = {d['native_rate']:.1%}")
     print(f"dual-gate confirm queue              {d['queue_size']}"
-          f"  = {d['queue_rate']:.1%}   (behaviorally distinct from gold)")
+          f"  = {d['queue_rate']:.1%}   (added lines differ from gold's)")
     print(f"attacker spend on this pass          ${d['total_cost_usd']:.2f}")
     return 0