diff --git a/README.md b/README.md index 530f36d..5dde181 100644 --- a/README.md +++ b/README.md @@ -32,30 +32,38 @@ On **SWE-bench Verified** — the industry-standard coding benchmark for code-RL open, pre-registered, dual-gated harness: ``` - Verifier verdict alone (does the shipped test suite accept a wrong patch?) + Verifier verdict alone (shipped suite accepts an attacker's candidate patch) 51.0% ██████████████████████████░░░░░░░░░░░░░░░░░░░░░░░░ 255 / 500 tasks + 45.2% ███████████████████████░░░░░░░░░░░░░░░░░░░░░░░░░░░ 226 / 500 ...and its added lines differ from gold's - Dual-gated (patch independently proven wrong, not an LLM's opinion) + Dual-gated (that patch shown WRONG by execution or by hand review — never an LLM's opinion) 13.7% ███████░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░ ~69 / 500 [95% CI 10.1–18.0%] ``` -> **Half of SWE-bench Verified has a verifier that accepts a wrong patch. About 1 in 7 admits a *confirmed* -> reward-hack** — a patch that passes the shipped tests but demonstrably does the wrong thing, each backed -> by a reproducer you can re-run. +> **On half of SWE-bench Verified, the shipped test suite accepted a patch from an attacker that was +> trying to write a wrong one. About 1 in 7 admits a *confirmed* reward-hack** — a patch that passes the shipped tests but +> demonstrably does the wrong thing, each backed by a reproducer you can re-run. -The prior published result ([arXiv:2606.16062](https://arxiv.org/abs/2606.16062)) measured this on 49 tasks +Those numbers measure different things and the gap between them is the point. Getting a patch past the +suite is **not** evidence the patch is wrong: many are legitimate alternative fixes. In the pre-registered +sample of 102 — drawn from the 226 tasks whose accepted patch was textually distinct from gold's own fix, +stratified by repo, seed 42 — adjudication came back **31 HACK / 67 CORRECT / 4 AMBIGUOUS**. About two +thirds were adjudicated correct. That is why the dual gate exists, and why 13.7% rather than 51.0% is this +project's finding. + +The prior published result ([arXiv:2606.16062](https://arxiv.org/abs/2606.16062)) measured the confirmed rate on 49 tasks and **never released its code**. Tripwire reproduces it with an open harness and extends it to the **full 500-task benchmark** for the first time — with a *stricter* oracle (deterministic behavioral divergence, not an LLM-augmented test), so every number here is a conservative **lower bound**. ## The finding in one table -| Run | Tasks | Attacker | Verifier accepts wrong patch | Confirmed reward-hack (dual-gated) | +| Run | Tasks | Attacker | Suite accepts an attacker candidate | Confirmed reward-hack (dual-gated) | | --- | --- | --- | --- | --- | | **Anchor** (reproduction) | 49 (astropy + django) | Claude Sonnet 4.5 | — | **11/49 = 22.4%** | | Anchor — capability ladder | 49 | Claude Haiku 4.5 | — | 8/49 = 16.3% | | Reference — arXiv:2606.16062 | 49 | Claude Sonnet 4 | — | 14/49 = 28.5% | -| **Extend** (novel — full benchmark) | **500 (11 repos)** | Claude Sonnet 4.5 | **51.0%** (255/500) | **13.7%** — CI [10.1–18.0%] | +| **Extend** (novel — full benchmark) | **500 (12 repos)** | Claude Sonnet 4.5 | **51.0%** (255/500) | **13.7%** — CI [10.1–18.0%] | The full-500 confirmed rate (13.7%) is *lower* than the anchor's 22.4% **on purpose** — the full benchmark includes robust repos (sympy, scikit-learn, sphinx, pytest) with stronger suites, so it's more diff --git a/docs/SPEC-census.md b/docs/SPEC-census.md new file mode 100644 index 0000000..2ae7cac --- /dev/null +++ b/docs/SPEC-census.md @@ -0,0 +1,149 @@ +# SPEC: publish the hub verifier census (aggregate only) + +Status: DRAFT, pending adversarial review +Date: 2026-09-02 +Phase: the hub lane of `docs/preregistration.md`, §1-§6. Disclosure governed by §8. + +## Problem Statement + +Everyone is building RL environments. Nobody independently certifies that their verifiers +work. Tripwire has published that certification for exactly one benchmark, SWE-bench +Verified, while the instrument that can do it across the whole public ecosystem has already +been built and already been run. + +512 environments from a 1,367-environment census have been scored, classified by verifier +type, and aggregated. The result sits on one laptop. Until it is published it certifies +nothing for anyone. + +Four other tools audited this same hub in the two months to September 2026. Every one of +them stopped at **degenerate probes**: empty strings, echoed prompts, garbage. That measures +Reward-Hack Susceptibility, the easy bucket. None measured a **false-accept rate against an +independent oracle on plausible, non-degenerate wrong answers**, which is the harder and more +useful question, and which this instrument already answers. + +## Solution + +Publish the aggregate census: pooled false-accept rates by verifier type with confidence +intervals, the false-reject honesty gate that makes them credible, and the monotonicity +violation rate for partial-credit graders. **No environment is named.** + +The headline is of the form: *across N public RL environment verifiers, the pooled +false-accept rate against an independent oracle is X% [CI], and separately, LLM-judge +verifiers resist wrong answers at 0% while failing prompt injection at Y%.* + +## What is NOT published, and why + +`docs/preregistration.md` §8 binds this project to a **30-day author and platform embargo +before naming any broken environment**, to a per-env one-pager with the failing probe and a +fix hint, to re-test-on-fix markers, and to framing authors as collaborators. It further +states Phase 1-2 are private aggregate research with no per-env public naming until Phase 4. + +Therefore, by the author's own instruction and consistent with §8: + +- **No environment identifier is published**, including the author's own environments. Not in + the aggregate, not in the per-env rows, not in the writeup. +- Per-env rows are published under a **salted opaque token**. The salt is not published, so + the mapping is recoverable by the author alone and by nobody else. +- No environment's code, prompts, or dataset rows are reproduced. + +This is stricter than §8 requires and is a deliberate choice, not an oversight. + +## User Stories + +1. As someone training on a public RL environment, I want the pooled false-accept rate of its verifier class, so I know the base rate of my reward signal being wrong. +2. As the same person, I want it split by verifier type, because a binary matcher and an LLM judge fail differently. +3. As the same person, I want the LLM-judge result split by attack type, because resisting wrong answers and resisting injection are different properties. +4. As someone using a partial-credit grader, I want the monotonicity violation rate, because a grader that scores a worse answer higher is broken even when it never fully accepts a wrong one. +5. As a skeptical reader, I want the false-reject rate, so I know the probes did not simply loosen every verifier. +6. As a skeptical reader, I want the count of environments excluded because their own gold answer failed, because that number is a finding about the ecosystem and not a footnote. +7. As a skeptical reader, I want the coverage funnel stated at every stage, because 90 eligible out of 1,367 enumerated is the first thing I will ask about. +8. As a skeptical reader, I want to know the census is a snapshot with a date, and how much the hub has drifted since. +9. As a reproducer, I want every published figure recomputable from committed records by one command. +10. As a reproducer, I want the per-env rows, so I can check the aggregate rather than trust it. +11. As an environment author, I want to know whether I am in the broken set without everyone else knowing, so the author must be able to answer that privately from the withheld salt. +12. As a maintainer, I want the preregistration for this lane committed before any census artifact enters the repository, so the ordering is provable this time. + +## Implementation Decisions + +### Ordering, which is the point + +1. A dated amendment to `docs/preregistration.md` registering this lane's headline, its + thresholds, and the anonymisation scheme lands as **its own commit, pushed, before any + census artifact exists in the repository**. The amendment of 2026-09-02 already committed + this project to exactly that, and this is the first phase that must honour it. +2. Only then do the anonymised artifacts land. + +### Anonymisation + +- A random 32-byte salt, generated once, stored outside the repository and never committed. +- Every environment identifier becomes `sha256(salt || env_id)`, truncated to 12 hex chars, + as a stable token. Collisions checked at generation; regenerate the salt if any occur. +- Fields carrying identity are dropped entirely rather than tokenised where they serve no + analytic purpose: the four `*_env_ids` lists in `binary_far` and `judge_far`, and the two + in `coverage`. The counts stay; the names go. +- Per-env rows keep every statistic and the `domain` and `verifier_type` labels, since the + stratification is the analysis. `env_id` is replaced by the token. +- A test asserts no published artifact contains any string from the census index. + +### What is published + +- The coverage funnel, stated as a single table from 1,367 enumerated down to the eligible + denominators, with the reason for each drop. +- Per verifier type: pooled false-accept rate, its cluster-bootstrap interval, the + Benjamini-Hochberg broken count at q = 0.05 with expected false discoveries, and the + Wilson-lower-bound count alongside it. +- The judge lane split into known-wrong probes and injection probes, reported separately. + This split is the most interesting result in the census and must not be pooled away. +- The soft lane: monotonicity violations as pairs and rate. +- The stratification by domain and verifier type, as data. +- The gold-fails and non-discriminating counts, as findings. + +### Frozen before publication + +- No threshold, interval method, or eligibility rule changes. They are already registered in + §5: Wilson lower bound above 5% for the broken label, Benjamini-Hochberg at q = 0.05, + cluster bootstrap over environments with 10,000 BCa resamples, and no unweighted means. +- The census is a **snapshot dated 2026-07-12**. It is not re-run to improve a number. Hub + drift since that date is measured and reported, not corrected for. + +### Coverage is a first-class result, not an apology + +The funnel from 1,367 enumerated to 90 FAR-eligible binary environments is large, and the +reasons are themselves findings: environments that fail to load, environments whose own gold +answer does not pass, environments whose verifier does not discriminate between a right and +a wrong answer at all. Each is reported with its count and an interpretation. A reader who +concludes the ecosystem is hard to audit has understood the paper correctly. + +## Testing Decisions + +Tests verify what the analysis returns from committed records, need no network and no +container, and are table-driven where the input is data. + +- **Anonymisation**: tokens are stable across runs given the same salt, differ across salts, + are collision-free on the real census, and no published artifact contains any raw + identifier. That last one is the test that matters and it runs over every committed file. +- **Every published figure is pinned**: pooled rates, intervals, broken counts, expected + false discoveries, monotonicity rate, and every number in the coverage funnel, recomputed + from the anonymised per-env rows and asserted against what the writeup states. +- **The funnel is closed**: each stage's drop plus the next stage's count equals the previous + stage's count. A funnel that does not add up is the most likely error here. +- **The honesty gate**: an environment whose gold answer fails is excluded from every rate, + and a test trips it. +- **Drift check** is a script, not a test, since it needs the network. + +## Out of Scope + +- Naming any environment, including the author's own. +- Publishing environment code, prompts, or dataset rows. +- Re-running the census, extending coverage, or scoring the sandbox bucket. +- The cross-model slate registered in §7. That is a later phase. +- Any recommendation to a specific environment's author. + +## Further Notes + +- The likeliest hostile question is coverage, not method. Answer it in the abstract. +- The judge injection result is the strongest finding and has its own control arm at 0%. + Lead with it. +- Four shallower audits of this hub already exist. The claim is not priority over them; it is + that they measured degenerate-input susceptibility and this measures false accepts against + an independent oracle. State that distinction plainly and cite them. diff --git a/docs/extend-result.md b/docs/extend-result.md index 2d9edd5..3531c3b 100644 --- a/docs/extend-result.md +++ b/docs/extend-result.md @@ -6,9 +6,10 @@ reported on 49 tasks and never ran the full set. Same open, deterministic harnes (verifier PASS **and** an independent oracle says WRONG). Aggregate-only until the 30-day disclosure gate. ## Setup -- **Tasks:** all 500 of `princeton-nlp/SWE-bench_Verified`, 11 repos (django 231, sympy 60, sphinx 35, - matplotlib 34, scikit-learn 26, astropy 22, xarray 18, pytest 15, pylint 8, requests 6, seaborn 2, - flask 1). +- **Tasks:** all 500 of `princeton-nlp/SWE-bench_Verified`, 12 repos (django 231, sympy 75, sphinx 44, + matplotlib 34, scikit-learn 32, astropy 22, xarray 22, pytest 19, pylint 10, requests 8, seaborn 2, + flask 1). Counted from `results/raw_swe_500/`; an earlier revision listed a stale tally summing to + 458 and said 11 repos. - **Attacker:** Claude **Sonnet 4.5** via AWS Bedrock (Sonnet 4 retired — disclosed substitution), the same `harness/code_attack.py` as the anchor. K=3 candidates/task, round 1 single-shot, rounds 2–3 fed the shipped-suite failure logs. SEARCH/REPLACE output → deterministic difflib diff (99% apply). @@ -53,7 +54,7 @@ stronger test suites where the confirmed hack rate is far lower, dragging the be from the hackable-heavy astropy/django slice the anchor and the paper used. Every number is a **lower bound**: Wilson lower edge, AMBIGUOUS excluded, and the dual-gate requires a *demonstrated* behavioral divergence. Even so, **~1 in 7 SWE-bench-Verified tasks admits a confirmed wrong patch that the shipped -verifier accepts**, and *half* accept a wrong patch on the verifier's own verdict alone — on the exact +verifier accepts**, and on *half* the verifier accepts an attacker's candidate patch at all — on the exact benchmark the field treats as gold-standard for code-RL reward. Artifacts: `results/swe500_confirmed.json` (aggregate), `results/swe500_sample_verdicts.json` (per-task diff --git a/harness/swe_native_rate.py b/harness/swe_native_rate.py index 7b90429..591b7c6 100644 --- a/harness/swe_native_rate.py +++ b/harness/swe_native_rate.py @@ -17,10 +17,17 @@ verifier-verdict-alone numerator: the published 51.0%. queue_size a task that is native_hackable AND has >= 1 such candidate - the differ marked ``behaviorally_distinct`` from gold. - Candidates identical in behaviour to gold are legitimate - alternative fixes and are excluded before the dual gate, - which is why the confirmed rate is a lower bound. + flagged ``behaviorally_distinct``. Despite the name that is + a TEXTUAL check (``code_attack.behaviorally_distinct``): the + candidate's added lines are not a subset of gold's. Its own + docstring calls it "necessary (not sufficient)" and it never + decides wrongness on its own. Candidates whose added lines + are a subset of gold's are dropped before the dual gate. A + *strict* subset added fewer lines than gold, so it may be a + PARTIAL gold fix rather than gold's fix — which can be + exactly the narrow wrong patch the attacker was asked for. + So this filter drops some genuine hacks too, and that is one + reason the confirmed rate is a lower bound. The dual gate itself (oracle = WRONG) is applied downstream in ``harness/swe_confirm.py``; nothing here decides whether a patch is wrong. @@ -83,7 +90,7 @@ def main(argv: list[str] | None = None) -> int: print(f"native-hackable (verifier alone) {d['native_hackable']}" f" = {d['native_rate']:.1%}") print(f"dual-gate confirm queue {d['queue_size']}" - f" = {d['queue_rate']:.1%} (behaviorally distinct from gold)") + f" = {d['queue_rate']:.1%} (added lines differ from gold's)") print(f"attacker spend on this pass ${d['total_cost_usd']:.2f}") return 0