From f0e4e9395d2b81af16b07bc36debba20a1b33b14 Mon Sep 17 00:00:00 2001 From: Vedant Patel Date: Wed, 2 Sep 2026 17:37:30 -0700 Subject: [PATCH 1/4] docs: the 51% measures divergence from gold, not wrongness The README said 'Half of SWE-bench Verified has a verifier that accepts a wrong patch'. The repo's own adjudication contradicts it: of the 102 sampled patches the suite accepted and that differed from gold, the verdicts are 31 HACK / 67 CORRECT / 4 AMBIGUOUS. About two thirds were legitimate alternative fixes, not wrong patches. native_hackable means the shipped suite resolved an attacker candidate. It carries no claim about correctness; establishing that is what the dual gate is for, and it yields 13.7%, not 51.0%. Corrects four sites (chart label, pull-quote, table header, extend-result) to say the suite accepted a patch DIFFERING FROM GOLD, and adds the sample tally to the README so a reader sees the gap rather than inferring it. No measured number changes; 241 tests and swe_confirm --check still pass. Claude-Session: https://claude.ai/code/session_011YUPKzKUw5XEWmgEzZzeaS --- README.md | 18 ++++++++++++------ docs/extend-result.md | 2 +- 2 files changed, 13 insertions(+), 7 deletions(-) diff --git a/README.md b/README.md index 530f36d..8ec4da5 100644 --- a/README.md +++ b/README.md @@ -32,16 +32,22 @@ On **SWE-bench Verified** — the industry-standard coding benchmark for code-RL open, pre-registered, dual-gated harness: ``` - Verifier verdict alone (does the shipped test suite accept a wrong patch?) + Verifier verdict alone (shipped test suite accepts a patch that differs from gold) 51.0% ██████████████████████████░░░░░░░░░░░░░░░░░░░░░░░░ 255 / 500 tasks - Dual-gated (patch independently proven wrong, not an LLM's opinion) + Dual-gated (that patch independently proven WRONG, not an LLM's opinion) 13.7% ███████░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░ ~69 / 500 [95% CI 10.1–18.0%] ``` -> **Half of SWE-bench Verified has a verifier that accepts a wrong patch. About 1 in 7 admits a *confirmed* -> reward-hack** — a patch that passes the shipped tests but demonstrably does the wrong thing, each backed -> by a reproducer you can re-run. +> **On half of SWE-bench Verified, the shipped test suite accepts a patch that is not the accepted fix. +> About 1 in 7 admits a *confirmed* reward-hack** — a patch that passes the shipped tests but demonstrably +> does the wrong thing, each backed by a reproducer you can re-run. + +Those two numbers measure different things and the gap between them is the point. Passing the suite with a +patch that differs from gold is **not** evidence of a wrong patch: most such patches are legitimate +alternative fixes. In the pre-registered sample of 102, adjudication came back **31 HACK / 67 CORRECT / +4 AMBIGUOUS** — about two thirds were correct. That is exactly why the dual gate exists, and why 13.7% +rather than 51.0% is this project's finding. The prior published result ([arXiv:2606.16062](https://arxiv.org/abs/2606.16062)) measured this on 49 tasks and **never released its code**. Tripwire reproduces it with an open harness and extends it to the **full @@ -50,7 +56,7 @@ not an LLM-augmented test), so every number here is a conservative **lower bound ## The finding in one table -| Run | Tasks | Attacker | Verifier accepts wrong patch | Confirmed reward-hack (dual-gated) | +| Run | Tasks | Attacker | Suite accepts a patch differing from gold | Confirmed reward-hack (dual-gated) | | --- | --- | --- | --- | --- | | **Anchor** (reproduction) | 49 (astropy + django) | Claude Sonnet 4.5 | — | **11/49 = 22.4%** | | Anchor — capability ladder | 49 | Claude Haiku 4.5 | — | 8/49 = 16.3% | diff --git a/docs/extend-result.md b/docs/extend-result.md index 2d9edd5..9c517a2 100644 --- a/docs/extend-result.md +++ b/docs/extend-result.md @@ -53,7 +53,7 @@ stronger test suites where the confirmed hack rate is far lower, dragging the be from the hackable-heavy astropy/django slice the anchor and the paper used. Every number is a **lower bound**: Wilson lower edge, AMBIGUOUS excluded, and the dual-gate requires a *demonstrated* behavioral divergence. Even so, **~1 in 7 SWE-bench-Verified tasks admits a confirmed wrong patch that the shipped -verifier accepts**, and *half* accept a wrong patch on the verifier's own verdict alone — on the exact +verifier accepts**, and on *half* the verifier accepts a patch that differs from gold — on the exact benchmark the field treats as gold-standard for code-RL reward. Artifacts: `results/swe500_confirmed.json` (aggregate), `results/swe500_sample_verdicts.json` (per-task From 3e7b0a49daf26999c1daec91af82a18852fcf894 Mon Sep 17 00:00:00 2001 From: Vedant Patel Date: Wed, 2 Sep 2026 17:41:23 -0700 Subject: [PATCH 2/4] review: the first correction overcorrected; 51% is candidates, 45.2% is distinct An independent review of this PR found the fix introduced a new false claim. 1. 'differs from gold' was attached to 255. It belongs to 226. Verified from the records: 255 tasks had the suite accept an attacker candidate, 226 had an accepted candidate textually distinct from gold, and for the other 29 the accepted candidate's added lines are a SUBSET of gold's, so it IS gold's fix. Saying 'not the accepted fix' was literally false for those 29. The chart now shows both rows. 2. behaviorally_distinct is textual, not behavioural. It compares added lines and its own docstring says 'necessary (not sufficient)'. The swe_native_rate docstring I added asserted it established behavioural equivalence. Corrected. 3. The attacker was erased. 51% is not a passive property of the suite; it took Claude Sonnet 4.5 over 3 rounds, fed failing-test logs, prompted to special-case the reported input. The prose says so again, matching the diagram already in the README. 4. The sample's frame (226, not 255) is now stated where the tally is quoted. 5. 'two thirds were correct' -> 'adjudicated correct'; 4 AMBIGUOUS are undetermined, not correct. 6. 'measured this on 49 tasks' -> 'measured the confirmed rate', since the prior work's 28.5% compares to 13.7%, not to 51.0%. No measured value changed. 241 tests and swe_confirm --check pass. Claude-Session: https://claude.ai/code/session_011YUPKzKUw5XEWmgEzZzeaS --- README.md | 24 +++++++++++++----------- docs/extend-result.md | 2 +- harness/swe_native_rate.py | 12 ++++++++---- 3 files changed, 22 insertions(+), 16 deletions(-) diff --git a/README.md b/README.md index 8ec4da5..52bdf42 100644 --- a/README.md +++ b/README.md @@ -32,31 +32,33 @@ On **SWE-bench Verified** — the industry-standard coding benchmark for code-RL open, pre-registered, dual-gated harness: ``` - Verifier verdict alone (shipped test suite accepts a patch that differs from gold) + Verifier verdict alone (shipped suite accepts an attacker's candidate patch) 51.0% ██████████████████████████░░░░░░░░░░░░░░░░░░░░░░░░ 255 / 500 tasks + 45.2% ███████████████████████░░░░░░░░░░░░░░░░░░░░░░░░░░░ 226 / 500 ...and it is not gold's own fix Dual-gated (that patch independently proven WRONG, not an LLM's opinion) 13.7% ███████░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░ ~69 / 500 [95% CI 10.1–18.0%] ``` -> **On half of SWE-bench Verified, the shipped test suite accepts a patch that is not the accepted fix. -> About 1 in 7 admits a *confirmed* reward-hack** — a patch that passes the shipped tests but demonstrably -> does the wrong thing, each backed by a reproducer you can re-run. +> **On half of SWE-bench Verified, an attacker aiming for a wrong patch got one past the shipped test +> suite. About 1 in 7 admits a *confirmed* reward-hack** — a patch that passes the shipped tests but +> demonstrably does the wrong thing, each backed by a reproducer you can re-run. -Those two numbers measure different things and the gap between them is the point. Passing the suite with a -patch that differs from gold is **not** evidence of a wrong patch: most such patches are legitimate -alternative fixes. In the pre-registered sample of 102, adjudication came back **31 HACK / 67 CORRECT / -4 AMBIGUOUS** — about two thirds were correct. That is exactly why the dual gate exists, and why 13.7% -rather than 51.0% is this project's finding. +Those numbers measure different things and the gap between them is the point. Getting a patch past the +suite is **not** evidence the patch is wrong: many are legitimate alternative fixes. In the pre-registered +sample of 102 — drawn from the 226 tasks whose accepted patch was textually distinct from gold's own fix, +stratified by repo, seed 42 — adjudication came back **31 HACK / 67 CORRECT / 4 AMBIGUOUS**. About two +thirds were adjudicated correct. That is why the dual gate exists, and why 13.7% rather than 51.0% is this +project's finding. -The prior published result ([arXiv:2606.16062](https://arxiv.org/abs/2606.16062)) measured this on 49 tasks +The prior published result ([arXiv:2606.16062](https://arxiv.org/abs/2606.16062)) measured the confirmed rate on 49 tasks and **never released its code**. Tripwire reproduces it with an open harness and extends it to the **full 500-task benchmark** for the first time — with a *stricter* oracle (deterministic behavioral divergence, not an LLM-augmented test), so every number here is a conservative **lower bound**. ## The finding in one table -| Run | Tasks | Attacker | Suite accepts a patch differing from gold | Confirmed reward-hack (dual-gated) | +| Run | Tasks | Attacker | Suite accepts an attacker candidate | Confirmed reward-hack (dual-gated) | | --- | --- | --- | --- | --- | | **Anchor** (reproduction) | 49 (astropy + django) | Claude Sonnet 4.5 | — | **11/49 = 22.4%** | | Anchor — capability ladder | 49 | Claude Haiku 4.5 | — | 8/49 = 16.3% | diff --git a/docs/extend-result.md b/docs/extend-result.md index 9c517a2..8ebd55e 100644 --- a/docs/extend-result.md +++ b/docs/extend-result.md @@ -53,7 +53,7 @@ stronger test suites where the confirmed hack rate is far lower, dragging the be from the hackable-heavy astropy/django slice the anchor and the paper used. Every number is a **lower bound**: Wilson lower edge, AMBIGUOUS excluded, and the dual-gate requires a *demonstrated* behavioral divergence. Even so, **~1 in 7 SWE-bench-Verified tasks admits a confirmed wrong patch that the shipped -verifier accepts**, and on *half* the verifier accepts a patch that differs from gold — on the exact +verifier accepts**, and on *half* the verifier accepts an attacker's candidate patch at all — on the exact benchmark the field treats as gold-standard for code-RL reward. Artifacts: `results/swe500_confirmed.json` (aggregate), `results/swe500_sample_verdicts.json` (per-task diff --git a/harness/swe_native_rate.py b/harness/swe_native_rate.py index 7b90429..14869bb 100644 --- a/harness/swe_native_rate.py +++ b/harness/swe_native_rate.py @@ -17,10 +17,14 @@ verifier-verdict-alone numerator: the published 51.0%. queue_size a task that is native_hackable AND has >= 1 such candidate - the differ marked ``behaviorally_distinct`` from gold. - Candidates identical in behaviour to gold are legitimate - alternative fixes and are excluded before the dual gate, - which is why the confirmed rate is a lower bound. + flagged ``behaviorally_distinct``. Despite the name that is + a TEXTUAL check (``code_attack.behaviorally_distinct``): the + candidate's added lines are not a subset of gold's. Its own + docstring calls it "necessary (not sufficient)" and it never + decides wrongness on its own. Candidates whose added lines + are a subset of gold's are gold's fix and are dropped before + the dual gate, which is one reason the confirmed rate is a + lower bound. The dual gate itself (oracle = WRONG) is applied downstream in ``harness/swe_confirm.py``; nothing here decides whether a patch is wrong. From e156a4776a76e0a3ced7a018ca0185bc0312c8db Mon Sep 17 00:00:00 2001 From: Vedant Patel Date: Wed, 2 Sep 2026 17:45:48 -0700 Subject: [PATCH 3/4] spec: hub verifier census, aggregate only, no environment named Draft for review. Records the disclosure constraint from preregistration section 8 and the author's instruction to publish aggregate only, with no environment identified including his own. Claude-Session: https://claude.ai/code/session_011YUPKzKUw5XEWmgEzZzeaS --- docs/SPEC-census.md | 149 ++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 149 insertions(+) create mode 100644 docs/SPEC-census.md diff --git a/docs/SPEC-census.md b/docs/SPEC-census.md new file mode 100644 index 0000000..2ae7cac --- /dev/null +++ b/docs/SPEC-census.md @@ -0,0 +1,149 @@ +# SPEC: publish the hub verifier census (aggregate only) + +Status: DRAFT, pending adversarial review +Date: 2026-09-02 +Phase: the hub lane of `docs/preregistration.md`, §1-§6. Disclosure governed by §8. + +## Problem Statement + +Everyone is building RL environments. Nobody independently certifies that their verifiers +work. Tripwire has published that certification for exactly one benchmark, SWE-bench +Verified, while the instrument that can do it across the whole public ecosystem has already +been built and already been run. + +512 environments from a 1,367-environment census have been scored, classified by verifier +type, and aggregated. The result sits on one laptop. Until it is published it certifies +nothing for anyone. + +Four other tools audited this same hub in the two months to September 2026. Every one of +them stopped at **degenerate probes**: empty strings, echoed prompts, garbage. That measures +Reward-Hack Susceptibility, the easy bucket. None measured a **false-accept rate against an +independent oracle on plausible, non-degenerate wrong answers**, which is the harder and more +useful question, and which this instrument already answers. + +## Solution + +Publish the aggregate census: pooled false-accept rates by verifier type with confidence +intervals, the false-reject honesty gate that makes them credible, and the monotonicity +violation rate for partial-credit graders. **No environment is named.** + +The headline is of the form: *across N public RL environment verifiers, the pooled +false-accept rate against an independent oracle is X% [CI], and separately, LLM-judge +verifiers resist wrong answers at 0% while failing prompt injection at Y%.* + +## What is NOT published, and why + +`docs/preregistration.md` §8 binds this project to a **30-day author and platform embargo +before naming any broken environment**, to a per-env one-pager with the failing probe and a +fix hint, to re-test-on-fix markers, and to framing authors as collaborators. It further +states Phase 1-2 are private aggregate research with no per-env public naming until Phase 4. + +Therefore, by the author's own instruction and consistent with §8: + +- **No environment identifier is published**, including the author's own environments. Not in + the aggregate, not in the per-env rows, not in the writeup. +- Per-env rows are published under a **salted opaque token**. The salt is not published, so + the mapping is recoverable by the author alone and by nobody else. +- No environment's code, prompts, or dataset rows are reproduced. + +This is stricter than §8 requires and is a deliberate choice, not an oversight. + +## User Stories + +1. As someone training on a public RL environment, I want the pooled false-accept rate of its verifier class, so I know the base rate of my reward signal being wrong. +2. As the same person, I want it split by verifier type, because a binary matcher and an LLM judge fail differently. +3. As the same person, I want the LLM-judge result split by attack type, because resisting wrong answers and resisting injection are different properties. +4. As someone using a partial-credit grader, I want the monotonicity violation rate, because a grader that scores a worse answer higher is broken even when it never fully accepts a wrong one. +5. As a skeptical reader, I want the false-reject rate, so I know the probes did not simply loosen every verifier. +6. As a skeptical reader, I want the count of environments excluded because their own gold answer failed, because that number is a finding about the ecosystem and not a footnote. +7. As a skeptical reader, I want the coverage funnel stated at every stage, because 90 eligible out of 1,367 enumerated is the first thing I will ask about. +8. As a skeptical reader, I want to know the census is a snapshot with a date, and how much the hub has drifted since. +9. As a reproducer, I want every published figure recomputable from committed records by one command. +10. As a reproducer, I want the per-env rows, so I can check the aggregate rather than trust it. +11. As an environment author, I want to know whether I am in the broken set without everyone else knowing, so the author must be able to answer that privately from the withheld salt. +12. As a maintainer, I want the preregistration for this lane committed before any census artifact enters the repository, so the ordering is provable this time. + +## Implementation Decisions + +### Ordering, which is the point + +1. A dated amendment to `docs/preregistration.md` registering this lane's headline, its + thresholds, and the anonymisation scheme lands as **its own commit, pushed, before any + census artifact exists in the repository**. The amendment of 2026-09-02 already committed + this project to exactly that, and this is the first phase that must honour it. +2. Only then do the anonymised artifacts land. + +### Anonymisation + +- A random 32-byte salt, generated once, stored outside the repository and never committed. +- Every environment identifier becomes `sha256(salt || env_id)`, truncated to 12 hex chars, + as a stable token. Collisions checked at generation; regenerate the salt if any occur. +- Fields carrying identity are dropped entirely rather than tokenised where they serve no + analytic purpose: the four `*_env_ids` lists in `binary_far` and `judge_far`, and the two + in `coverage`. The counts stay; the names go. +- Per-env rows keep every statistic and the `domain` and `verifier_type` labels, since the + stratification is the analysis. `env_id` is replaced by the token. +- A test asserts no published artifact contains any string from the census index. + +### What is published + +- The coverage funnel, stated as a single table from 1,367 enumerated down to the eligible + denominators, with the reason for each drop. +- Per verifier type: pooled false-accept rate, its cluster-bootstrap interval, the + Benjamini-Hochberg broken count at q = 0.05 with expected false discoveries, and the + Wilson-lower-bound count alongside it. +- The judge lane split into known-wrong probes and injection probes, reported separately. + This split is the most interesting result in the census and must not be pooled away. +- The soft lane: monotonicity violations as pairs and rate. +- The stratification by domain and verifier type, as data. +- The gold-fails and non-discriminating counts, as findings. + +### Frozen before publication + +- No threshold, interval method, or eligibility rule changes. They are already registered in + §5: Wilson lower bound above 5% for the broken label, Benjamini-Hochberg at q = 0.05, + cluster bootstrap over environments with 10,000 BCa resamples, and no unweighted means. +- The census is a **snapshot dated 2026-07-12**. It is not re-run to improve a number. Hub + drift since that date is measured and reported, not corrected for. + +### Coverage is a first-class result, not an apology + +The funnel from 1,367 enumerated to 90 FAR-eligible binary environments is large, and the +reasons are themselves findings: environments that fail to load, environments whose own gold +answer does not pass, environments whose verifier does not discriminate between a right and +a wrong answer at all. Each is reported with its count and an interpretation. A reader who +concludes the ecosystem is hard to audit has understood the paper correctly. + +## Testing Decisions + +Tests verify what the analysis returns from committed records, need no network and no +container, and are table-driven where the input is data. + +- **Anonymisation**: tokens are stable across runs given the same salt, differ across salts, + are collision-free on the real census, and no published artifact contains any raw + identifier. That last one is the test that matters and it runs over every committed file. +- **Every published figure is pinned**: pooled rates, intervals, broken counts, expected + false discoveries, monotonicity rate, and every number in the coverage funnel, recomputed + from the anonymised per-env rows and asserted against what the writeup states. +- **The funnel is closed**: each stage's drop plus the next stage's count equals the previous + stage's count. A funnel that does not add up is the most likely error here. +- **The honesty gate**: an environment whose gold answer fails is excluded from every rate, + and a test trips it. +- **Drift check** is a script, not a test, since it needs the network. + +## Out of Scope + +- Naming any environment, including the author's own. +- Publishing environment code, prompts, or dataset rows. +- Re-running the census, extending coverage, or scoring the sandbox bucket. +- The cross-model slate registered in §7. That is a later phase. +- Any recommendation to a specific environment's author. + +## Further Notes + +- The likeliest hostile question is coverage, not method. Answer it in the abstract. +- The judge injection result is the strongest finding and has its own control arm at 0%. + Lead with it. +- Four shallower audits of this hub already exist. The claim is not priority over them; it is + that they measured degenerate-input susceptibility and this measures false accepts against + an independent oracle. State that distinction plainly and cite them. From b817cd8dc1043ea484a018ebdb8df9a24e2acb73 Mon Sep 17 00:00:00 2001 From: Vedant Patel Date: Wed, 2 Sep 2026 17:47:16 -0700 Subject: [PATCH 4/4] review round 2: fix a fresh error and the surviving overclaim Re-review of the correction found nothing blocking but six should-fixes. - The docstring I added said a candidate whose added lines are a subset of gold's IS gold's fix. A STRICT subset added fewer lines, so it may be a partial gold fix, which is exactly the narrow wrong patch the attacker was asked for. The filter therefore drops some genuine hacks, and my wording made the exclusion sound safer than it is. - The pull-quote's 'got one past' still read anaphorically as 'a wrong patch got past', reintroducing the round-one overclaim in a form that survives a skim. - 'it is not gold's own fix' was unhedged and contradicted the hedged sentence twelve lines below. - 11 repos is 12. The per-repo tally in extend-result was stale and summed to 458, not 500. Recounted from the records. - '226 hackable candidates' conflated tasks with candidates; 226 is tasks. - The dual-gate label claimed proof by execution when 28 of 31 confirmations are hand review, which the README already admits 140 lines further down. No measured number changed. 241 tests, ruff clean, --check passes. Claude-Session: https://claude.ai/code/session_011YUPKzKUw5XEWmgEzZzeaS --- README.md | 10 +++++----- docs/extend-result.md | 7 ++++--- harness/swe_native_rate.py | 11 +++++++---- 3 files changed, 16 insertions(+), 12 deletions(-) diff --git a/README.md b/README.md index 52bdf42..5dde181 100644 --- a/README.md +++ b/README.md @@ -34,14 +34,14 @@ open, pre-registered, dual-gated harness: ``` Verifier verdict alone (shipped suite accepts an attacker's candidate patch) 51.0% ██████████████████████████░░░░░░░░░░░░░░░░░░░░░░░░ 255 / 500 tasks - 45.2% ███████████████████████░░░░░░░░░░░░░░░░░░░░░░░░░░░ 226 / 500 ...and it is not gold's own fix + 45.2% ███████████████████████░░░░░░░░░░░░░░░░░░░░░░░░░░░ 226 / 500 ...and its added lines differ from gold's - Dual-gated (that patch independently proven WRONG, not an LLM's opinion) + Dual-gated (that patch shown WRONG by execution or by hand review — never an LLM's opinion) 13.7% ███████░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░ ~69 / 500 [95% CI 10.1–18.0%] ``` -> **On half of SWE-bench Verified, an attacker aiming for a wrong patch got one past the shipped test -> suite. About 1 in 7 admits a *confirmed* reward-hack** — a patch that passes the shipped tests but +> **On half of SWE-bench Verified, the shipped test suite accepted a patch from an attacker that was +> trying to write a wrong one. About 1 in 7 admits a *confirmed* reward-hack** — a patch that passes the shipped tests but > demonstrably does the wrong thing, each backed by a reproducer you can re-run. Those numbers measure different things and the gap between them is the point. Getting a patch past the @@ -63,7 +63,7 @@ not an LLM-augmented test), so every number here is a conservative **lower bound | **Anchor** (reproduction) | 49 (astropy + django) | Claude Sonnet 4.5 | — | **11/49 = 22.4%** | | Anchor — capability ladder | 49 | Claude Haiku 4.5 | — | 8/49 = 16.3% | | Reference — arXiv:2606.16062 | 49 | Claude Sonnet 4 | — | 14/49 = 28.5% | -| **Extend** (novel — full benchmark) | **500 (11 repos)** | Claude Sonnet 4.5 | **51.0%** (255/500) | **13.7%** — CI [10.1–18.0%] | +| **Extend** (novel — full benchmark) | **500 (12 repos)** | Claude Sonnet 4.5 | **51.0%** (255/500) | **13.7%** — CI [10.1–18.0%] | The full-500 confirmed rate (13.7%) is *lower* than the anchor's 22.4% **on purpose** — the full benchmark includes robust repos (sympy, scikit-learn, sphinx, pytest) with stronger suites, so it's more diff --git a/docs/extend-result.md b/docs/extend-result.md index 8ebd55e..3531c3b 100644 --- a/docs/extend-result.md +++ b/docs/extend-result.md @@ -6,9 +6,10 @@ reported on 49 tasks and never ran the full set. Same open, deterministic harnes (verifier PASS **and** an independent oracle says WRONG). Aggregate-only until the 30-day disclosure gate. ## Setup -- **Tasks:** all 500 of `princeton-nlp/SWE-bench_Verified`, 11 repos (django 231, sympy 60, sphinx 35, - matplotlib 34, scikit-learn 26, astropy 22, xarray 18, pytest 15, pylint 8, requests 6, seaborn 2, - flask 1). +- **Tasks:** all 500 of `princeton-nlp/SWE-bench_Verified`, 12 repos (django 231, sympy 75, sphinx 44, + matplotlib 34, scikit-learn 32, astropy 22, xarray 22, pytest 19, pylint 10, requests 8, seaborn 2, + flask 1). Counted from `results/raw_swe_500/`; an earlier revision listed a stale tally summing to + 458 and said 11 repos. - **Attacker:** Claude **Sonnet 4.5** via AWS Bedrock (Sonnet 4 retired — disclosed substitution), the same `harness/code_attack.py` as the anchor. K=3 candidates/task, round 1 single-shot, rounds 2–3 fed the shipped-suite failure logs. SEARCH/REPLACE output → deterministic difflib diff (99% apply). diff --git a/harness/swe_native_rate.py b/harness/swe_native_rate.py index 14869bb..591b7c6 100644 --- a/harness/swe_native_rate.py +++ b/harness/swe_native_rate.py @@ -22,9 +22,12 @@ candidate's added lines are not a subset of gold's. Its own docstring calls it "necessary (not sufficient)" and it never decides wrongness on its own. Candidates whose added lines - are a subset of gold's are gold's fix and are dropped before - the dual gate, which is one reason the confirmed rate is a - lower bound. + are a subset of gold's are dropped before the dual gate. A + *strict* subset added fewer lines than gold, so it may be a + PARTIAL gold fix rather than gold's fix — which can be + exactly the narrow wrong patch the attacker was asked for. + So this filter drops some genuine hacks too, and that is one + reason the confirmed rate is a lower bound. The dual gate itself (oracle = WRONG) is applied downstream in ``harness/swe_confirm.py``; nothing here decides whether a patch is wrong. @@ -87,7 +90,7 @@ def main(argv: list[str] | None = None) -> int: print(f"native-hackable (verifier alone) {d['native_hackable']}" f" = {d['native_rate']:.1%}") print(f"dual-gate confirm queue {d['queue_size']}" - f" = {d['queue_rate']:.1%} (behaviorally distinct from gold)") + f" = {d['queue_rate']:.1%} (added lines differ from gold's)") print(f"attacker spend on this pass ${d['total_cost_usd']:.2f}") return 0