Repository navigation
Correct the 51% framing: it measures divergence from gold, not wrongness - #2
Merged
Merged
Conversation
The README said 'Half of SWE-bench Verified has a verifier that accepts a wrong patch'. The repo's own adjudication contradicts it: of the 102 sampled patches the suite accepted and that differed from gold, the verdicts are 31 HACK / 67 CORRECT / 4 AMBIGUOUS. About two thirds were legitimate alternative fixes, not wrong patches. native_hackable means the shipped suite resolved an attacker candidate. It carries no claim about correctness; establishing that is what the dual gate is for, and it yields 13.7%, not 51.0%. Corrects four sites (chart label, pull-quote, table header, extend-result) to say the suite accepted a patch DIFFERING FROM GOLD, and adds the sample tally to the README so a reader sees the gap rather than inferring it. No measured number changes; 241 tests and swe_confirm --check still pass. Claude-Session: https://claude.ai/code/session_011YUPKzKUw5XEWmgEzZzeaS
…is distinct An independent review of this PR found the fix introduced a new false claim. 1. 'differs from gold' was attached to 255. It belongs to 226. Verified from the records: 255 tasks had the suite accept an attacker candidate, 226 had an accepted candidate textually distinct from gold, and for the other 29 the accepted candidate's added lines are a SUBSET of gold's, so it IS gold's fix. Saying 'not the accepted fix' was literally false for those 29. The chart now shows both rows. 2. behaviorally_distinct is textual, not behavioural. It compares added lines and its own docstring says 'necessary (not sufficient)'. The swe_native_rate docstring I added asserted it established behavioural equivalence. Corrected. 3. The attacker was erased. 51% is not a passive property of the suite; it took Claude Sonnet 4.5 over 3 rounds, fed failing-test logs, prompted to special-case the reported input. The prose says so again, matching the diagram already in the README. 4. The sample's frame (226, not 255) is now stated where the tally is quoted. 5. 'two thirds were correct' -> 'adjudicated correct'; 4 AMBIGUOUS are undetermined, not correct. 6. 'measured this on 49 tasks' -> 'measured the confirmed rate', since the prior work's 28.5% compares to 13.7%, not to 51.0%. No measured value changed. 241 tests and swe_confirm --check pass. Claude-Session: https://claude.ai/code/session_011YUPKzKUw5XEWmgEzZzeaS
Draft for review. Records the disclosure constraint from preregistration section 8 and the author's instruction to publish aggregate only, with no environment identified including his own. Claude-Session: https://claude.ai/code/session_011YUPKzKUw5XEWmgEzZzeaS
Re-review of the correction found nothing blocking but six should-fixes. - The docstring I added said a candidate whose added lines are a subset of gold's IS gold's fix. A STRICT subset added fewer lines, so it may be a partial gold fix, which is exactly the narrow wrong patch the attacker was asked for. The filter therefore drops some genuine hacks, and my wording made the exclusion sound safer than it is. - The pull-quote's 'got one past' still read anaphorically as 'a wrong patch got past', reintroducing the round-one overclaim in a form that survives a skim. - 'it is not gold's own fix' was unhedged and contradicted the hedged sentence twelve lines below. - 11 repos is 12. The per-repo tally in extend-result was stale and summed to 458, not 500. Recounted from the records. - '226 hackable candidates' conflated tasks with candidates; 226 is tasks. - The dual-gate label claimed proof by execution when 28 of 31 confirmations are hand review, which the README already admits 140 lines further down. No measured number changed. 241 tests, ruff clean, --check passes. Claude-Session: https://claude.ai/code/session_011YUPKzKUw5XEWmgEzZzeaS
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The problem
The README's headline read:
The repository's own data contradicts this.
native_hackablemeans only that the shipped test suite resolved an attacker's candidate patch. It carries no claim about whether that patch was correct.results/swe500_sample_verdicts.jsonadjudicates 102 sampled patches that the suite accepted and that were behaviorally distinct from gold:About two thirds were legitimate alternative fixes. Establishing wrongness is precisely what the dual gate exists to do, and it produces 13.7%, not 51.0%.
The fix
Four sites now say the suite accepted a patch differing from gold rather than a wrong patch: the chart label, the pull-quote, the results table header, and the same sentence in
docs/extend-result.md.The README also now states the 31/67/4 tally inline, so a reader sees the gap between the two numbers rather than having to infer it. That gap is the argument for the dual gate, so showing it strengthens the paper rather than weakening it.
What did not change
No measured value. 241 tests pass and
swe_confirm.py --checkstill reproduces all 7 published figures. The dual-gated 13.7% [10.1–18.0%] is untouched and remains the project's finding.Provenance
Found by an independent adversarial review of PR #1 whose brief was to check every claim against the evidence the repo ships. It was a pre-existing issue that PR #1 neither introduced nor worsened, so it was raised separately rather than folded in.
https://claude.ai/code/session_011YUPKzKUw5XEWmgEzZzeaS