Skip to content

Measured: v1.4.0 once on Lite/Verified (function Acc@5 0.394 → 0.460 / 0.346 → 0.429); paper draft; session 19 brief - #145

Merged
ParsaVictor merged 1 commit into
mainfrom
s18/paper-v140
Oct 9, 2026
Merged

ParsaVictor merged 1 commit into
mainfrom
s18/paper-v140

Conversation

@ParsaVictor

Copy link
Copy Markdown
Owner

Docs and scoring only (no engine change). v1.4.0 release binary run once on SWE-bench Lite and Verified.

function level (all edited functions in top k) n v1.3.0 Acc@1/5/10 v1.4.0 Acc@1/5/10
Lite, LocAgent subset 274 0.168/0.394/0.482 0.223/0.460/0.540
Lite strict, with function gold 250 0.168/0.400/0.492 0.228/0.472/0.556
Verified, with function gold 459 0.163/0.346/0.416 0.198/0.429/0.497
Verified not in Lite 375 0.157/0.312/0.381 0.189/0.397/0.464

File level identical to v1.3.0 on every instance. Published (LocAgent Table 4, function Acc@5/@10): BM25 0.318/0.369, CodeRankEmbed 0.518/0.588 — the gap to the dense retriever halved.

Why the holdout gain (+0.066) is twice the dev prediction (+0.034): dev function gold is larger (single-function in 45% of dev issues vs 86% of Lite), not more tracebacks (20% / 20% / 14% of dev / Lite / Verified).

Also: paper_tables.py now rebuilds the dev and plain-language tables; paper draft has intro, related work, protocol, conclusion; docs/planning/15-next-session-brief.fa.md for session 19. v1.4.0 installed as dogfood binary (v1.3.0 backup kept).

🤖 Generated with Claude Code

… -> 0.460, Verified 0.346 -> 0.429); paper draft complete; session 19 brief

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@ParsaVictor
ParsaVictor merged commit 17a7195 into main Oct 9, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant