The SWE-Bench Leaderboard Can No Longer Tell Models Apart

Addy Osmani has a rule for the file every coding team now keeps at the root of its repository. “Every line in a good AGENTS.md should be traceable back to a specific thing that went wrong,” he writes in Agent Harness Engineering , describing a file built out of scar tissue — one rule per remembered failure. Nobody adds the obvious next clause: every failure was observed under one particular model. Two papers, submitted two days apart in September 2026 — one on 15 September, the other on 17 September — explain why that omission now costs something. One audits 254 public submissions to the SWE-bench Verified leaderboard and finds it can no longer statistically separate its own top thirty entries. The other runs 176 matched harness configurations across four models and shows the setup wrapped around a model, not the model itself, decides the score — and that setup differs by model. Rank an agent by the leaderboard, and it can’t; rank it by the harness, and the harness you built doesn’t carry over.

Thirty entries that used to compete, and no longer disagree

The numbers are stark. On SWE-bench Verified, the two leading entries “each resolve 396 of 500 instances,” and the top ten submissions share 285 successes and only 51 failures — they are solving the same instances and missing the same ones. Exact paired McNemar tests, the audit reports, “separate none of the 29 adjacent Verified top-thirty pairs at alpha=0.05.” Yet the same data shows that changing only the scaffold around a single model can move its score by 29.8 points, against an 8.8-point spread across the whole top thirty. That gap is the story in one number, and it is not new: Addy Osmani reports a team, keeping the same model, “moved a coding agent from Top 30 to Top 5 by changing only the harness.” The audit is the measurement behind that anecdote. It is also not the first scoreboard to lose its grip this year — merge rate splintered into disagreeing rankings across four separate studies back in August, and a single public number is proving a thin thing to rank coding agents by twice running.

A harness tuned for a weaker model wastes effort on a stronger one

The second paper explains why teams cannot just copy the winning scaffold. Planning “scaffolds accuracy for weaker models while reducing costs for stronger ones, with minimal accuracy changes” once a model is already capable — the step that rescues a weak model mostly just adds expense to a strong one. Predefined tools help models with lower bash proficiency; proficient models reach the same results more cheaply with a bare bash interface. Context management matters chiefly under tight budgets, mainly by preventing overflow. None of that is a universal best practice — each is a function of the model’s own capability profile, the same lesson an earlier study found for naming a coordinator inside multi-agent teams : the label does nothing, the underlying wiring does.

Neither paper measures what happens inside a real team’s own configuration, though — both score benchmark instances under varied scaffolds, not a working AGENTS.md file. Read across from “optimal harness settings differ by model” to that file’s tool allowlist and context-compaction rules, and the conclusion follows: a harness tuned against one model’s blind spots is mistuned the day that model is swapped. It is the developer maintaining that file — the one whose AGENTS.md instructions were already shown to measurably steer agent behavior — who flips the model dropdown the week a better release ships, inherits rules written against a model no longer running, and cannot use the leaderboard to defend the switch. That this is already operative shows up in Claude Code’s own changelog , dated 17 September — the same week both papers went up: version 2.1.274 “changed /code-review to use leaner inline review prompts for every model that has no tuned settings of its own.” A vendor is already sorting its models into tuned and untuned, in a release note anyone can read.

The convergence result has an honest limit, and a serious rival

The strongest objection is that this saturation may be one benchmark split’s ceiling, not proof models have stopped differing — the same audit found its larger Test split separated 14 of 23 adjacent pairs. The harness paper cuts the same way against an over-strong reading of itself: for stronger models, planning’s benefit is mostly cost, not accuracy, raising the chance that as models improve, harness elaboration matters less for quality and more for the invoice. And Birgitta Böckeler’s Thoughtworks framework for “feedforward guides and feedback sensors” makes the opposite bet on purpose, treating the harness as model-independent scaffolding and the model beneath it as generic — the strongest case that recalibration isn’t needed, argued rather than measured.

None of that dissolves the asymmetry: the ranking meant to justify a model swap has lost its resolving power exactly where teams need it, at the top of the field, while the thing that actually decides outcomes resets on every swap. The socket wrench set stays in the drawer; it’s the dial on top of it that needs recalibrating, and nobody has written down when.