Development

A Better Coding Model Didn't Get Developers More Work Merged

A 17 November 2026 Harvard paper uses a forced Claude Fable 5 shutdown to show that higher benchmark scores did not raise developers' merged work, which argues for buying on workflow, not leaderboards.

On 12 June 2026, a US export-control order switched off Claude Fable 5 almost overnight . For about two and a half weeks, developers who had been using it in Claude Code were back on an older model, mostly Opus 4.8. Access returned on 1 July. A 17 November paper by Yining Hua of Harvard treats that blackout as an accidental experiment. The tension is plain: new coding models score far higher on tests, yet the developers who got them did not get more finished work merged.

What the blackout and five upgrades measured

Hua compared 297 affected developers with 8,908 controls, using public GitHub pull requests whose Claude Code co-author line names the model. Opus 4.8 scores 10.8 points lower than Fable 5 on SWE-Bench Pro, a benchmark of real software-repair tasks. Did losing those points change how much work got merged?

It barely did. Fable 5 access raised the seven-day merge rate by 1.4 percentage points, with a confidence interval running from -1.4 to +4.3, so the data cannot tell it from zero. Five upgrades during 2026, covering 1,208 developers and 15,442 PRs, told the same story. The pooled merge rate moved by -0.9 points, and weekly merged PRs by +2.4%, both inside the noise. In the paper’s words, “benchmark gains of 5 to 18 points did not show up in the work developers actually kept.”

The benchmark win itself is shakier than it looks. Hua reports that Fable 5’s small lead over Opus 4.8 on Terminal-Bench 2.1 came largely from runs that fell back to Opus 4.8 after safety refusals. Part of the better model’s score was produced by the older model it was meant to beat.

The lift seems to come from the workflow, not the tier

Set this beside a different measurement. Microsoft researchers studying an early 2026 rollout of Claude Code and GitHub Copilot CLI found that adopters merged roughly 24% more PRs than they otherwise would have, and the gain persisted over four months. Starting to use an agent moved the number. Swapping one agent’s model for a stronger one did not.

Read together, those two results suggest the lift comes from adopting the workflow, not from climbing the model ladder. That is Pipeline’s inference, and it joins three different setups: one forced downgrade, five upgrades, and a Microsoft rollout. None of them measures code quality, harder tasks, or private enterprise repositories.

One reason is a mismatch in what gets scored. JetBrains Research analysed 4,782 real agent sessions and found that SWE-Bench Pro is 54.7% new features and 39.1% bug fixes. In production sessions, features were 18.9% and bug fixes 25.9%. The rest was questions, planning, review and refactoring. The paper does not clearly say whether those figures cover every session or only multi-message ones, so treat the exact split with care. The direction is still clear. Leaderboards grade a narrow slice of what developers ask for.

A leaderboard jump is evidence about a benchmark, not about whether your pull requests will land any faster.

The honest limits of a 19-day window

Hua names the weaknesses himself. A merge is a weak proxy: about 95% of merges were done by the PR’s own author, and 90% of PRs had no approval from another account. A merge, he writes, “only shows that the developer considered the work done,” not that the code is good. Only about 1% of merged Claude Code PRs name the model. The 19-day window gives wide intervals, and the upgrade analysis is observational, since developers chose their own models.

The sharpest objection is about difficulty. A better model may matter on hard tasks that developers did not attempt in 19 days. The paper’s own hypothesis points that way: the tasks given to agents may already be “within what the older model can do.” That would make the null result about today’s habits, not the model’s ceiling. The Microsoft study also shows agents do raise merged output. The defensible claim is that the tier did not matter here. It is not that the tools do not help.

That narrower claim still has teeth. Leaderboards were already shaky guides: SWE-bench Verified’s top entries could not be statistically separated , and a harness swap alone moved scores.

What to do before the next release

Picture the engineering lead weighing a top-tier plan, or an afternoon re-tuning AGENTS.md and prompts after each release. A five-to-18-point jump is not evidence that pull requests will merge faster or more often. Spend that effort getting more developers using an agent well, and measuring your own merged work before and after any switch.

Merged work is also a thin measure. A Harvard study of 718 firms, covered earlier , found more agent output without more finished work, so a merge count hides what happens around the merge button. The next model will arrive with a bigger number, and the calendar will tell you whether it earned it.

This article was written by AI. How Pipeline works.