A junior engineer interviewed for a Harvard job-market paper puts the problem plainly: “People could make a huge number of commits quickly, but code review was still the bottleneck.” One senior engineer was the only person allowed to approve changes on their project, and could look at the junior’s work about once a week. The paper, by Fiona Chen and James Stratton, tracks 718 firms using Jellyfish engineering analytics. After adopting coding agents they wrote far more code, yet finished no measurably more work, because every extra pull request came back needing more fixes.
More code in, the same work out
The study covers about 300 million work events from January 2021 to March 2026, using a staggered difference-in-differences design (comparing firms that adopted agents at different times). Agent adoption raised lines of code by 30%, commits by 20% and pull requests by 23%, according to the paper. Resolved Jira issues and epics moved up slightly but not significantly. The authors’ estimates rule out an output increase larger than 12% of the baseline mean. AI Weekly’s write-up of the same figures puts the extra code at roughly 1,500 lines per worker per month.
The review numbers are where the paper gets interesting. Average pull-request review time rose 49%. The share of pull requests with changes requested rose by 0.12 from a 0.13 baseline, which the authors call “nearly doubling”, and comments per pull request rose 35%. Pull-request size showed no significant change.
| Category | Increase after adoption (%) |
|---|---|
| Lines of code | 30 |
| Commits | 20 |
| Pull requests | 23 |
| Review time | 49 |
| Comments per pull request | 35 |
The bottleneck is the draft, not the pile
“Review is the bottleneck” is already a familiar line. The sharper detail is that pull requests did not get bigger, so reviewers are not simply reading more. Each draft appears to need more repair. Read across from those averages, that suggests agent output is arriving less finished than human output did.
The paper does not measure that split. It reports firm-level averages and offers a model in which, “if the effect occurs through the quality channel”, relief “would require improvements in the technology itself”. Pipeline’s conclusion is a reading of the evidence through that model. If it holds, the usual managerial fixes aim at the wrong target. Firms already tried one: the share of workers doing review rose 14% after adoption. They tried another too. Nearly 80% of firms had used AI review tools by March 2026, yet those tools wrote only 23.3% of review comments and appeared on 10.8% of pull requests. AI review is nearly everywhere and barely present.
If agents hand reviewers drafts that need more repair, adding reviewers or bolting on an AI checker treats the volume of code, not its quality.
Consider the people at each end. A senior engineer who approves pull requests spends more of the week sending drafts back. An engineering manager reporting agent ROI in commits or pull requests is reporting what goes into review, not what ships. This also connects to Wealthfront’s AI reviewer , which spent real money per pull request to say nothing on most of them: a reviewer bolted on at the end is a costly place to find quality problems.
What the data cannot settle
The counterpoint is strong. The jump in changes requested may reflect stricter standards, not worse code. The same junior engineer describes new rules requiring senior approval for AI-generated code after outages caused by AI-generated commits, so some of the extra load comes from policy. The data is also observational, ends in March 2026 before current-generation agents, measures output through Jira resolution, and comes from Jellyfish’s customers. We found no direct published criticism of the paper yet.
An independent result cuts both ways. In an NBER working paper , Demirer, Musolff and Yang study more than 500,000 GitHub developers and find the same drop-off: a 240% commit effect from autonomous agents shrinks to 80% for projects and 30% for releases. Value does get through, just far less than coding activity suggests. A different dataset reaching a similar shape makes a pure policy explanation harder to sustain, though it does not rule it out.
It also complicates Spotify’s argument that a flat rework rate means agent-written pull requests are not riskier. Rework and changes requested are different measures, and neither says how well a reviewer understood what they approved.
For now, the safest reading is narrow. Counting commits tells you how fast drafts are produced. It says nothing about how many of them survive the red pen, and the stack on the senior engineer’s desk is where the difference shows up.
