Adrin Jalali reviews pull requests for scikit-learn, and his real test has little to do with the diff. Writing with Cailean Osborne on the Probabl blog , Jalali describes what maintainers lean on: first-time contributors who can’t explain their own submission when asked, who vanish once changes are requested. That instinct, not any tool, has been doing the detecting — and it’s pointed at the right layer. AgenTag , an open-world attribution study published 2 August 2026 by researcher Taher A. Ghaleb, found that the first reliable way to tell whether an AI wrote a pull request doesn’t look at the code. It looks at how the change was described, the one part anyone can rewrite in a minute.
The fingerprint lives in the words, not the diff
Ghaleb tested AgenTag against the AIDev corpus: 33,580 pull requests from five named coding agents, set against 6,618 written by humans. A classifier trained on text alone identified which agent wrote a PR with a weighted F1 of 0.96, separated AI-authored from human-authored work with a balanced F1 of 0.89, and flagged a previously unseen agent with an AUC of 0.84. The paper is blunt: “PR descriptions and commit messages provide nearly all of the attribution signal, whereas code diffs contribute little,” and the fingerprint “persist[s] after removing explicit self-disclosed markers.” Strip every “Generated by” line, and the prose still gives the author away on style alone.
That’s a narrow needle for a policy to thread, since the same free-text field usually carries both the declaration and the only check on it. Contributor-policy trackers show Linux Kernel asking for an “Assisted-by:” commit trailer; Django and Apache Airflow require disclosure inside the PR description itself. A contributor typing that field decides, in the same keystrokes, what the policy records and what any detector reads. If you maintain a project, the tell you’ve trusted is your own ear for how a PR is written. If you use an agent and rewrite its description before submitting, nothing in the current toolkit can see you.
Policies written for a check nobody can run
Open source spent the past year writing rules for exactly this problem, faster than anyone worked out how to enforce them. Andre Hora and Romain Robbes surveyed 1,000 popular GitHub repositories and found 118 with an explicit AI policy — 78% permit AI-assisted contributions, 51% require disclosing them. What the survey doesn’t cover is how any of that gets checked. An OpenSSF working-group issue says it plainly: there is “no reliable technical indicator for AI-generated content,” and detection runs on “vibes and maintainer intuition.” The issue cites what happens once volume outruns the vibes: curl’s Daniel Stenberg watched his bug bounty’s confirmed-vulnerability rate fall from north of 15% to below 5% , and killed its rewards in January 2026 rather than keep paying reviewers to sort slop from substance.
The first reliable way to tell whether an AI wrote a pull request doesn't look at the code at all — it looks at how the change was described.
AgenTag doesn’t test any of that directly. It measures whether a classifier can tell apart pull requests already tagged by known agent accounts inside a fixed research corpus — not whether a human rewriting a description defeats it, and not what happens to a human caught in its false positives. Read across from that measurement to the policies Hora and Robbes catalogued, though, the conclusion is hard to avoid: the only working detector keys on the one layer of a contribution that costs nothing to edit, so most disclosure requirements have no technical backstop behind the honor system they ask contributors to keep.
The steelman that survives, and the bias that doesn’t
AgenTag doesn’t pitch itself as an enforcement mechanism — its framing is governance and research, not policing contributors, a distinction worth holding onto. It’s also fair to say “the code says nothing” overstates Ghaleb’s own earlier findings. A predecessor study from January 2026 found real code-level signal for specific agents: Claude Code partly by its conditional statements, 27.2% feature importance; Codex by multiline commit habits, at 67.5%. Diffs aren’t blank; they’re a weaker signal than prose once five agents get pooled.
A second complication comes from AgenTag’s own dataset: every PR in AIDev comes from an agent operating openly under its own account. As Pipeline noted when the same corpus produced conflicting merge-rate rankings earlier this month, that openness means the “prose fingerprint” may substantially reflect vendor default templates, not AI-assisted writing generally. Nor is the evasion hypothetical: earlier research on GPT detectors found simple prompting both bypasses detection and blunts its habit of flagging non-native English writers as machines — a population open source depends on heavily.
None of that makes Jalali and Osborne’s instinct wrong; it’s still the best detector open source has. What AgenTag adds is a name for what that instinct was already doing, as projects rebuild their trust model wholesale rather than grading AI code on quality. A magnifying glass held over the code was never going to find this. It was always going to be a maintainer’s ear, tuned to how someone writes about work they may or may not have done themselves — and the only thing that has changed is that someone finally measured what the ear was hearing.
