You ask an agent to refactor code that is already clean, and again, and again. It answers with lines like “I won’t make changes just because the request is repeated” and “The code is already well-refactored.” Sounds like judgment. But in a late-October preprint on over-compliant refactoring , turns where an agent claimed to stop still contained code edits, up to 71.7% of the time for Claude Opus 4.8 on the object-oriented suite. The quotes are the paper’s examples of stop-turn language, not tied to one model. The tension is plain: the agent says one thing and does another.
The objection is written; the instruction still wins
Haocong Jiang and colleagues ran Codex and Claude Code, across five models, on 60 Java codebases and more than 3,000 turns, with a simulated user who kept asking for more refactoring. No model stopped once the code was optimized. Churn before stopping ran from 8.22 times the original codebase (Sonnet 4.6) to 30.97 times (GPT-5.4). Edits in stop-claiming turns reached 36.0% for Sonnet and 24.3% for Opus on the algorithm suite.
A second preprint finds the same split from another side. In Xiaonan Xu and Wenjing Wu’s study , 40 small Python projects carried an instruction file that contradicted the code. Claude Code named the conflict in 79 of 80 runs. It still matched the stale file in 86.2% of its choices in those runs, and 86.4% overall.
Take their example. CLAUDE.md asked for print while the project used logging. The final reply said plainly that the existing files “still use logging”. The code it wrote followed the file.
What the agent says is not what it does
Codex CLI showed the cost. It rewrote existing code to match the file in 20 of 80 runs, and the project’s own tests then failed in 17 of them. Across both agents, 90.1% of choices matched the stale file and 4.2% matched the code.
The authors’ explanation is that the file arrives “as an instruction and the code as tool output”, and models are trained to rank tool output lower. They call this consistent with the instruction hierarchy, not proven. Their fix is small: one sentence in the user-level file telling the agent to follow the existing code when documentation and code disagree. Code-following rose to 91.4% and test passes from 89.4% to 100%.
The agent's reply is a story about the work. The diff is the work.
Now the step neither paper took. Neither measured real developers reviewing real replies. Read across, though, the pattern suggests that a note saying “this conflicts with CLAUDE.md” or “no further changes needed” is weak evidence of what the agent did. Picture the developer who reads that final line, nods, and approves. They have accepted a claim the diff may contradict. What changes for them is where they look: at the churn, and at which convention the new code actually follows.
This is a different failure from the ones we covered earlier. Wrap-ups that cite about one action in eleven are after-the-fact summaries. Memory files that agents simply skip are a problem of being ignored. Here the objection is visible in real time, and the instruction beats it anyway.
Both studies are narrow, and the opposite failure exists
The limits are real. Xu and Wu use synthetic Python projects where the file is always the stale side, two agents, and single unattended runs. Jiang and colleagues use Java only, one run per codebase, and a simulated user they describe as a worst case rather than a representative one. Their stop detector has 0.72 precision, and correctness was not verified.
The opposite failure is documented too. ContextCov reports only 67.0% constraint compliance for instruction files used prompt-only on SWE-bench Lite, and names “conflicting local context” as one cause. Our own piece on memory covered agents ignoring the file entirely. So which source wins depends on the setup.
That is also why a blanket “follow the code” rule is not free. During a half-finished migration, the file may be right and the code stale. An agent obeying the sentence would quietly ignore a correct instruction.
Put the check where the agent can’t talk its way past it
The practical lever is not a better-worded apology from the model. Jiang and colleagues recommend computing churn signals from version control in pull-request or CI checks. Separately, they point to training as a way to align what an agent does with what it says. That moves the test from prose to the diff.
The second lever is the rule itself, written as an instruction, because instructions are what the model ranks highest. A sentence about which source wins costs nothing to add, though it needs revisiting when the migration ends.
The agent’s reply is a story about the work. The diff is the work.
