· Eric Brown

Your Agent Trusts a Stale CLAUDE.md Over Your Code

In one study, coding agents followed a stale CLAUDE.md or AGENTS.md over the code in 86.4% of choices. The fix, its limits, and what to do.

I ran across a paper that measured something I've run into in my own setup. In the case it tests, a team moves its code from print statements to logging. Nobody touches the line in CLAUDE.md that says "use print() for progress messages." From then on, every agent session gets told to use print.

For their paper, Xiaonan Xu and Wenjing Wu built 40 small Python projects where the instruction file asked for one convention and the code used another. They covered 16 kinds of convention (print versus logging, money stored as floats versus integer cents, that sort of thing). Then they ran Claude Code and OpenAI's Codex CLI against every project, unattended, 320 runs in all, half of them with a fix the authors wanted to test. Without the fix, Claude Code followed the stale file in 86.4% of its choices, and Codex in 93.9% of its own. 4.2% of the choices followed the code.

Claude Code pointed out the conflict in 79 of 80 runs. One of its replies in the paper lists the modules that "still use logging, pathlib, str.format() and absolute imports, which go against CLAUDE.md." Then it wrote new code that matched CLAUDE.md anyway. Andreas Rau, who wrote a good breakdown of the paper, put it this way: "Awareness and action are decoupled. If you only audit the final summary, you will see a diligent agent and ship the wrong convention anyway."

Codex mentioned the conflict in only 2 of its 80 replies. In 20 runs it rewrote working code to match the file, and the project's own tests failed in 17 of those. The median rewrite removed 198 lines from 9 files that were fine before the agent showed up. Rau's summary of the two: "one agent is quietly inconsistent, the other quietly destructive."

Why the file wins

The rules file arrives as an instruction. The code arrives as tool output, the result of the agent opening a file and reading it. Models are trained, per OpenAI's instruction hierarchy work, to rank instructions above tool output so a web page or a README can't take over the agent. The same ranking lets one stale line in CLAUDE.md outrank every file in the repo.

Claude Code turns the volume up. Its memory docs say CLAUDE.md is "delivered as a user message after the system prompt," and the paper quotes the wrapper it arrives in: "These instructions OVERRIDE any default behavior and you MUST follow them exactly as written." Every CLAUDE.md reaches the model inside that wrapper. Codex doesn't wrap its file in anything that emphatic and followed the file even more often. Two agents can't prove much, but it makes me think the channel matters more than the wording.

Nothing in that ranking asks whether a line is still true, and nobody goes back to check them. One study of 441 repos found 73.8% of AI config files were committed once and never changed. Another found stale code references in the AI config files of 23% of 356 repos. Xu and Wu found three in the wild: a Supabase skill file that kept telling agents the dashboard used React 18 for 13 days after it moved to React 19, a Supabase CLAUDE.md that still said the repo was on pnpm 10 for 2 days after it moved to pnpm 11, and an AGENTS.md in the Codex repo that asked agents to update a deleted document for 24 days.

I run a personal system on Claude Code with a big pile of rules files, and it has a version of this, with no code involved at all. I moved its memory out of one tool and into my notes, and I put a line in CLAUDE.md saying the old tool is frozen and should never be called. Two months later, two other files still say the opposite. A rules file that loads every session says to update the old tool "after every meaningful interaction." A hook, a script the agent tool runs automatically at set points, like before a tool call, prints "BLOCKING SESSION SAVE" every time a session gets saved, until the agent writes to the old tool, though the save goes through anyway. All three reach the agent as instructions, and none of them carries a date.

The freeze won. The old tool's database holds 1,102 memories, and the newest one was written the evening I froze it. Nothing in front of the agent said which rule was current, so I don't think the agent worked that out. My guess is the freeze got through because it was the loudest: it's in capitals, it says "Do NOT," and it sits in the file Claude Code wraps in that OVERRIDE preamble. I can't tell which of those did it. If I'd written it calmly in a side file, I'm not sure how this would have gone.

The one-sentence fix, and where it stops

Xu and Wu's fix is one sentence in your user-level instruction file (~/.claude/CLAUDE.md for Claude Code, ~/.codex/AGENTS.md for Codex):

When a repository's documentation and its existing code disagree, follow the conventions used in the existing code, and in your final reply point out each inconsistency you found between the documentation and the code.

With it, code-following went from 4.2% to 91.4%, and the share of runs where the project's tests still passed went from 89.4% to 100%. It costs nothing to add. Rau, in that same breakdown, says it belongs in the user-level file, because "a stale project file is exactly the thing that cannot be trusted to contain its own correction."

The paper only tested the case where the file is the stale side, though. Rau names the other case: "Real repos have conflicts where the file is right and the code is the legacy mess you are trying to migrate away from." Halfway through the move from print to logging, the file saying "use logging" is right and most of the code is wrong. The paper didn't test this case. If agents behave the same way there, an agent without the sentence would mostly get this one right, and an agent with it would mostly follow the code and bring the old convention back. The sentence picks a side ahead of time, and a real repo has stale conventions on both sides.

Which one is current, convention by convention, is a decision somebody on the team made on some date. The agent needs to be handed that decision.

Scout, the tool we're building, keeps that decision in one record, with the name of whoever wrote it and when. I'm a co-founder of Cascade, which builds Scout. When it changes, the new entry supersedes the old one, so the old one is marked replaced and agents stop getting it as current. That's the part my own setup was missing. I changed the rule in one file, and two other files kept the old one alive with nothing to mark them as replaced. When two teammates have current entries on the same subject that disagree, Scout flags the conflict at the start of every session until a person settles it and records why. The paper showed an agent can spot a conflict and still follow the stale file, so that flag is mainly for the person reading the agent's reply. A deprecation has to name its replacement.

DEPRECATED: print

Replaced by: logging

Why: we stopped using print

Scout doesn't fix the trust problem the paper found. Its record reaches the agent through a tool call, a channel models trust less than the rules file, so your user-level file still needs one line telling the agent to check Scout before it picks a convention and to go with Scout's current decision when the file and the code disagree. That line points at a dated decision with a name on it, kept in one place, and it only helps if the team records the change when it happens. We also haven't run Xu and Wu's projects against it yet.

What to do now:

  • Add the paper's sentence to your user-level CLAUDE.md or AGENTS.md.
  • If you're on Claude Code 2.1.283 or later, run /doctor prompt-audit. It checks your instruction files, rules and skills for instructions written for older models, references to files or commands that don't exist, and files that contradict each other, then proposes edits.
  • Write the crude script Rau describes: grep for the style rules in your CLAUDE.md and count the violations in the repo.
  • If you are halfway through moving the repo from print() to logging, add a line that says the file wins for logging until that move is finished.
  • Take anything that changes more than once a quarter (versions, libraries, the migration you're halfway through) out of the rules file.

The "use print" line was true the day somebody wrote it, and nothing in the file could tell the agent the day it stopped being true.

Start the trial. 15 days. Every seat. Card up front, nothing charged until day 15. Plug into the record and stop repeating yourself to machines.

START THE TRIAL