· Eric Brown

No, We Changed That Later: Why Your Agent Keeps Reviving Old Decisions

Rules files only grow, so a reversed decision sits next to its replacement and the agent reads both. Why it happens, what the research says, and how to retire a decision for real.

In October a Hacker News thread called "Agents don't need memory, they need documentation" pulled 392 points and 312 comments. A commenter named kaydub described what happened after their team moved its decisions into files the agent reads: "Now in some sessions, every so often I have to remind the LLM, 'no, we changed that later, we do it this way now.'"

In another comment kaydub went further: "I've had the LLMs undo past decisions MORE from the docs than from having no docs. Old decisions always end up in the docs. If the LLM gets a whiff of an old decision, but doesn't get the update to that decision, well now you're doing things the old way again."

That matches what I see. Writing decisions down is supposed to stop the agent from relitigating them, and it mostly does, right up until a decision changes. Then the written record brings the old decision back.

Rules files only grow

Adding a line to CLAUDE.md or AGENTS.md feels safe. Deleting one feels risky, because you can't always remember why it was there, and the person who wrote it may have known something you don't. So when a decision changes, the usual move is to add the new rule and leave the old one where it is.

A study of 441 repos found that 73.8% of AI configuration files were committed once and never modified. In a review of 100 popular repos, dos Santos and colleagues found 24 AGENTS.md files with a single commit while the projects around them kept moving, a pattern they call Init Fossilization. The same study found context bloat in 42% of the files and confirmed conflicting instructions in 16 of the 100 repos.

When two lines disagree, the agent doesn't know which one came later. Claude Code's memory docs are plain about it: "if two instructions contradict each other, Claude may pick one arbitrarily." Len van der Hof put it more memorably: "Two rules that contradict do not throw an error. The agent obeys one on Monday, the other on Tuesday, and the model gets the blame."

And the agent will follow what's written even when the code says otherwise. Mohammadi and colleagues found that agents follow a written standard even when it prescribes the worse code, and a study I wrote about last week found Claude Code and Codex followed a stale rules file over the code in most of their choices.

It happened to me this morning. I run a personal system on Claude Code that publishes a weekly news brief to my site. In March I wrote a branch model into my global CLAUDE.md for everything I do (feature branch, then dev, then main, through pull requests, never straight to main). In June I wrote a skill file that walks the agent through publishing the brief, and one step of it said to merge the post branch into main and push. In September I edited that same step to add a preview gate and left the merge to main exactly as it was.

The agent followed the skill. The post went straight to main, and I found out when it was already live. Both instructions reached the agent as instructions. My guess is the skill won because it was more specific and it was the file loaded for that task. The fix took five minutes: I rewrote the step to go through dev. Those five minutes belonged in June, when I wrote the step, and I had a second chance in September, when I was editing it.

Everyone's instinct is that the most recent decision should win. A rules file can't do that on its own.

It needs a subject. Two lines about the same thing need to know they're about the same thing, or nothing can tell they conflict. "Merge to main and push" and "never merge directly to main" don't share any words a tool could match on.

It needs a date and an author. "Newest" only means something if each entry says when it was written and by whom. Lines in a markdown file have neither, and git blame on a file with years of edits isn't something the agent reads.

It needs a retire step. When the new decision lands, the old one has to stop being served as current. In a text file the only way to do that is to delete or rewrite the line, and that's exactly the step people skip.

Memory tools don't fix this, and some make it worse. Most of them recall by similarity, so an old decision that closely matches today's question comes back with full weight. Another commenter in the HN thread, b-karl, wrote that "memories go stale and pollute the context over time, especially when the state changes externally." A teammate's refactor is that kind of change, and your memory store never hears about it.

What retiring a decision looks like

Scout, the tool we're building, keeps decisions as entries with a subject, an author and a time, and changing one is its own step. A new entry supersedes the old one. The old entry stays in the history, so you can see what the team used to do and why it changed, but agents stop getting it as current. A deprecation ("we stopped merging straight to main, go through dev") has to name its replacement and the reason.

When two people's current entries on the same subject disagree, Scout flags the conflict at the start of every session until somebody settles it, keeping one and recording why.

It has limits. An entry without a subject can't be superseded or conflict with anything, because there's nothing to match it against. Supersede is automatic only for your own entries on a subject. When two teammates disagree, a person still has to settle it. And none of it helps if the change never gets recorded, which is the same discipline problem a rules file has, moved somewhere it's easier to see. It also wouldn't have caught my skill file. Scout only knows what's recorded in Scout, and that step lived in a markdown file it never reads. Its entries reach the agent through a tool call, which models trust less than the rules file, so your user-level file still needs a line telling the agent to check Scout first.

For Monday:

  • When you reverse a decision, rewrite or delete the old line in the same change. Don't add the new rule beside it.
  • Search every instruction file for the old decision, including skills, commands and hooks, before you call the change done. My skill file was the one I forgot.
  • When the agent does something the old way, ask it to quote the instruction it followed. That usually points straight at the line you missed.
  • On Claude Code 2.1.283 or later, run /doctor prompt-audit. It checks your instruction files for references that no longer exist and for files that contradict each other.
  • Keep anything that changes more than once a quarter out of the rules file, so there's less to go stale.

My skill file said main, my CLAUDE.md said dev, and the agent had no way to tell which one I meant.

Start the trial. 15 days. Every seat. Card up front, nothing charged until day 15. Plug into the record and stop repeating yourself to machines.

START THE TRIAL