Remembers Everything, Learns Nothing

You gave your agent a memory and it still repeats the same mistake. What makes it improve is a loop that tests each failure and turns the ones that recur into procedures.

The agent broke a rule it had written down

The agent had the rule. You can find it in its memory file, line 1,140 of about 1,800: do not reformat the config, the deploy is strict about its indentation. You wrote it there three weeks ago, the last time it broke the deploy. This morning the agent reformatted the config. The deploy broke. It remembered the rule and broke it anyway.

Anyone who has run an agent for more than a week has met some version of this. You added memory. You did the responsible thing, gave the agent a place to keep what it learned, and then watched it keep the wrong things, or keep the right things somewhere it never looks. Run 100 came out no sharper than run 1. The store filled up and the behaviour stayed exactly where it was.

It remembered too much, and it kept all of it in one pile, where 1,800 lines bury every instruction equally. The rule loaded. It just loaded as one line among the hundreds it had needed once and never again.

Memory is several different things wearing one name. Store them in one place and you get a slow agent, a swollen context, and a file full of rules that quietly disagree.

That is the whole problem. The fix has two parts: sort what the agent keeps by how often it changes, and put a loop in charge of what gets to stay.

Memory is four things, sorted by how often each changes

Pull the pile apart with one question: how often does this change? The answer tells you where each piece belongs, because the four kinds of memory want four different homes.

At one end sit the things that almost never change. The agent’s standing rules, its constitution, the few constraints that hold on every task. Those belong in one short, always-loaded file, the kind Claude Code keeps in CLAUDE.md and Codex keeps in AGENTS.md, and short is the load-bearing word. A fresh session can burn a real slice of its budget loading its own instructions before you have typed a thing, so a line earns its place in that file by one test: would the agent get this wrong without it, on most tasks? The config rule passes, which is why it belonged here, in the fifty lines the agent reads every time, not on line 1,140 of a log it barely skims.

A step along are the things that change now and then, and only for certain tasks. Workflows, procedures, the steps for cutting a release. Those become skills, each in its own small file the agent loads only when the task calls for it, so you can keep fifty of them and pay for none until the one you need comes up.

Further along is what changes every single run. What the agent did, what broke, the fix it landed on. That is raw trajectory, and it goes in an append-only log, a plain learnings.md you only ever add to and compress later, never editing in place.

At the far end sits the big reference body that changes rarely but in bulk. Documentation, old code, whatever corpus the agent searches. That lives in an external store it queries once, early, under tight hygiene. Facts live there. Lessons live a layer up, in the log.

Sort memory by how often it changes, and each kind lands where the agent will look for it. Mix them, and the rule you need every time sits buried among everything you needed once.

This is the part most “give your agent a memory” guides get right and then quietly undo, by letting all four drain back into one file. The whole trick is keeping them apart. Hold the four kinds separate and each stays usable. Let them merge and you slide back into the single pile, config rule and all.

Four layers, keyed to rate of change. The rule you need every session belongs in the always-loaded file, not on line 1,140 of a log.

The loop is the part that compounds

Separation stops the rot. It does not, on its own, make the agent better. A tidy store is still a store, and a store only remembers. None of what follows pays off on one-off work, where a wrap-up is pure overhead; the loop earns its keep only when the same failure keeps coming back. Improvement comes from a loop that runs on top of the store, and the loop has four moves.

It starts with a wrap-up. After every task, the agent appends a few plain lines to the log:

## 2026-06-05  deploy auth changes
did:       edited config.yaml, ran the deploy
failed:    deploy rejected the file, the parser choked on the new indentation
fix:       restored the original formatting, deploy passed
next time: do not reformat config.yaml, the deploy is strict about indentation

The last line is the one that earns its keep. That next time is the promotable lesson, the single thing that might change what the agent does on its next run. You wire this up with one standing rule in the always-loaded file: after each task, append a wrap-up to learnings.md in this shape. The model will mostly remember on its own, and a Claude Code stop hook makes it certain, running the wrap-up the moment the agent finishes. No wrap-up, no raw material, and everything downstream starves.

Then an evaluation runs. On a schedule, you re-run a set of tasks drawn from real past failures and check whether the agent still handles them. This is the step that turns “it feels worse lately” into a logged, specific entry you can act on.

Consolidation comes next. Once a week, a pass compresses the log, archives the dead lines, and holds the active file to something an agent can read in one sitting, a few hundred lines rather than a few thousand. Without it, the append-only log becomes the 1,800-line pile again by a slower route.

Promotion is where it pays off. When the same next time line shows up three times or more, it graduates. It stops being a log entry and becomes its own skill, a file at .claude/skills/deploy-prep/SKILL.md:

---
name: deploy-prep
description: Use before any deploy. Stops the config.yaml reformatting that has broken the deploy three times.
---
Before any deploy, leave config.yaml formatting untouched; the deploy is strict about indentation.
Run `make check-config`, then deploy.

That description line is the load-bearing part. The agent reads it every session and pulls the skill in only when a deploy comes up, so the rule stays out of the way until the moment it matters. The buried log line is now a procedure the agent runs without being told, and the three redundant entries get cut. The break stops happening.

A skill is still context, though. The agent reads it and usually obeys, and for most lessons usually is the right bar. For a failure that is cheap to trigger and expensive to suffer, promote it one more step, out of memory and into enforcement: a pre-deploy check that fails loudly, or a Claude Code hook that blocks the edit before it lands. The make check-config line in that skill is the seed of it. Memory tells the agent what to do. A hook makes the wrong move impossible.

A store remembers. A loop improves. The difference is whether a failure the agent logged ever becomes a procedure it runs without being asked again.

Notice what compounds. The store only grows. The loop is the part that keeps finding the failures that recur and turning them into procedures, and three of its four moves throw things away or move them up the stack. Only the first one adds.

Wrap-up feeds the log. Evaluation is the step most skip. Consolidation keeps it lean. Promotion turns a recurring lesson into a skill, and for the failures that must never recur, one step further into a hook.

The evaluation is what separates learning from theatre

One of those four moves is doing more work than the rest, and it is the one almost everyone skips. Run the loop without the evaluation step and the wrap-up notes are self-reported and unchecked. The agent writes “fixed the config issue” and nothing on earth confirms it. The log fills with confident receipts for work that may not hold. You get a beautiful record of intentions and no idea whether the agent is improving or quietly getting worse.

Anthropic’s engineering team put the cost plainly in their guidance on evaluating agents: teams without evals get stuck in reactive loops, fixing one failure and creating the next, unable to separate a real regression from noise. ^¹ An evaluation is the instrument that turns “did this lesson stick” into something you can observe. It is the test that the config still survives a deploy after the agent swore it learned.

This is the part worth sitting with. The evaluation is the hard, tedious, expensive step, the one you are most tempted to defer, and it is precisely the one doing the work. Consolidation and honest forgetting are core engineering, the mechanism that keeps a growing transcript from turning into noise, and then into poison. Skip the test and every other layer you built just adds volume.

Persistence without a test is theatre. The evaluation is the only thing that can tell you whether a remembered lesson made the agent better or only made the file longer.

There is a sharper trap underneath. Optimise your memory system on how much it stores, lines logged, lessons captured, store size, and you are measuring activity. The number climbs while the agent stands still. A handful of real tasks the agent must still pass, run on a schedule, is worth more than any size metric, because it measures the only thing you wanted: did remembering change what the agent can do. The config break is eval case one:

task: add a feature flag to config.yaml and deploy
pass: the deploy succeeds and config.yaml is unchanged except for the new flag

Grade the outcome, the deploy passing, and let the agent reach it however it likes. The runner that does this is humble: a short script hands the agent each task in a clean workspace, then checks the pass line. A dozen lines of shell cover the scale this piece is about, and an off-the-shelf eval harness is the same loop once you outgrow it. Run a handful of these on a schedule and “is the agent getting better” stops being a feeling and becomes a number you can read.

What poisons a memory

When a memory system fails, the store is rarely what broke. The governance did, the policy for what gets to persist and how conflicts get resolved. Three failures cause most of the damage, and none of them are about storage.

The silent merge is the cheapest to fix and the easiest to miss. Two notes disagree, the agent picks one, and a real contradiction vanishes into a single confident line nobody flagged. The better setups converge on one fix: when sources conflict, mark it with a literal tag and let a human resolve it, so the disagreement stays visible instead of dissolving into one quiet error.

Auto-deployed consolidation is the newest of the three, and the most seductive. Anthropic’s Dreaming, a research preview from May 2026, runs a scheduled pass between sessions that rewrites an agent’s memory store from its recent work. ² It is genuinely useful, and Anthropic built in the safeguard that matters: the original store stays read-only, and the rewrite arrives as a separate output you approve before the agent runs on it. Point the agent at the new store unread and you can promote a hallucinated merge to a standing rule. Read before swap. The consolidation gives you a candidate, not a fact.

The bloated constitution is the slow one, the failure you already met. Everything important gets added to the always-loaded file, because adding feels safe, until the rule you need every time is buried on line 1,140. Keeping it short is what keeps the rest legible. One cousin is worth naming: a workspace that keeps your instructions loaded still does not remember your last session, so assume it does and you will lose context without knowing why.

Most memory failures are governance failures. The store did not break. The policy did, by merging two truths into one confident error that no one read before it became a rule.

The one-week test

You do not need a vector database to find out whether any of this applies to you. You need a week.

For one week, end every agent task with three lines:

failed: what broke
fix:    what actually worked
next:   the rule for next time, or nothing

Write nothing else, store it nowhere clever, a plain file is fine. At the end of the week, count which next lines repeat three times or more.

If nothing clusters, your agent does not yet have a memory worth building, and no database will give it one. The failures are not recurring, which means there is nothing stable to promote, and a bigger store would only hold more noise. That is a real and useful answer. It tells you the work is in the task, not the memory.

If something does cluster, you have found your first skill. The recurring failure is the one that should stop being a note and start being a procedure, and you now know exactly which one to lift first. The config rule was mine. Yours will be sitting in those three lines by Friday.

From there the build order is short. Promote that first lesson into a skill, then write one eval case so the agent has to keep passing it. Reach for weekly consolidation only when the log grows long enough to slow things down, and for an external store only when you have a real corpus to search. Each layer earns its place, and none is required on day one.

The store was never the hard part. The loop is, and the test above is the smallest honest version of it. Everything else, the four layers, the evaluation, the conflict markers, is how you scale what the test says is worth scaling.

---

Which failure does your agent repeat most, and is it still living in a log instead of a procedure?

---

The one-week test on one card. Run it before you build anything.

New to The Durability Curve? It is a standing argument about what still holds when the surface gets cheap. Subscribe for the rest, or start with what survives

1. Mikaela Grace, Jeremy Hadfield, Rodrigo Olivares, and Jiri De Jonghe, “Demystifying Evals for AI Agents,” Anthropic Engineering, 2026. The piece argues that multi-turn agent evaluation is a coordination problem as much as a scoring one, and distinguishes pass@k, the probability that at least one of k attempts succeeds, from pass^k, the probability that all of them do, as answers to different reliability questions. https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents

2. Anthropic introduced "Dreaming" for Claude's Managed Agents as a research preview at its Code with Claude event on 6 May 2026. It runs a scheduled, between-session process that consolidates an agent's external memory store and surfaces patterns from recent sessions, which Anthropic compares to the way the brain replays the day during sleep. The original store stays read-only and the consolidated version is produced as a separate output for human review before the agent adopts it.