A coding agent is three hours into a refactor. It has read forty files, tried two approaches, abandoned one, and its running notes now stretch to a quarter of a million tokens of old file contents, dead plans, and tool output nobody will ever read again. Then it does something baffling: it re-introduces the exact bug it fixed an hour ago, because the fix scrolled out of view and it no longer remembers the constraint it set for itself in step one.
Nothing crashed. The model didn’t get dumber. Its memory just filled up with junk, and the one thing that mattered, the goal, got buried under everything that didn’t. The agent remembered too much.
This is the failure everyone hits and nobody plans for. We treat the context window, the text an agent can “see” at once, as a filing cabinet you keep stuffing. It is nothing of the sort. It is working memory, and the real design problem is not what to store. It is what to forget.
Working memory, not a filing cabinet
Here is the mental model worth carrying: an agent’s memory is a hierarchy, like a computer’s. The context window is RAM, small, fast, and expensive, because every token in it is re-read on every single step. Everything else, past turns, documents, facts learned along the way, is disk, cheap and roomy but useless until you deliberately fetch a piece of it back into RAM.
That framing comes straight from MemGPT, the 2023 paper that proposed running an LLM like an operating system: a tiny in-context working memory plus a large external store, with the agent paging data between them exactly as an OS swaps pages between RAM and disk. The insight isn’t that memory is big. It’s that working memory is intentionally small, so the scarce, expensive space stays full of only what the next step needs.
A bigger window will not save you
The obvious escape is to buy a bigger window. Frontier models now advertise a million tokens, so why curate at all? Because more context is not better context, and there’s now hard evidence.
Chroma’s context rot study (July 2025) ran eighteen models, including the flagships, and found the same thing in all of them: accuracy quietly degrades as the input grows, long before you hit the limit. On a million-token model, a clearly observable drop shows up somewhere around three to four hundred thousand tokens. It is not overflow. The window has plenty of room left. The model simply reasons worse when it’s wading through a swamp. There’s an older, related finding, lost in the middle: a model attends best to the start and end of its context and worst to the middle, so a fact buried mid-window may as well not be there.
And here’s the part that surprises people about what fills the window. It is almost never your prompt. It is tool output, one agent reading a heavy file, another scraping a long page, a verbose JSON blob from an API. That’s what balloons the context, spikes the bill, and pushes your original instructions out of the model’s best attention.
Four ways to forget
If working memory has to stay lean, you need a policy for getting things out of it. There are only four real moves, and a decent agent uses several.
Truncate. Keep the last N messages, drop the oldest. Cheap and dumb: it has recency bias, so it happily deletes the goal you set at the very start.
Summarize (compaction). Replace a long stretch of history with a short summary of what happened and what was decided. This is the workhorse. It’s how a coding agent turns forty pages of debugging into “root cause: TLS mismatch; fixed by rotating the CA; webhook retries still open.”
Retrieve. Keep the details in an external store and pull back only the ones relevant to the current step. This is what gives an agent memory across sessions, not just within one.
Scratchpad. Don’t ask the model to hold it at all. Write the plan, the results, the intermediate state to a file or a database, and read it back later. Coding agents lean on this because their artifacts routinely dwarf any window.
What to keep, what to throw away
The moves are the easy part. The hard part is the decision they all depend on: what earns a slot in working memory? The trap is to sort by recency, keeping whatever happened last. Sort by salience instead.
Keep, always, in the window: the current goal, the active plan, the constraints, and the tool results the next step will actually use. Compress the moment it’s settled: finished reasoning, resolved sub-problems, paths you already ruled out, all of it collapses to a line or two. And discard outright the things that only cost tokens: acknowledgements, chatter, superseded plans, and above all the raw tool logs that bloated the window in the first place. For our refactor agent that gets concrete: keep the target API and the constraint it set in step one, compress the two approaches it tried and abandoned into a sentence each, and let go of the forty raw file-reads, the very material that pushed the goal out of view.
It helps to name what kind of memory you’re holding. Episodic memory is what happened (“the user rejected the first design”). Semantic memory is durable fact (“this service is written in Rust”). Procedural memory is how-to (“the deploy runbook”). Facts and procedures are worth writing to long-term storage; most of the blow-by-blow is worth summarizing and letting go.
Where it goes wrong
Read the memory frameworks’ source code and the failure modes stop being abstract. Over-summarization is the big one: Letta’s compaction replaces old messages with a model-written summary and drops the originals, so any detail the summary skips is simply gone. Get the summary wrong and the agent confidently proceeds on a distorted past.
Then there’s poisoning, and it’s worse than it sounds. A wrong fact, a hallucination, or an instruction smuggled in through a malicious document can get written to long-term memory and resurface for weeks. This is not hypothetical: LangMem, a popular memory library, ships with updates enabled but deletes disabled by default, which means a bad memory tends to get rewritten or quietly retained unless you go out of your way to allow deletion. And the “smart” memory managers are often less smart than the label. mem0’s default write path, reading its actual code, extracts facts in add-only mode and de-duplicates by exact text hash, not by meaning, so two phrasings of the same fact both survive. Memory management is also not free: every summary and every extraction is another model call, on top of the embeddings, quietly taxing the system you built to save tokens.
The rest of the rogues’ gallery is familiar once you look for it: stale memory the world has moved past, contradictory memory where two retrieved facts disagree and the model picks one at random, and plain retrieval misses, where the right memory exists but the ranker never surfaces it.
Forgetting is now a feature you can buy
The good news is that the platforms have noticed. Anthropic now offers server-side context editing that automatically strips old tool inputs and outputs from the history once they’ve been used, so the agent remembers that it called a tool without carrying the raw payload, and a memory tool that lets the model write notes to a file store and start the next session on a clean window. The work you used to hand-roll in a summarization loop is becoming an API setting. Their own numbers on the lift are, predictably, their own numbers, so weigh them accordingly, but the direction is real: curation is moving from a thing you build to a thing you configure.
None of it changes the core discipline, though. A bigger window, a memory library, a native compaction flag, they all still force the same four decisions on you: what stays live, what gets compressed, what gets written down, and what gets thrown away. That last one is the whole game. This is the discipline behind the failure we called out when your agent quietly loses the thread on a long task, and it’s the same instinct as running small models by default: spend the expensive, scarce resource, here it’s the context window, only where it earns its place.
So the best agent is not the one with the biggest memory or the one that remembers the most. It’s the one that forgets on purpose. That gap, between an agent that hoards and an agent that curates, is where we work at Gracient: building the memory systems that keep an agent’s working set small, current, and trustworthy over hours of real work, so it still knows, at step five hundred, what it set out to do at step one. Let’s talk.