“Memory” is a word borrowed from human cognition, and it does not map onto what agents do. Agents do not remember. They write state to storage and retrieve fragments of it later, through a pipeline you designed and can inspect. When a team says an agent “has no memory,” what they usually mean is that nobody decided what to persist, where to persist it, or how to fetch it back. That is a design gap, not a missing faculty.
Why the word causes bad architecture
The metaphor invites a single-box solution, and single boxes fail differently depending on what you actually needed. Teams reach for a vector database, call it “the agent’s memory,” and stop thinking. But a support agent that needs to recall a customer’s last three tickets has a different problem than one that needs to keep a running total across a fifty-step task. The first is retrieval over structured records. The second is state that must be updated correctly at every step, or the agent silently drifts off task by turn twenty. Naming both “memory” makes them look like the same problem. They are not, and the fix for one does not touch the other.
State: what changed this session
State is the mutable record of what has happened in the current task, and it needs to be explicit, versioned and small. This is the running total, the list of steps already taken, the current plan the agent is executing against. It should live in a structure you control directly, a dict, a row, a document, not inferred from scanning a transcript. The common failure is letting state live only in the conversation history, so every new turn requires the model to re-derive facts it already established. That is expensive and it is where agents lose the thread. If you are building anything with more than a few sequential steps, look at the state-machine patterns in the workflows library before reaching for a memory product.
Retrieval: what to pull before acting
Retrieval is a search problem, and it should be evaluated like one, with precision and recall, not with vibes about whether the agent “remembered.” This is where embeddings and vector stores genuinely help, because the question is “which of these thousand past interactions is relevant to this new query,” and that is a ranking task. The mistake is treating retrieval quality as fixed once you pick an embedding model. It is not. Chunk size, metadata filters, and re-ranking change outcomes more than the choice of vector database does, and none of that shows up if your only metric is “did the agent seem to know things.”
Context: what fits in the prompt right now
Context is the assembled input for this specific call, and it is the only one of the three that the model ever actually sees. State and retrieval both exist to produce context, and the job of the orchestration layer is deciding what earns a place in that window given a finite budget. Most “memory failures” reported by users are actually context assembly failures: the right fact was in storage, retrieval found it, and then it got truncated or crowded out by less relevant material before the call was made. Debugging this requires looking at the literal prompt sent to the model, not the abstraction above it.
What breaks when you skip the distinction
Systems that call everything memory tend to fail in one of two directions. Either they persist too much, so retrieval degrades as noise accumulates and every query returns marginally relevant fragments alongside the useful ones, or they persist too little because nobody assigned ownership of what should be written down, so the agent re-learns the same facts every session. Neither failure is visible in a demo. Both show up after a few weeks of real usage, which is exactly the point where a client asks why the agent that worked in the pitch stopped working in production.
What to actually build
Treat state, retrieval and context as three separate pipelines with three separate success criteria, and write down what each one owns before writing code. State needs a schema and an update rule. Retrieval needs a relevance benchmark, even a crude one, ten queries with known correct answers you check after every change. Context needs a budget and an explicit priority order for what gets dropped first when the budget is exceeded. None of this requires a new product category. It requires treating “the agent remembers things” as three engineering questions instead of one marketing feature, which is the kind of decomposition we walk through in services work when a memory layer someone bought off the shelf turns out to be doing none of the three jobs well.
The word “memory” will keep showing up in vendor pitches because it sells. Build the pipeline anyway, and call the parts what they are.