Agent Memory: What an AI Agent Should Remember, and Where
Short-term context, long-term memory and task state are different problems. How to store each, keep context windows small, and avoid remembering the wrong things.
Models are stateless
An LLM remembers nothing between calls. Everything an agent "knows" during a step is what you put into that step's context window. Memory is therefore an application design problem.
| Kind | What it holds | Lifetime | Typical storage |
|---|---|---|---|
| Working memory | Current conversation, recent tool results | One task or session | The prompt itself |
| Task state | Plan, completed steps, intermediate results | Until the task finishes | A database row or graph state |
| Long-term memory | User preferences, facts learned, past outcomes | Across sessions | Database or vector store |
Working memory: keep it small
Context windows are large but not free: long prompts cost more, run slower and can bury the important parts. Common techniques:
- Keep the last few turns verbatim and summarize older turns.
- Store large tool outputs outside the prompt and include only the relevant excerpt.
- Drop tool results once the step that needed them is complete.
Task state: make it explicit
Don't rely on the model to remember where it is in a multi-step job. Persist state in code:
state = {
"goal": "Reconcile yesterday's payments",
"steps_done": ["fetched_payments", "fetched_ledger"],
"mismatches": [{"payment_id": "P102", "reason": "amount differs"}],
}Frameworks such as LangGraph model this as a state object passed between nodes, with checkpoints so a failed run can resume instead of starting over.
Rendering agent memory layers
Long-term memory: be selective
Write only durable, useful facts ("prefers SQL examples in Snowflake syntax"), not every message. Retrieve by relevance — often with embeddings — and include only a handful per step.
A simple rule
Put in the prompt only what the next step needs, persist everything required to resume, and remember long-term only what you would be comfortable showing the user.
Finished this note?