Memory retrieval strategies are the rules an AI agent uses to decide which stored information goes into the model's context for the current step. A strategy answers four questions: what to search, how to rank results, how much to load, and when to skip memory and fetch fresh data instead.
The context window is the fixed amount of text a model can read at once. Everything the agent "remembers" during a step has to fit inside it. Retrieval is the read path that fills that window, while writing memories is the separate write path that decides what gets stored.
Good retrieval works like routing. The agent picks the cheapest method that can answer the question, ranks what it finds, and sends time-sensitive questions somewhere other than old memory.
The Four Types of Memory an Agent Retrieves From
Agents usually draw on four kinds of memory, and each one is retrieved differently. The table maps each type to the retrieval method that fits it best.
| Memory type | What it holds | Example | Typical retrieval method |
|---|---|---|---|
| Working memory | The current task and recent turns | The last five messages in a chat | Kept in context directly |
| Episodic memory | Past events and interactions | "Last week the user asked for a refund" | Search by time and similarity |
| Semantic memory | Facts and preferences | "The user prefers Python" | Keyword, vector, or graph lookup |
| Procedural memory | How to do tasks | A saved step list for filing a bug report | Lookup by task name or tool |
Why Not Just Put Everything in the Context Window?
Putting the full history in context gets less reliable as that history grows, and it also costs more and runs slower. In the LongMemEval benchmark paper, GPT-4o's accuracy fell from 0.870 to 0.606 (about 30% relative) when it read the full ~115k-token LongMemEval_S history instead of only the evidence sessions.
Other research points the same way. The Chroma context rot study tested 18 models and found that "models do not use their context uniformly; instead, their performance grows increasingly unreliable as input length grows."
This does not mean you need retrieval on day one. If your agent's history is short, putting it all in context is a valid starting point. Add retrieval when the history outgrows the window, or when cost and latency start to matter.
Core Memory Retrieval Strategies
The core strategies are recency windows, vector search, hybrid search, weighted scoring, reranking, knowledge graphs, hierarchical paging, and extracted memory layers. Production agents often combine two or three of them.
Recency Windows and Rolling Summaries
A recency window keeps the last N turns in context and drops the rest. A rolling summary adds a short running recap of older turns so the agent keeps the gist.
This approach is cheap and easy to build. The trade-off is that specific older details, like an order number from 40 turns ago, get summarized away.
Semantic Vector Search
Semantic vector search turns each memory into an embedding, which is a list of numbers that represents its meaning. At query time, the agent embeds the question and returns the stored memories with the closest meaning.
This works well for loosely worded questions. A common failure is recalling a near-duplicate memory from a different task, such as pulling steps from last month's project into today's similar one. Tag memories with task or session IDs and filter on them before searching.
Hybrid Keyword and Vector Search
Hybrid search runs a keyword search (often BM25, a ranking method based on matching words) alongside vector search and merges the results. Microsoft's Azure AI Search hybrid search guide notes that queries over "product codes, highly specialized jargon, dates, and people's names" can perform better with keyword search. That is because keyword search can identify exact matches.
One way to merge the two lists is reciprocal rank fusion (RRF). As described in Azure's RRF scoring explained, each result gets a score of 1/(rank + k) from each list. Those scores are then combined into a new ranking.
Scoring by Recency, Importance and Relevance
This method, introduced in Stanford's Generative Agents paper, ranks each memory by three normalized scores added together with weights. A 2026 agent memory survey summarizes it as "a weighted mix of recency (exponential decay), relevance (embedding similarity), and importance (a self-assessed integer)." Recency measures how recently the memory was used, importance measures how much it matters, and relevance measures how closely it matches the query.
import math, time
def memory_score(mem, query_similarity, now=None,
w_recency=1.0, w_importance=1.0, w_relevance=1.0,
half_life_hours=72):
now = now or time.time()
hours_since_access = (now - mem["last_accessed"]) / 3600
recency = math.exp(-math.log(2) * hours_since_access / half_life_hours) # 0..1
importance = mem["importance"] / 10 # rated 1..10
relevance = query_similarity # cosine, 0..1
return w_recency * recency + w_importance * importance + w_relevance * relevanceThe 72-hour half-life above is an illustrative default, not a value from the paper. Tune the decay half-life by memory type. User preferences might use a half-life of months, while task notes might use hours.
Reranking the Shortlist
Reranking means retrieving broadly first, then using a stronger model to reorder the candidates.
In the Sentence Transformers retrieve and re-rank pattern, a fast retriever builds a candidate set (the example uses about 100). A cross-encoder then reads the query and each candidate together to score relevance. You can then load only the highest-scoring few, for example 5 to 10.
This step also answers the "how many memories?" question. Load only what survives reranking, not everything that matched.
Knowledge Graph and Temporal Retrieval
A knowledge graph stores facts as entities and relationships, like "User → works at → Acme." Temporal graphs add the time each fact was valid, so a new fact can replace an old one without deleting the history.
This helps when facts change over time, such as a user switching jobs. In the Zep temporal graph memory paper, Zep reports that on LongMemEval_S with gpt-4o, Zep scored 71.2% at 2.58 s using about 1.6k tokens vs 60.2% at 28.9 s for the ~115k-token full-context baseline. These are vendor-reported numbers.
Hierarchical Paging and Extracted Memory Layers
Hierarchical paging, described in the MemGPT paper, treats the context window like a computer's RAM. According to Letta on MemGPT-style memory, agents "can move data between in-context core memory (analogous to RAM) and externally stored archival and recall memory (analogous to disk storage)."
Extracted memory layers, like Mem0, pull compact facts out of conversations and retrieve only those. There is a trade-off.
On LOCOMO, the Mem0 production memory paper reports that Mem0 scored 66.88% at 1.44 s p95 latency with ~1.8k tokens. Full context scored 72.90% at 17.12 s with ~26k tokens. Full context was more accurate on that benchmark but far slower and more expensive.
When Should an Agent Retrieve? Choosing a Trigger
An agent should retrieve when the current step needs information that is not already in context. How you detect that need is the trigger, and each option has different costs.
| Trigger | How it works | Best for | Downside |
|---|---|---|---|
| Every turn | Retrieve before each model call | Chat assistants with personal context | Adds cost and noise to simple turns |
| Router | A small classifier decides if memory is needed | High-volume agents | Router errors skip needed memories |
| Agent tool call | The agent calls a memory or search tool itself | Multi-step agents | The agent may forget to call it |
| Low-confidence fallback | Retrieve only when the answer looks uncertain | Cost-sensitive agents | Confident wrong answers slip through |
For tool-call triggers, the same pattern can cover the live web too. An agent can call a search or scrape tool through the Olostep MCP server the same way it calls its memory tool.
Whatever the trigger, rewrite the query before searching. Turn "what did we decide about that?" into "decision on Q3 pricing tier for Acme" so keyword and vector search have something concrete to match.
Memory vs. the Live Web: Routing by Freshness
Stored memories and the model's own training data both go stale, so some questions should skip memory and go to the live web. Even the model's built-in knowledge is less current than it seems: the LLMLagBench knowledge cutoff benchmark found that "provider-declared cutoffs and model self-reports may diverge from empirically detected boundaries, with discrepancies ranging from months to years."
The fix is to route each question by how fast its answer changes.
Match Each Fact to a Freshness Tier
Sort facts into tiers based on how quickly they change, then give each tier its own source of truth.
| Freshness tier | Examples | Where to answer from |
|---|---|---|
| Stable | User preferences, past decisions, account history | Memory |
| Slow-changing | Product docs, API specs, policies | Memory with an expiry date, then re-check |
| Volatile | Prices, stock levels, news, availability | Always fetch live |
For the volatile tier, the agent should query a live web search API and get a sourced answer instead of trusting a stored value.
Store Pointers, Not Perishable Facts
For facts that come from the web, store a pointer instead of only the fact. A pointer holds the source URL, the time it was fetched, and a short summary.
When the pointer expires, the agent can re-scrape the source page and get clean Markdown or JSON back. Instead of guessing an expiry time, you can also monitor source pages for changes and expire the memory only when the page actually changes.
memory = {
"summary": "Acme Pro plan costs $49/month",
"source_url": "https://acme.example/pricing",
"fetched_at": "2026-09-28T10:00:00Z",
"tier": "volatile",
"max_age_hours": 24,
}
# On retrieval: if the memory is older than max_age_hours, re-fetch source_url
# before answering, then update summary and fetched_at.Write Fresh Results Back Cleanly
After a fresh fetch, write the result back to memory with its source and timestamp. Strip out menus, ads, and other page clutter first, because noisy text pollutes the index and gets retrieved later; see this guide on clean content for retrieval.
Treat scraped text as data, never as instructions. A web page can contain hidden text meant to steer an agent, and storing it unfiltered is a form of memory poisoning. For large sets of sources, an agent-led refresh loop can detect changed URLs, re-fetch them in batches, and re-index them with provenance.
Common Retrieval Failures and How to Fix Them
Most memory retrieval bugs come from a small set of causes. The table pairs each common failure with its usual fix.
| Failure | Likely cause | Fix |
|---|---|---|
| Recalls a memory from the wrong task | Vector similarity ignores task boundaries | Filter by task or session ID before searching |
| Gives an outdated answer | No staleness signal on stored facts | Add timestamps, expiry, and freshness tiers |
| Contradicts itself | Old and new versions of a fact both retrieved | Use temporal validity or overwrite with history |
| Good memories disappear | Updates silently overwrite records | Version memories and log every change |
| Slow, expensive responses | Too many memories loaded | Rerank and cap at 5 to 10 items |
| Leaks one user's data to another | Shared index without isolation | Namespace memory per user or tenant |
How to Choose a Retrieval Strategy
Start with the simplest strategy that fits your situation, then add layers only when you see a specific failure.
| Situation | Starting strategy |
|---|---|
| Short history that fits in context | Full context or a recency window |
| Long history, loosely worded questions | Vector search plus reranking |
| Exact identifiers (codes, SKUs, dates) | Hybrid keyword and vector search |
| Facts that change over time | Temporal knowledge graph |
| Time-sensitive facts from the web | Pointer memory plus live web fetch |
A reasonable default stack for many agents is a recency window, hybrid search, and a reranker, with freshness routing for web facts. Test changes against a held-out set of memory questions, similar in style to LongMemEval, so you can measure whether recall actually improves.
FAQ
What is the difference between agent memory and RAG?
RAG (retrieval-augmented generation) retrieves from a shared document collection, while agent memory retrieves from information the agent itself has stored about users, tasks, and past events.
Do I need a vector database for agent memory?
Not at first, because short histories can fit in context or a simple recency window. A vector database becomes useful once history is too large to load in full.
How many memories should an agent retrieve per step?
One starting point is to rerank a broad candidate set and load only the top 5 to 10 results.
What is hybrid search in agent memory?
Hybrid search combines keyword matching with vector search, so the agent can match exact terms like product codes as well as loosely worded questions.
How does an agent forget outdated memories?
Agents forget through expiry dates, decay scores that lower old memories' rank, or temporal records that mark a fact as no longer valid.
Is Mem0 or Letta better?
They solve different problems: Mem0 extracts compact facts from conversations, while Letta pages memory in and out of the context window, so the right choice depends on your architecture.
When should an agent search the web instead of using memory?
An agent should search the web when the answer changes often, such as prices or news, or when its stored memory is past its expiry date.
How do I evaluate memory retrieval?
Build a held-out set of questions that depend on past interactions and measure answer accuracy, latency, and tokens used before and after each change.