The Memory Stack: A Map of AI Memory

A context window is not durable memory. An agent that must recall preferences or prior work across sessions needs explicit state, storage, write policy, retrieval policy, and maintenance. These layers overlap with RAG but have different lifetimes and correctness requirements.

Define memory by its operations

Memory is state that the agent reads as input, writes after a call, or maintains between calls. Retrieval turns stored state into context, write policy decides what persists, and maintenance handles consolidation, deletion, conflict resolution, and embedding changes.

Model parameters contain learned knowledge but are not an operational memory store because a conversation cannot update them directly. Fine-tuning is a separate batched process; this article concerns state that the application can read and write.

The clearest formal frame is CoALA; Cognitive Architectures for Language Agents (Sumers, Yao et al., 2023), the Princeton paper that ported cognitive-science vocabulary into LLM engineering and named the four memory types every agent eventually grows: working, episodic, semantic, procedural. The shape of the contemporary stack hasn’t moved much since CoALA; what’s changed is the depth of each layer’s tooling and the empirical evaluations. The recent Memory in the Age of AI Agents survey (Hu et al., December 2025) cataloged the field at 47 authors’ worth of breadth and proposes the next finer-grained taxonomy (factual / experiential / working from the function angle; token-level / parametric / latent from the form angle). For an engineering audience, CoALA’s four-type frame is still the right one to start from; the survey is where you go when you need vocabulary for the edge cases.

Separate the four memory types

CoALA and the classical psychology taxonomy separate working, episodic, semantic, and procedural memory. Each type has its own lifetime, write pattern, retrieval method, and failure modes.

• Working memory. The scratchpad the agent uses during the current task: intermediate reasoning, partial plans, tool results not yet committed elsewhere. Lives in-context by default. Equivalent to the call stack of a running program. The working-memory article goes deep on the substrates: scratchpads, typed state objects, external notebooks, blackboards, and how each interacts with the conversation buffer.
• Episodic memory. “What happened”: a record of past interactions, observations, and outcomes, indexed by time and context. The user’s message from last Tuesday is an episode; the tool result you logged at 3:14am is an episode. Writes are appends; reads are recall queries. The closest distributed-systems analogue is a write-ahead log; append-only, ordered, replay-able.
• Semantic memory. “What is true”: generalized facts extracted from episodes (or seeded from external knowledge). “The user prefers dark mode” is semantic; “the API key rotates every 90 days” is semantic. Writes are distillations from episodes (or direct facts from outside the loop); reads are factual lookups. The analogue is a key-value store or a knowledge graph; the read pattern matters more than the write pattern, because semantic facts are read constantly.
• Procedural memory. “How to do X”: cached skill, learned patterns of action that succeed for a class of tasks. “To onboard a customer, do A then B then C” is procedural. Writes are slow (a procedure is only worth caching after you’ve seen it succeed a few times); reads happen at the moment the agent recognizes a familiar task shape. The analogue is a compiled-code cache: the JIT compiler caches hot paths because dispatching them is faster than recompiling. Procedural memory caches successful action sequences because reasoning them out from scratch is expensive.

The four types are not orthogonal in implementation; most production systems back episodic and semantic with the same vector store and just tag entries differently. They are orthogonal in purpose, and getting them confused is a common source of design pain. Storing every tool result as a “fact” in semantic memory bloats the knowledge base with episodes that should have been logged elsewhere. Storing distilled preferences as raw episodes makes them hard to find because they look identical to all the other observations the agent recorded.

Design read, write, and maintenance paths

The verbs are where the engineering hides. Every memory layer must answer all three independently.

Write policies decide what is worth storing. Saving every turn increases extraction cost and fills retrieval with low-value records. A practical policy classifies candidate memories, extracts a stable representation, checks for duplicates or conflicts, and persists only useful state.

Read policies answer “what’s relevant now?” The naive answer (“cosine-similarity top-K”) is the same RAG default that the retrieval cascade subtree spent eight articles improving on. Memory retrieval has the same surface but stricter signals available: recency (the episode from yesterday is usually more relevant than one from six months ago), importance (the episode you flagged at write time as critical should outrank the one you flagged as routine; see the episode segmentation and salience scoring piece for how that score is actually produced), and use-frequency (the fact the agent has retrieved 50 times is probably worth re-ranking up). The Generative Agents paper (Park et al., 2023) introduced the formulation that has become canonical: score = recency × importance × similarity, with each term in its own [0, 1] range. The memory retrieval policies article works the formula in detail; per-candidate normalization, the LRU/LFU/ARC cache-replacement parallel, read-driven last_read updates, and the Ebbinghaus-curve-shaped retrieval-count boost. Today’s frame: a memory system that uses only similarity will retrieve like RAG and forget like a goldfish.

Maintenance handles compaction, consolidation, reflection, deletion, conflicts, and embedding migrations. Without it, stale facts accumulate beside current ones and retrieval quality declines.

Distinguish memory from RAG

Memory and RAG overlap at retrieval but solve different lifecycle problems.

RAG indexes a static corpus; memory indexes a growing stream. A RAG index is built once (or periodically) over a known body of documents. A memory store has a write happening on every relevant turn, and the write distribution shifts as the conversation evolves. The retrieval pass over a memory store has to weigh recency in a way RAG doesn’t, because the marginal document being added today is more likely to be relevant than the one from a year ago.

RAG retrieves to ground a single response; memory retrieves to maintain identity across responses. A RAG hit on the right Wikipedia article makes the next sentence correct. A memory hit on the right user-preference fact makes the next thousand responses feel like they’re talking to the same agent. The metric of success is different: RAG measures recall@k and faithfulness on a per-query basis; memory measures multi-session consistency, the user’s perception of being known, the agent’s ability to avoid re-asking what it was told yesterday. Most of the canonical memory benchmarks; LongMemEval (Wu et al., 2024), LoCoMo (Maharana et al., 2024); measure this multi-session-recall axis explicitly. Standard RAG eval suites don’t. The memory evaluation article is the detailed treatment on these benchmarks and the custom-eval shape that wraps them.

RAG retrieval happens against a corpus the system didn’t author; memory retrieval happens against a corpus the system itself wrote. This sounds minor and is the source of half the failure modes. When the agent wrote the corpus, the agent has to distill before it writes (so the corpus doesn’t bloat with every utterance), reflect across writes (so the corpus develops higher-order claims), and resolve conflicts (so contradictory writes don’t poison retrieval). RAG inherits its corpus quality; memory builds its corpus quality.

Memory has a maintenance burden RAG doesn’t. A RAG index doesn’t need reflection, doesn’t need salience scoring, doesn’t need to handle the user contradicting themselves. A memory system needs all of these and the subtle thing is that they’re not optional: a memory system without them works fine for the first 100 sessions and then degrades.

RAG is a component of many memory systems. Episodic and semantic stores can reuse vector search, hybrid retrieval, reranking, and query transformation. Memory adds write policy, maintenance, temporal state, and coordination across stores.

Code: a minimal three-tier memory in Python

The smallest interesting build is a three-tier memory; in-context buffer, episodic vector store, semantic key-value store; with explicit read/write policies. The code below is deliberately framework-light so you can see what each layer is doing; later articles will replace pieces with Mem0, Letta, Zep, or LangGraph stores. Install: pip install anthropic chromadb. Uses the Anthropic SDK for the model and Chroma as the local vector store.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80

import time
import json
from anthropic import Anthropic
import chromadb

client = Anthropic()
chroma = chromadb.Client()
episodic = chroma.get_or_create_collection("episodic")
semantic: dict[str, str] = {}  # key-value: simplest semantic store

SYSTEM = (
    "You are a personal assistant with memory. When the user shares a durable "
    "preference, fact about themselves, or ongoing project, return a JSON line "
    "of the form {\"remember\": {\"key\": <slug>, \"value\": <fact>}} BEFORE your "
    "reply text. Only emit such a line when the fact is durable; do not store "
    "every utterance."
)

def write_episodic(session_id: str, role: str, text: str):
    """Append-only episodic log. Every meaningful turn goes here."""
    episodic.add(
        documents=[text],
        metadatas=[{"session": session_id, "role": role, "ts": time.time()}],
        ids=[f"{session_id}-{int(time.time()*1000)}"],
    )

def write_semantic(key: str, value: str):
    """Distilled, durable facts only. The model decides what qualifies."""
    semantic[key] = value

def read_memory(query: str, k: int = 5) -> str:
    """Compose the memory block injected before the user message."""
    # episodic: top-k by similarity, recency-weighted
    hits = episodic.query(query_texts=[query], n_results=k)
    now = time.time()
    scored = []
    for doc, meta in zip(hits["documents"][0], hits["metadatas"][0]):
        age_days = (now - meta["ts"]) / 86400
        recency = max(0.1, 1.0 - age_days / 30)  # decay over ~30 days
        scored.append((recency, doc))
    episodes = "\n".join(f"- {d}" for _, d in sorted(scored, reverse=True))

    # semantic: dump every known fact (small store; in prod, retrieve subset)
    facts = "\n".join(f"- {k}: {v}" for k, v in semantic.items())

    return f"## Known facts about user\n{facts or '(none)'}\n\n## Recent episodes\n{episodes or '(none)'}"

def parse_write(text: str) -> tuple[str, str | None]:
    """Strip the optional {'remember': ...} line and return (reply, write)."""
    lines = text.split("\n", 1)
    if lines[0].strip().startswith("{") and "remember" in lines[0]:
        try:
            payload = json.loads(lines[0])
            return (lines[1] if len(lines) > 1 else ""), payload["remember"]
        except (json.JSONDecodeError, KeyError):
            return text, None
    return text, None

def turn(session_id: str, user_msg: str) -> str:
    memory_block = read_memory(user_msg)
    write_episodic(session_id, "user", user_msg)

    resp = client.messages.create(
        model="claude-opus-4-7",
        max_tokens=1024,
        system=SYSTEM + "\n\n" + memory_block,
        messages=[{"role": "user", "content": user_msg}],
    )
    assistant_raw = "".join(b.text for b in resp.content if b.type == "text")
    reply, write = parse_write(assistant_raw)

    if write:
        write_semantic(write["key"], write["value"])
    write_episodic(session_id, "assistant", reply)
    return reply

# Day 1
print(turn("user-42", "I'm vegetarian and I'm planning a trip to Tokyo next month."))
# Day 7, new session
print(turn("user-42", "Can you suggest some places for lunch?"))

Writing is an explicit decision: the model emits a {"remember": ...} line only for durable facts, and the parser separates that instruction from the reply. Production systems should replace this heuristic with a calibrated classifier or structured-output schema. Retrieval uses recency-weighted similarity rather than raw cosine. Semantic and episodic memory remain separate stores because their write policies, read patterns, and growth rates differ.

The example omits reflection, compaction, conflict resolution, and provenance. A production store needs those maintenance and audit paths.

Further reading

• Cognitive Architectures for Language Agents (Sumers, Yao, Narasimhan, Griffiths, 2023): the canonical paper that named the four-type memory taxonomy and the working/long-term split for LLM agents. Required background for the rest of this subtree; it’s a long paper but the memory sections (§3, §4) are self-contained.
• Memory in the Age of AI Agents (Hu et al., December 2025): the most recent comprehensive survey, with the finer-grained factual/experiential/working and token/parametric/latent taxonomies and a catalog of every memory framework as of late 2025. Use as a reference encyclopedia; don’t read end-to-end.
• MemGPT: Towards LLMs as Operating Systems (Packer et al., 2023): the paper that made the OS analogy concrete and shipped a working virtual-context-management system. The intellectual root of Letta and the design model most modern memory frameworks borrow from.
• Generative Agents: Interactive Simulacra of Human Behavior (Park et al., 2023): the Stanford sandbox paper that introduced the memory-stream architecture, the recency × importance × similarity retrieval formula, and reflection as a first-class operation. The single most-cited reference in the memory literature; read this if you read nothing else.