$ cat ai-engineering/long-term-memory.md

Long-Term Memory: Vector-Backed Episodic Storage

Long-term episodic memory with vector storage, episode boundaries, selective writes, and recency-weighted retrieval.

Jatin Bansal@blog:~/ai-engineering$ open long-term-memory

A long-term memory tier lets an agent reuse relevant information after a session ends. Pasting complete prior transcripts into every prompt is expensive and noisy; durable episodic storage keeps selected experiences outside the prompt and retrieves them only when needed.

The storage contract

Long-term episodic memory is a durable, append-mostly store of past agent experiences. Each entry represents a discrete event such as a user turn, tool result, or completed interaction and includes enough metadata to stand alone. Corrections usually append a superseding episode instead of rewriting history. Entries stay outside the prompt until the harness retrieves them, which makes every recall consume retrieval latency and context budget.

The conversation buffer is in-context and session-scoped. A working-memory scratchpad is mutable task state. Semantic memory contains distilled facts, often extracted from episodes, and procedural memory contains reusable instructions. These forms may share one database, but their read and write policies remain different.

Choose the unit of recall

Three common granularities, in order of how much each is worth using.

Per-message episode; each user or assistant turn is its own row. Highest fidelity, highest cardinality. Best when the agent needs to retrieve specific things the user said (“when did the user mention they were vegetarian?”). Worst when relevance is contextual; a one-line “yes” message is meaningless without its question, but a per-message episode strips that pairing. Mitigation: store the previous-turn ID as metadata and re-fetch the pair when a hit lands on a short reply.

Per-exchange episode; each user turn plus the assistant’s reply is one row. Better contextual integrity, lower cardinality. The dominant default in production frameworks. Mem0 defaults to pair-shaped writes through its add(messages, user_id=...) API; the framework processes the message list and emits one or more memory entries per call.

Per-session episode; each session compresses to a single summary row with a date range. Lowest fidelity, lowest cardinality. Works when the unit of “I’d like to recall this past interaction” is the whole session (“the conversation we had last Tuesday about Lisbon”). Almost always paired with one of the higher-fidelity tiers; you want the session summary for fast similarity search and the per-message entries for the detailed lookup once a session is retrieved.

The right choice is workload-dependent. A customer-support agent recalling “did the user mention which OS they were on?” wants per-message episodes. A long-running personal assistant recalling “what were we working on last week?” wants per-session summaries. Most production systems end up running two tiers in parallel; a summary tier for navigation and a per-exchange tier for the details; which adds storage cost but pays for itself in retrieval quality. The memory benchmark literature is consistent on the point: hierarchical granularity outperforms flat-per-message in every long-multi-session benchmark published in the last 18 months. The full mechanics of deciding the unit at write time; the segmentation algorithm, the cognitive-science grounding from event segmentation theory, and the layered combination of fixed-window, semantic-shift, prediction-error, structural, and agent-emitted signals; get their own deep dive in the episode segmentation and salience scoring article.

Gate writes by future value

The naive “store every turn” policy works for a while and then collapses. The collapse happens around the 1k-episode mark, when retrieval signal-to-noise crosses the threshold where the right episode is still in the store but no longer in the top-K. A defensible write policy classifies each turn before writing.

Three policies in increasing sophistication:

Heuristic write gate. Skip system messages, skip empty assistant turns, skip pure clarifications (“Can you repeat that?”). Cheap, no model call, catches the worst noise.
LLM-classified write gate. A small-model call returns {should_write: bool, importance: float, type: "preference"|"fact"|"event"}. Slower, ~50ms per turn at small-model latencies, but the precision is dramatically better. This is the policy Mem0’s fact-extraction pipeline uses, and the Mem0 paper attributes most of their recall lift to it.
Deferred write at session end. Skip per-turn writes entirely; at session close, run a summarization pass over the whole session and write 2–5 distilled episodes. Cheapest at write time, lowest fidelity, ideal for short bounded interactions.

The trap is to skip the write policy and tell yourself you’ll add it later. By the time you have 10k uncurated episodes, the cost of cleaning the corpus exceeds the cost of building the gate from the start. The memory write policies article covers the classifier design; the four-stage pipeline, the journal-and-checkpoint pattern, and the hot-path-vs-deferred-vs-background trade-off; the rule of thumb for now is have a policy, even a heuristic one, on day one.

Rerank recalled episodes

The retrieval pass over an episodic store is not the same as RAG retrieval over a static corpus. Pure cosine similarity gives the agent the most textually similar episode, which is often not the most useful one; the user’s preference from last month is more useful than a textually similar message from a year ago that has since been contradicted.

The canonical formulation, from Generative Agents (Park et al., 2023), is a weighted combination:

text

1
score(episode | query) = α·recency(episode) + β·importance(episode) + γ·similarity(query, episode)

with each term normalized to [0, 1]. Recency uses an exponential decay since the episode was last retrieved (not just written; episodes that keep proving useful keep their freshness); importance is an LLM-assigned 1–10 score at write time, normalized; similarity is cosine over the embedding. The weights are typically equal (α = β = γ = 1) as the published baseline; tuning them is a workload-specific calibration step. The Generative Agents paper rated importance with a prompt that explicitly anchored “1 = mundane (brushing teeth)” and “10 = pivotal (entering college, getting divorced)”; that anchoring matters; LLM ratings without anchored scales drift toward all-5s. The episode segmentation piece works the anchored 1-10 salience prompt in detail (and the per-segment-vs-per-turn unit question), so this section can stay focused on the read-time rerank.

In production this means every read happens in two stages: a coarse vector recall pulls the top-K candidates by similarity (K ~ 3-5× the final budget), then the rescore stage applies the recency and importance terms and returns the top-N. This is the same two-stage retrieve-then-rerank pattern that wins on RAG, with different rerank signals. The memory retrieval policies article works the formula in detail; per-candidate normalization, read-driven last_read and retrieval-count updates, the LRU/LFU/ARC cache-replacement parallel, and the workload-specific weight tuning; and is the read-path companion to today’s substrate. Today’s frame is: don’t ship cosine-only retrieval against an episodic store and expect it to feel like memory.

Code: Python; episodic store with recency-weighted retrieval

The smallest interesting build: an LLM-gated write path, a recency-weighted read path, and a turn loop that demonstrates cross-session recall. Uses the Anthropic SDK for the model and Chroma as the local vector store. Install: pip install anthropic chromadb.

python

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
import json
import math
import time
import uuid
from anthropic import Anthropic
import chromadb

client = Anthropic()
chroma = chromadb.Client()
episodes = chroma.get_or_create_collection("episodes")

MODEL = "claude-opus-4-7"

# --------- write path: classify, then maybe store ---------
WRITE_GATE_PROMPT = """Decide whether this turn is worth durably remembering for
future sessions. Return JSON only.

Schema: {"should_write": bool, "importance": int 1-10, "type": "preference"|"fact"|"event"|"none"}

Importance scale: 1 = mundane filler, 5 = useful context, 10 = pivotal fact about
the user or task that would change future answers.

Turn:
- role: {role}
- content: {content}"""

def classify_for_write(role: str, content: str) -> dict | None:
    resp = client.messages.create(
        model="claude-haiku-4-5",  # small-model gate keeps the write path cheap
        max_tokens=200,
        messages=[{"role": "user",
                   "content": WRITE_GATE_PROMPT.format(role=role, content=content)}],
    )
    try:
        verdict = json.loads("".join(b.text for b in resp.content if b.type == "text"))
        return verdict if verdict.get("should_write") else None
    except (json.JSONDecodeError, KeyError):
        return None  # malformed -> skip; never silently store junk

def write_episode(session: str, user: str, role: str, content: str):
    """Append-only write. Classification is what makes this episodic, not log-spam."""
    verdict = classify_for_write(role, content)
    if not verdict:
        return
    episodes.add(
        documents=[content],
        metadatas=[{
            "session": session, "user": user, "role": role,
            "ts": time.time(), "last_read": time.time(),
            "importance": verdict["importance"] / 10.0,
            "type": verdict["type"],
        }],
        ids=[str(uuid.uuid4())],
    )

# --------- read path: recency × importance × similarity ---------
def read_episodes(user: str, query: str, top_n: int = 5, recall_k: int = 20) -> list[dict]:
    """Two-stage: coarse vector recall, then rescore with recency and importance."""
    hits = episodes.query(
        query_texts=[query],
        n_results=recall_k,
        where={"user": user},  # tenant isolation; never skip this in multi-user systems
    )
    if not hits["ids"][0]:
        return []
    now = time.time()
    scored: list[tuple[float, str, dict, str]] = []
    for doc_id, doc, meta, distance in zip(
        hits["ids"][0], hits["documents"][0], hits["metadatas"][0], hits["distances"][0]
    ):
        # Recency: exponential decay since last read, half-life ~7 days
        age_days = (now - meta["last_read"]) / 86_400
        recency = math.exp(-age_days / 7)
        importance = meta["importance"]
        # Chroma returns L2 distance; convert to a [0,1] similarity proxy
        similarity = 1.0 / (1.0 + distance)
        # Equal weights baseline; calibrate per workload.
        score = recency + importance + similarity
        scored.append((score, doc_id, meta, doc))
    scored.sort(reverse=True)
    # Update last_read for retrieved episodes — episodes that keep proving useful
    # stay fresh; episodes nobody reads decay out of the top-K naturally.
    for _, doc_id, meta, _ in scored[:top_n]:
        meta["last_read"] = now
        episodes.update(ids=[doc_id], metadatas=[meta])
    return [{"text": doc, "meta": meta} for _, _, meta, doc in scored[:top_n]]

# --------- turn loop ---------
SYSTEM_TEMPLATE = """You are a personal travel assistant with episodic memory.

## Relevant past episodes (retrieved on demand)
{episodes}

Use the episodes to answer in a way consistent with what the user has told you
before. Do not re-ask things the episodes already answer."""

def turn(session: str, user: str, user_msg: str) -> str:
    retrieved = read_episodes(user, user_msg, top_n=5)
    episodes_block = "\n".join(
        f"- [{e['meta']['type']}, importance={e['meta']['importance']:.1f}] {e['text']}"
        for e in retrieved
    ) or "(none)"
    resp = client.messages.create(
        model=MODEL,
        max_tokens=1024,
        system=SYSTEM_TEMPLATE.format(episodes=episodes_block),
        messages=[{"role": "user", "content": user_msg}],
    )
    reply = "".join(b.text for b in resp.content if b.type == "text")
    write_episode(session, user, "user", user_msg)
    write_episode(session, user, "assistant", reply)
    return reply

# Tuesday session
print(turn("s-001", "u-42", "I'm vegetarian, allergic to peanuts, traveling with a toddler."))
# Thursday session — different session ID, same user
print(turn("s-002", "u-42", "What should I eat for lunch in Lisbon?"))

the write path is gated by a small-model classifier; the gate is what separates “episodic memory” from “transcript dump.” A naive harness that writes every turn will fill the store with noise inside a week. The where={"user": user} filter is mandatory; long-term memory without tenant isolation is a data-leak waiting to happen, and the next time the agent serves user B it must not retrieve user A’s episodes. last_read decays, not ts; episodes that keep proving useful keep their freshness; this is the LRU/LFU hybrid the Generative Agents paper formalized, and it matters more than the absolute decay constant. The single biggest bug I see in hand-rolled memory implementations is decaying on write timestamp only; an episode the agent has retrieved 50 times should not decay at the same rate as one written and never read.

Framework choices

Three frameworks worth knowing by the boundary they draw between episodic and semantic stores.

Mem0 runs an LLM-gated fact-extraction pipeline at write time: add(messages, user_id=...) accepts a conversation, extracts durable facts and preferences, and writes them as memory entries with type tags. The result is closer to “semantic memory with episodic timestamps” than a pure episodic log; Mem0 prioritizes the distilled claim over the raw episode. Their published numbers (Mem0 paper, Chhikara et al., 2024) show large recall gains over per-message storage and dramatic cost reduction at retrieval time; the 2026 numbers from their State of AI Agent Memory 2026 post extend that with the Mem0g graph extension. Best fit: workloads where the user’s preferences and durable facts matter more than the chronological log.

Letta (the MemGPT descendant) separates recall memory; raw conversation messages stored verbatim, semantically searchable; from archival memory; processed, summarized, indexed passages; and pins a small core-memory block in-context with persona, human, and task fields. Recall is the episodic log; archival is closer to semantic; core is the working-memory equivalent. The three-tier shape maps directly onto the cognitive taxonomy and is the closest off-the-shelf framework to “all four CoALA tiers as first-class concepts.”

LangGraph stores are a lower-level primitive: a JSON-document store organized by namespace and key, with optional embedding-based search. LangGraph deliberately does not enforce a memory model on top of the store; you decide whether your namespace holds episodes, facts, procedures, or a mix. The companion LangMem SDK layers an opinionated memory model (semantic, episodic, procedural namespaces with hot-path and background management) on top of the store API. Best fit: teams that want to build their own memory model and need a clean storage primitive rather than a prescriptive framework.

OpenAI Agents SDK ships a session abstraction with persistent backends (SQLAlchemySession, Redis, MongoDB, Dapr, advanced SQLite) but treats sessions as the short-term tier; long-term episodic memory is a layer you add on top, typically using one of the frameworks above or rolling a vector-store-backed get_items/add_items extension. The production memory frameworks article works the full Letta/mem0/Zep/Graphiti comparison matrix; today’s takeaway is that the three frameworks above represent three different stances; distill at write (Mem0), log everything + tiered retrieval (Letta), give me a store primitive and I’ll build the model (LangGraph); and the right choice is downstream of which stance fits your workload.