Hierarchical Memory: Working / Episodic / Semantic Tiers

A long-running coding agent has every individual memory tier wired up correctly. The conversation buffer trims with head-tail eviction. The working-memory scratchpad tracks the plan and completed steps. The vector-backed long-term store holds three months of past episodes. The knowledge graph carries entities and bi-temporal validity. Every tier works in isolation. The agent still ships a regression: on turn 200 of a refactor it confidently asks the user a question whose answer is in the archival store, in the recall log, and on the user’s pinned profile card: three separate tiers, three separate copies of the same fact, none of them in the prompt the model just got. Each tier did its job. The system as a whole has no policy for which tier to read first, what to promote into the prompt, or what to demote out of it when budget tightens. That policy is what hierarchical memory is.

Definition

Hierarchical memory assigns each tier a fixed read-path role, an access cost, and explicit movement rules. Fast in-context memory is small and expensive per token; cold storage is larger and slower. Promotion moves frequently needed data toward the prompt, while demotion moves stale or low-value data away from it. The harness exposes these operations through a memory API so placement changes are visible and testable.

A tagged single store remains flat because its entries share one access path. RAG retrieves from a mostly static corpus; hierarchical memory also moves an agent’s own state between storage tiers. The cognitive taxonomy classifies information as semantic, episodic, procedural, or working memory. Hierarchy instead describes placement: a semantic fact may be hot when read on every turn or cold when rarely needed.

Paging as the systems model

The systems parallels are:

The agent’s memory tiers are a CPU cache hierarchy. L1 is core/in-context memory: tiny, sub-nanosecond latency, always-resident. L2/L3 is recall memory: medium-sized, single-digit-millisecond latency, paged in by reference. Disk is archival memory: effectively unbounded, tens-of-milliseconds latency, accessed by explicit query. The CPU cache hierarchy’s defining decisions: separate instruction and data caches, write-through vs write-back, inclusive vs exclusive caching: all have agent-memory analogues. The cognitive taxonomy article already pointed at the four-type-to-cache-tier mapping; hierarchical memory is the operational version of that mapping: the policies that decide when each tier gets read and written, not just what each tier is for.

MemGPT’s virtual context management is OS paging, almost literally. The MemGPT paper frames the LLM context window as physical memory and the persistent stores as disk; the system uses function calls as the page-fault mechanism to move information between the two. When the agent decides it needs a fact that isn’t in the prompt, it calls a recall_search or archival_search tool: the equivalent of a page-fault that traps to the kernel. The kernel (the harness) fetches the requested page (the matching memory entries) and writes them into the function-call return, which the model then sees on the next forward pass. The trap-and-fetch loop is identical in shape to a page fault on a modern OS; the only difference is that the unit of paging is a memory entry rather than a 4KB page. This is why the OS analogy is operational rather than aesthetic: the entire design vocabulary (page tables, working sets, TLB shootdowns, demand paging, write-back) ports over with very few modifications.

Promotion and demotion policies are cache-replacement policies. When the in-context block fills up, something has to give. The same algorithm families that govern CPU cache replacement (LRU, LFU, ARC, CLOCK) apply to the in-context block: Letta’s core memory blocks are explicitly designed for agent-driven LRU-ish replacement (the agent decides what to overwrite when a new fact takes priority). The memory retrieval policies article covers the read-time scoring formulas: per-candidate normalization, the LRU/LFU/ARC trade-off applied to episode rerank, read-driven last_read updates: and is the companion to today’s tier-replacement view. Memory-conflict handling covers what happens when promotion races with demotion: the agent equivalent of the cache-coherence problem in multi-core systems.

Three-tier architecture

The reference design extends the MemGPT paper’s two-tier model with the third tier used by Letta:

Core memory (the hot tier). A small, always-in-context block: typically a few hundred to a few thousand tokens: partitioned into named blocks. Letta’s defaults are persona (who the agent is), human (what the agent knows about the user), and a task block. Mem0’s equivalent is the pinned-facts layer. The block is always in the prompt, on every turn, and is the only tier the model can read without a tool call. The agent can self-edit it via core_memory_append and core_memory_replace tools, but the block is bounded: when it fills, the agent has to choose what to overwrite or what to demote to a slower tier.

Recall memory (the warm tier). Searchable conversation history, stored outside the prompt but queryable by tool call. Letta backs it with a database table of message turns; the agent invokes a recall_search tool (date filter or text filter) to retrieve matching turns into context for the current call. The retrieved turns are not persistent in the prompt: they’re injected as tool results for the current step and evicted naturally as the buffer rolls forward, the same way a paged-in disk page leaves the cache when the working set shifts. This is the tier that holds the long-term episodic store in its operational role: the per-turn or per-exchange log, indexed for similarity and recency.

Archival memory (the cold tier). Semantically searchable cold storage for facts, knowledge, summarized passages, and anything else that doesn’t need to be on the fast path. The agent invokes an archival_search tool (semantic similarity over an embedding index) to retrieve from this tier on demand. The archival tier is where reflection outputs land (distilled facts and higher-order beliefs derived from a window of episodes), where imported documents go, and where the agent’s procedural-memory equivalent (cached recipes, successful action sequences) usually lives. Letta backs it with a vector store; Mem0 backs it with its primary Qdrant/Chroma collection plus the optional Mem0g graph extension.

The reason this is three tiers and not five (working, short-term, long-term episodic, long-term semantic, procedural) is that the working/short-term split is within the core tier (different blocks of the same in-context surface), and the episodic/semantic/procedural split is within the archival tier (different metadata tags on the same vector store). The tier count is about access path, not about content type: three access paths, four content types, the matrix cells get populated as the workload demands.

Promotion: how bytes get hotter

A byte is in the warm tier; the agent needs it on every turn. How does it get promoted to the hot tier? Two policies, both worth knowing.

Agent-driven promotion (the MemGPT default). The agent reads a fact via recall_search or archival_search, recognizes it as something the user will reference repeatedly this session, and emits a core_memory_append tool call to pin it into the core block. The promotion is deliberate and visible; the agent has agency over what gets hot. The trade-off is that the agent has to be prompted to do this: without explicit instructions in the system prompt (“if you find yourself repeatedly looking up the same fact, pin it to core memory”), the model won’t promote on its own, and you’ll end up with a recall-thrashing pattern where every turn calls recall_search for the same fact.

Harness-driven promotion (the prefetch pattern). The harness, before the model runs, retrieves a small number of high-scoring entries from the warm or cold tier and injects them into the prompt as if they were in core memory. The agent doesn’t have to ask; the kernel pre-pages. This is the pattern Mem0’s memory.search follows when invoked at turn start: the harness embeds the latest user query, retrieves top-K facts from the vector store, and renders them as a “## Relevant memories” block in the system prompt. The trade-off is the inverse: the agent never knows a byte is promoted (it just appears in context), which means promotion costs token budget on every turn whether the agent would have asked for it or not. This is exactly the demand-paging-vs-prefetching trade-off from OS design, and the same answer holds: hybrid wins: pre-page the cheap, high-confidence pages (the user’s name, the active task) and demand-page the speculative ones.

Two write-time companions to promotion. Pinning is the explicit “this content is hot, do not demote without asking” annotation: the persona block in Letta, the user-profile facts pinned by Mem0’s classifier, the system-prompt fragments that an agent harness renders unconditionally. Hot-set learning is the inverse: the harness observes which warm-tier entries get promoted often and auto-promotes them ahead of demand, the same way a TLB amortizes the cost of repeated address translations. The hot-set is also what should seed a fresh session’s core block: the user’s most-promoted facts from the prior session are excellent priors for the new session’s hot tier, and the cross-session identity article covers the typed-record version of that seeding: the durable user profile, the session-start materialization, and the persona-clock model that decides what to seed when the same human is wearing a different hat.

Demotion: how bytes get cooler

The inverse problem and the harder one. The hot tier is small; when a new fact wants in, an old fact has to leave. The choice of which old fact to demote is the cache-replacement problem applied to agent memory.

Common demotion policies include:

LRU within the hot tier. Demote the core-memory block that has been read or referenced the least recently. Cheap to implement, well-understood, the default for almost every cache. The complication for agents: “read” is hard to detect: the model attends to in-context content implicitly, so the harness doesn’t see a clear read signal. The proxy is to track which blocks the model cites in its output (or which blocks were referenced in the last K turns) and demote the rest.

Agent-driven explicit demotion. The agent decides which core block is no longer relevant and emits a core_memory_demote (or archival_insert followed by core_memory_clear) tool call. This is what Letta defaults to: the agent self-manages its core blocks because it has the best signal for what’s no longer task-relevant. The trade-off is again prompt-sensitivity: the agent needs explicit instructions and tool affordances, and a tired prompt produces an over-pinned core block that never evicts.

Importance-decay demotion. Each core block carries an importance score (assigned at write time, like the Generative Agents importance score; the episode segmentation piece covers producing that score with an anchored 1-10 prompt and avoiding the all-7s collapse), and demotion is by lowest-importance-first. This works well when the write-time classifier produces calibrated scores and badly when it doesn’t; the failure mode is over-confident classifiers that mark every fact as a 10/10 and the demotion becomes effectively random.

The deeper problem with demotion is the loss-of-history failure mode. If a core block gets demoted to the warm tier, the warm tier should keep it: otherwise the demotion is silent data loss. Letta’s pattern is to insert into archival memory on demotion, preserving the fact as a searchable entry even when it’s no longer in-context. The harness invariant: demotion is never a deletion; it is a tier-shift. Deletion is a separate, explicit operation, and memory conflict and forgetting covers when explicit deletion is the right answer.

Placement by tier

A pragmatic guide, derived from production patterns across Letta, Mem0, and the MemGPT paper:

Core (hot) tier. The user identity (name, role, account). The active task description. A few critical preferences (“user is vegetarian, allergic to peanuts”). The agent’s persona/instructions for this conversation. The last 3-5 turns of conversation. Anything the model reads on every call. Bounded by token budget: typically 1-4K tokens in production.

Recall (warm) tier. The full conversation log for the current session and recent past sessions, indexed for date- and text-search. Tool-call results from earlier in the session that the model might need to re-examine. The full episodic store from long-term memory: every meaningful turn, retrievable by similarity and recency. Bounded by the storage backend, not by tokens: millions of entries are fine.

Archival (cold) tier. Reflection outputs: distilled facts derived from windows of episodes. Imported documents the agent should be able to consult (user-uploaded PDFs, account history, knowledge-base articles). Cached procedures from procedural memory. Older sessions’ summaries. The knowledge graph from the graph memory article often sits at this tier: entities and relations queried via traversal, not pulled into context speculatively.

The placement decision for any given fact reduces to one question: how often will the model read this on the average turn? If the answer is “every turn,” it’s a hot-tier candidate. If “occasionally,” warm tier. If “rarely but it must be findable,” cold tier. The classifier that makes this decision can be a heuristic (anything tagged persona is hot, anything tagged event is warm, anything tagged fact_with_low_priority is cold) or a model call at write time; the right choice tracks the workload’s variance. A workload where the hot-tier content is stable across sessions (a personal assistant for one user) can use heuristics; a workload where every conversation has different “always-relevant” content (a customer-support bot serving thousands of accounts) is closer to needing a learned classifier.

Three tiers with Letta

The smallest interesting build: a Letta-backed agent with core, recall, and archival memory, where the agent self-manages tier promotion via tool calls. Install: pip install letta-client and start a local Letta server (docker run -d -p 8283:8283 letta/letta:latest). Letta is the canonical productized version of MemGPT’s hierarchical architecture; rolling your own would reproduce roughly the same code.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57

# pip install "letta-client>=0.5"
from letta_client import Letta

client = Letta(base_url="http://localhost:8283")

# --------- agent creation: define the hot tier explicitly ---------
# Memory blocks are the core (hot) tier. Each block has a name, a value,
# and a token budget; the agent reads and self-edits them via tools.
agent = client.agents.create(
    name="travel-assistant",
    model="anthropic/claude-opus-4-7",
    embedding="openai/text-embedding-3-small",
    memory_blocks=[
        {
            "label": "persona",
            "value": "I am a travel-planning assistant. I remember the user's "
                     "dietary needs and travel constraints across sessions.",
            "limit": 1000,  # token budget; core block won't grow past this
        },
        {
            "label": "human",
            "value": "(empty: populate as you learn about the user)",
            "limit": 2000,
        },
    ],
)

# --------- conversation: the agent self-manages the hierarchy ---------
# When the agent learns something durable, it calls core_memory_append
# (promotes to hot tier) or archival_insert (demotes to cold tier).
# When it needs a fact not in context, it calls archival_search or
# conversation_search (warm tier).
def turn(user_msg: str) -> str:
    resp = client.agents.messages.create(
        agent_id=agent.id,
        messages=[{"role": "user", "content": user_msg}],
    )
    # Letta returns the full step trace; the last assistant message is the reply.
    return next(
        m.content for m in reversed(resp.messages)
        if m.message_type == "assistant_message"
    )

# Tuesday: the agent learns durable facts.
# Watch the agent self-promote to core memory (the human block).
print(turn(
    "I'm vegetarian, allergic to peanuts, and traveling with a toddler. "
    "Planning a 5-day Lisbon trip in July."
))

# Thursday: a fresh API call, the core memory persists across sessions.
# The agent has the user's profile in its hot tier without needing to ask.
print(turn("What restaurants should we try for lunch?"))

# Months later: a question about something only the recall/archival tier has.
# The agent demand-pages by calling conversation_search or archival_search.
print(turn("When we talked about Lisbon, did I mention any specific neighborhoods?"))

the memory blocks are explicit and bounded: the limit on each block is the hard token cap on the hot tier; the agent has to evict or summarize when it hits the cap, which is the cache-pressure signal that drives demotion. The agent self-manages the hierarchy via tools: Letta auto-injects the memory-management tools (core_memory_append, core_memory_replace, archival_insert, archival_search, conversation_search) and the system prompt explaining when to use each; the OS-paging operations are first-class to the agent rather than hidden inside the harness. The persistence is automatic: the persona and human blocks survive across messages.create calls because Letta stores them on the agent, the same way an OS process’s pinned pages survive context switches. The hierarchy isn’t a single-session feature; it’s the structural property that makes the agent stateful across sessions.

For deeper customization (custom blocks beyond persona/human, sleep-time agents that consolidate the archival tier in the background, multi-agent shared blocks), the Letta memory management docs are the reference. The sleep-time agent pattern: a secondary agent that runs in the background to consolidate fragmented memories, deduplicate the archival tier, and promote frequently-accessed warm entries into the hot tier: is the production version of the hot-set-learning idea, and the sleep-time compute article covers it.