The Cognitive Taxonomy: Semantic, Episodic, Procedural

A coding agent ships on Monday with a single in-context buffer and a vector store called “memory.” On Wednesday a senior engineer asks why the bot keeps re-deriving the same five-step deployment ritual it has executed forty times this week. On Thursday a PM asks why the bot re-asks every new user whether their codebase uses pnpm or npm. On Friday a junior engineer asks why the bot just told the new joiner that the team uses Postgres, when the team migrated to Cassandra three months ago. Three complaints, three different memory failures, all routed to the same vector store because the team was missing the same vocabulary. The Wednesday complaint is a missing procedural memory. The Thursday complaint is a missing semantic store. The Friday complaint is a missing conflict-resolution policy on the semantic store. Without separate names for these layers, every disagreement on the team becomes “we need better memory” and every fix is the wrong fix. The cognitive taxonomy: working, episodic, semantic, procedural: is how you stop talking past each other.

Memory types and their origin

The four-type frame is not a 2023 invention. It is a cognitive-science distillation that the LLM literature inherited largely intact, and knowing the lineage helps you reason about the edge cases.

Endel Tulving’s 1972 episodic-vs-semantic distinction drew the first sharp line: episodic memory is autobiographical and context-bound (it has a when and a where); semantic memory is factual and context-free. The cup of coffee you had on Tuesday morning at the corner café is episodic; “coffee contains caffeine” is semantic. Tulving’s later five-system taxonomy added priming, procedural, and perceptual memory, but the episodic/semantic line remained the critical one.

Larry Squire’s 1980s declarative-vs-non-declarative taxonomy added the second cut: declarative memory (episodic + semantic) is what you can say; non-declarative memory (procedural + priming) is what you can do. Procedural memory is “knowing how”: bike riding, touch typing, the muscle memory of the deployment script you don’t have to think about any more. The brain-systems evidence is overwhelming: amnesic patients can lose declarative memory entirely (they can’t remember anything new from yesterday) while retaining procedural memory (they can still learn new motor skills). The two systems are physically separate.

The CoALA paper: Cognitive Architectures for Language Agents (Sumers, Yao, Narasimhan, Griffiths, 2023): ported this vocabulary into LLM engineering by collapsing Tulving’s five into four and giving each a precise computational role: working memory for the current call’s scratchpad, episodic memory for past observations indexed by time, semantic memory for distilled facts about the world, procedural memory for cached action sequences. The split matters because each cell has different storage characteristics: and choosing the wrong substrate for a given cell is the most common memory design mistake in production agents. The recent SOAR architecture overview (Laird, 2022) makes the same split explicit on the symbolic-AI side and has done so since the 1980s; the LLM world is rediscovering what cognitive architectures already had.

Working memory: the L1 data cache

What it is. The scratchpad the agent uses during the current task: the partial plan being assembled, the intermediate tool results not yet committed elsewhere, the reasoning trace the model is iterating on. Lives in-context by default and is gone the moment the context window flushes. CoALA’s working memory is essentially “the bytes the model is staring at right now that aren’t part of long-term storage”: the running variables of the agent’s mental program.

Substrate. Almost always the prompt itself. In a ReAct agent the working memory is the thought/action/observation history; in a plan-and-execute agent it is the structured Plan object plus the executor’s running state. Some frameworks (LangGraph’s State, OpenAI Agents SDK’s session state, Letta’s core memory blocks) make working memory an explicit object in the harness rather than implicit in the prompt: but it still gets serialized into the prompt before every model call. The boundary is the same as the in-context vs storage boundary from yesterday: working memory is the in-context side. The working memory detailed treatment later in this subtree walks through each substrate (chain-of-thought scratchpad, typed dataflow graph, external notebook, multi-agent blackboard) with runnable code.

Write pattern. Constant during a task. Every tool result, every model output, every observation lands in working memory by default. The harness’s job is to keep working memory below the token budget; the context engineering article covered the JIT-vs-AOT mechanics that decide what stays in the working set and what gets paged out.

Read pattern. Implicit. The model reads working memory because the model reads the prompt; there is no separate retrieval step. Latency is the latency of the next attention pass.

Failure mode. Eviction without persistence. The classic anti-pattern is letting the harness silently truncate the oldest messages and assuming nothing important was in them. The deployment script the agent figured out in turn 12 gets evicted by turn 80 and the agent re-derives it from scratch in turn 81: which is the Wednesday complaint from the hook (partially; procedural memory is the more direct fix, see below). The fix is not “make working memory bigger.” The fix is to commit important working-memory items into a more durable tier: episodic or procedural: before eviction happens. Working memory’s job is speed, not persistence; using it as the storage tier is using L1 as your hard drive.

Episodic memory: the write-ahead log

What it is. “What happened, when, in what context.” An append-only, time-ordered record of past interactions, observations, and outcomes. The user’s question from Tuesday is an episode; the tool result you logged at 03:14 is an episode; the assistant turn where the agent committed a decision is an episode. Each episode carries enough context to be replayable in isolation: at minimum: the actor (user/assistant/tool), the timestamp, the content, and the session ID. Better systems also store an importance score and a coarse-grained “what kind of thing is this” tag for filtered retrieval. The long-term memory article is the detailed treatment on this tier as it actually ships in production: episode boundaries, write gates, the recency-weighted read path, and the framework choices that go with it.

Substrate. Almost universally a vector store with structured metadata: the text gets embedded for similarity search, and the metadata (timestamp, session, role, importance) supports filtering and recency-weighted ranking. The vector databases article covered the substrate choices; for episodic memory the salient knobs are filtered-ANN support and the cost of metadata-heavy queries. Mem0 keeps its episodic store in Qdrant or Chroma; Zep uses a hybrid Postgres + Neo4j layout where the temporal graph carries the metadata; Letta calls this layer “recall memory” and backs it with the conversation message buffer plus an embedded archive.

Write pattern. Append-only, every meaningful turn. The closest distributed-systems analogue is the write-ahead log: monotonic, ordered, replayable, never updated in place. Updates are achieved by appending a correction episode, not by rewriting the original: the same pattern as an event-sourced ledger, and for the same reason: the historical record is the source of truth, and rewriting history makes audit and rollback impossible. The write itself is usually cheap: embed the text, insert the row, done. The expensive part is the filtering: deciding which turns are worth recording at all. A naive “store every turn” policy works for a while and then falls over around the 10k-episode mark, when retrieval signal-to-noise collapses.

Read pattern. Recency-weighted similarity search, almost always. The canonical formulation is the Generative Agents memory-retrieval score (Park et al., 2023): score = α·recency + β·importance + γ·similarity, each term normalized to [0, 1], with exponential time decay on recency and an LLM-rated importance score assigned at write time. The signals are orthogonal: pure cosine retrieves on “what’s textually close,” recency biases toward what just happened, importance biases toward what the agent flagged as critical. The episode segmentation and salience scoring article works the anchored 1-10 importance prompt in detail, and the memory retrieval policies article is the detailed treatment on the read-side rerank that consumes it: per-candidate normalization, the LRU/LFU/ARC cache-replacement parallel, read-driven last_read and retrieval-count updates. For now the principle: episodic retrieval is not RAG retrieval, even when the substrate is the same vector store, because the read signals include time and salience.

Failure mode. Two big ones. First, write amplification: storing every utterance bloats the corpus, kills retrieval precision, and grows infrastructure costs proportionally. The fix is an explicit write policy: a classifier (heuristic or learned) that decides which turns earn an episode. Second, temporal staleness: a fact mentioned in March about the team using Postgres is still in the episodic store in May after the team migrated to Cassandra, and recency-weighted retrieval pulls the newer episode but not before the older one biases the prompt. This is the Friday complaint from the hook; the fix is not in episodic memory at all but in the semantic memory’s conflict-resolution policy, which we’ll get to in two sections.

Semantic memory: the keyed fact store

What it is. “What is true about the world that this agent should know.” Distilled, context-free facts extracted from episodes (or seeded from outside the loop). “The user prefers dark mode.” “The API key rotates every 90 days.” “The team uses Cassandra as of March 2026.” Semantic memory is what makes the agent feel like it knows the user/system/domain rather than constantly relearning them.

Substrate. Variable, and the choice matters more than for episodic memory. Three common shapes:

• Key-value or document store for explicitly keyed facts: user_preferences.theme = "dark", infra.database = "cassandra". Read pattern is direct lookup, which is the fast path. Mem0’s “facts” layer and Letta’s core-memory blocks (the user/persona/task blocks pinned in-context) are this shape. Best for facts the agent will read on every call.
• Vector store with structured tags for facts whose keys are not known in advance: “things the user has expressed an opinion about,” “things about the deployment environment.” Read pattern is similarity search; latency is a retrieval round-trip. Best when the corpus of facts is large enough that pinning all of them in-context isn’t tractable.
• Knowledge graph when relationships between facts matter: who reports to whom, which service depends on which, which preference was stated by which family member. Graphiti and Zep’s graph backend take this approach. The graph-memory article goes deep on when graphs beat vectors; the short version is “when your facts have entity-relation structure, when temporal point-in-time queries matter, or when you need to traverse multi-hop relations.”

Write pattern. Slow, distillation-gated. A new semantic fact gets written when the agent (or a write-time classifier) decides an episode contains a durable, generalizable claim worth lifting out. The write is usually a model call: take the recent episode, return a JSON object of the form {"key": "...", "value": "...", "confidence": 0.8}. The Mem0 paper (Chhikara et al., 2024) shows empirically that aggressive write-time filtering is where most of the recall accuracy comes from: and that the naive “extract every fact you can” policy actively hurts downstream retrieval because the corpus fills with low-value claims.

Read pattern. Two-mode. Keyed facts are read on every relevant call (the agent’s name, the user’s known preferences, the active task: what Letta pins as “core memory”). Searched facts are retrieved on demand by similarity. The two modes are not interchangeable: a fact that needs to be present on every call should be pinned, not searched, because the recall@k of similarity search is never 1.0. A common production mistake is putting the user’s name in the searchable store and then wondering why the agent occasionally addresses them by the wrong name.

Failure mode. Three.

1. Stale facts. “The team uses Postgres” written in March is still present in May after the migration. The Friday-hook complaint. The fix is a conflict-resolution policy at write time: when a new fact for an existing key arrives, the system has to decide whether to overwrite, append-with-timestamp, or flag for human review. Most production systems are too permissive here; they let the latest write win silently, which means a single hallucinated extraction can corrupt a long-lived fact. A more defensible default is timestamped versions with a confidence threshold for overwrite: the new value has to clear a confidence bar, and the old value is retained as a historical version.
2. Over-extraction. Storing every observation as a “fact” pollutes the store with episodes that should have been logged in episodic memory. A good write policy classifies the extraction type: is this a durable fact (semantic), an event (episodic), or both? Most things are both, but the primary home matters because reads against the wrong store retrieve the wrong shape of result.
3. The cold-start problem. A fresh agent has no semantic memory; it needs to bootstrap from somewhere. Three common starting points: seed from a static knowledge base (the customer’s product documentation), seed from the user’s existing profile (their CRM record), or accept the cold-start cost and let the agent learn from interactions. The choice depends on the use case; the trap is pretending the cold-start problem doesn’t exist. The cross-session identity article is the detailed treatment on the user-profile flavor: the cold-start staircase that ascends from anonymous to claimed to bootstrapped to matured as the data arrives, and the confidence-marker discipline that keeps the early steps honest.

Procedural memory: the JIT-compiled-routine cache

What it is. “How to do X.” Cached action sequences that succeed for a class of tasks. “To deploy this service, do checkout → build → migrate → push → verify.” “To onboard a new user, do welcome-message → preference-collection → first-task-suggestion.” Procedural memory is the agent’s learned skills, separated from both the conversation history (episodic) and the world facts (semantic) because they have a fundamentally different lifecycle: they’re written rarely, after success has been observed; they’re read frequently, every time a task with the right shape appears.

Substrate. Almost always a separate index: distinct from episodic and semantic: keyed by task description embeddings, valued by the code/prompt/plan that succeeds for that task. Voyager (Wang et al., 2023), the Minecraft lifelong-learning agent, is the cleanest production-shaped example: its skill library is JavaScript functions indexed by the embedding of their natural-language descriptions, and on each new task the top-5 most-relevant skills get retrieved and injected into the prompt. The result is striking: Voyager unlocks Minecraft tech-tree milestones up to 15.3× faster than prior agents, and the skill library generalizes to fresh Minecraft worlds. The mechanism is exactly the JIT-compiled-routine cache from a virtual machine: compile the hot path once, retrieve it next time, skip the cost of re-deriving it.

Write pattern. Slow and success-gated. A procedure is only worth caching after you’ve seen it succeed: once is a coincidence, twice is suggestive, three times is a pattern. A defensible write policy is to record candidate procedures on success (with the task description, the action sequence, and a usage counter starting at 1) and promote them to the retrieval-eligible tier only after they’ve been used or re-derived N times. Voyager actually skips the “see it three times” check and writes every successful skill; this works in their setting because Minecraft tasks are deterministic enough that one success is strong signal. In a fuzzier domain (customer-support workflows, code-review heuristics) you want a higher bar.

Read pattern. Fast retrieval at the recognition moment: when the agent receives a new task, the harness runs a similarity search against the procedural store before the model starts thinking, and injects the top matches as candidate plans. This is the JIT inlining step. The latency of the retrieval is amortized against the much larger latency of the model re-deriving the procedure from scratch.

Failure mode. Two.

1. Brittleness to task drift. A cached procedure is a snapshot of “what worked last time,” and the environment may have changed. A deployment script that worked in March may fail in May because the build system was upgraded. The fix is graceful fallback: when a retrieved procedure fails, the agent should drop back to first-principles reasoning and (if successful) update the procedural store with the new variant. This is the agent equivalent of cache invalidation, and it has the same hard problem at the core: knowing when the cached version has gone stale.
2. Over-generalization. A procedure that worked for one user’s deployment doesn’t necessarily work for another’s. The fix is good keying: the embedding the procedure is indexed by needs to capture the task shape, not just the surface words. “Deploy the user-service” and “deploy the billing-service” should retrieve similar procedures; “deploy the user-service to prod” and “deploy the user-service to staging” should retrieve different ones. The keying scheme is the design decision that makes or breaks the procedural store.

How the memory types interact

The interactions matter more than the storage layout. Three pairs to keep straight:

• Episodic feeds semantic via reflection. Higher-order semantic claims get extracted from accumulated episodes. The agent reads a hundred episodes about the user’s interactions and writes a semantic fact “this user is technical, prefers terse responses, dislikes emoji.” Reflection is the maintenance step that turns episodic raw material into semantic distillate: the dedicated piece covers the Generative Agents importance-threshold trigger, the salient-question-then-evidence-anchored-insight pipeline, and the self-reinforcing-error failure mode. An agent that has episodic memory but no reflection has a memory that grows but doesn’t learn.
• Episodic feeds procedural via success-attribution. When an agent succeeds at a task, the harness can scan the episodic log of the just-completed task, extract the action sequence, and write it to the procedural store. This is the cache-warming step for procedural memory and it’s typically run as a post-task hook, not in-line.
• Semantic and procedural are read together at task start. When a new task arrives, the agent typically reads relevant semantic facts (“this user prefers terse responses”) and relevant procedural skills (“here’s how you did this kind of task last time”) in the same retrieval pass, even if they live in separate stores. The two together are the agent’s context for the task; episodic memory is consulted later, only if the agent decides it needs to.

The clean version of the read order at task start is: pin core semantic facts → retrieve top-K procedural matches → retrieve relevant episodic context if needed → run. Most failed memory designs get this read order wrong by either pulling everything every turn (token explosion) or pulling nothing and falling back on the model’s parametric knowledge (the Thursday complaint from the hook: re-asking the package manager question because the answer isn’t in any memory tier, just lost in last week’s vanished conversation buffer).