$ cat ai-engineering/procedural-memory.md

Procedural Memory and Skill Caching

Procedural memory for agents using success-gated skills, task-shape keys, and retrieval-time validation.

Jatin Bansal@blog:~/ai-engineering$ open procedural-memory

A coding agent can recall previous deployments and repository facts yet still re-derive the same deployment sequence every week. Episodic memory records what happened, and semantic memory stores what is true. Procedural memory stores how a task succeeded: a reusable action sequence keyed by task shape and retrieved before planning begins.

The Voyager skill library; the canonical reference implementation

Voyager (Wang, Xie, Jiang et al., 2023) is the cleanest production-shaped example of a procedural-memory tier and is the implementation every LLM-era skill library is in conversation with. The mechanism, in five steps.

Step 1; Skill proposal. Voyager’s automatic curriculum generates a task (“mine 5 iron ore”) and proposes JavaScript code (using the Mineflayer bot APIs) that should accomplish it.

Step 2; Iterative refinement via execution feedback. The proposed code runs in the Minecraft environment. Execution errors, environment feedback (inventory state, world state), and a self-verification module are fed back into the prompt for another round of code generation. The loop runs until the task succeeds or a step budget exhausts.

Step 3; Skill admission on verified success. When the self-verification module confirms success, Voyager adds the successful JavaScript program to the skill library. The library is keyed by an embedding of the natural-language description of the skill (generated by GPT-3.5 as a separate description-generation step), not by the code’s text. The value is the JavaScript code itself.

Step 4; Skill retrieval at the start of a new task. For each new task, Voyager queries the skill library with the embedding of the task’s plan-and-environment-feedback string and retrieves the top-5 most relevant skills. The retrieved skills are inserted into the GPT-4 prompt as code examples; “here are skills that succeeded for similar tasks; use them as building blocks.”

Step 5; Skill composition. New skills built by the agent often call previously stored skills as subroutines, so the library compounds: a high-level skill (“build a stone pickaxe”) is implemented as a sequence of lower-level skill calls. The library grows from a few primitive skills to hundreds of composed routines over a Voyager run.

Voyager’s reported results are the empirical case that procedural memory pays off. The paper reports that Voyager obtains 3.3× more unique items, traverses Minecraft’s tech tree milestones up to 15.3× faster, and generalizes to fresh Minecraft worlds in ways the baseline (no skill library) does not. The mechanism is the JIT-cache parallel made concrete: the first iron-mining task takes the model many code-generation iterations to solve; the second iron-mining task retrieves the cached skill and executes it in one shot.

Voyager’s design transfers to other procedural stores in several ways.

Embedding the description, not the code. The retrieval key is the embedding of “mine iron ore in the overworld”; the task description. Voyager generates the description separately (a description-extraction LLM call after the code succeeds) precisely because the code itself is the wrong retrieval key. Procedures with semantically equivalent intent but different code shouldn’t cluster apart in embedding space; procedures with similar code but different intent shouldn’t cluster together. The description is the abstract over which similarity actually means something.

No repetition threshold. Voyager writes every first success, no waiting. This is the right call in Minecraft (a single self-verified success is reliable signal in a deterministic environment) and the wrong call in fuzzier domains. A customer-support agent shouldn’t cache a deeply-conditioned, single-success procedure as if it were a general skill. Production ports almost always add a repetition threshold of 2-3 successes before promotion.

Composition over re-derivation. Voyager’s most compounding design choice is that new skills call old skills. The library isn’t a flat set of recipes; it’s a hierarchy where “build a stone pickaxe” calls “mine wood” and “craft sticks” and “mine stone” as primitives. The LLM-era extension; function-call composition over a typed skill registry; is the property that turns a flat cache into a library, and it’s what makes the asymptotic cost of solving a complex task drop over time.

Keying schemes; the make-or-break design decision

The embedding key determines which procedures appear related. Common schemes have different failure modes.

Raw user request. Embed exactly what the user typed. Failure mode: surface-word brittleness. “Deploy the user service” and “ship the user service” miss each other. “Can you deploy the user service?” and “deploy the user service please” might cluster together for the wrong reason. Don’t use this for anything past a prototype.

Procedure text. Embed the code or the step list itself. Failure mode: code clustering on irrelevant similarities. Two procedures with similar control flow but different intents will cluster together; two procedures with the same intent but different implementations will cluster apart. The retrieval ranks on what the procedures look like, not on what they do. Wrong layer of abstraction for procedural memory.

Generated task description. Voyager’s choice. After a procedure succeeds, an LLM generates a normalized natural-language description of “what kind of task is this,” and the embedding of that description is the key. Failure mode: description-generator inconsistency; the same task gets two different descriptions on two different generation passes, and the resulting embeddings don’t cluster. Mitigated by using a single description-generator model with a stable prompt, and (in heavier setups) a regularization step that forces similar tasks toward similar descriptions.

Task-signature schema. A structured, multi-field key: {intent, object, environment, constraints} rather than free-text. The embedding key is then either a concatenated string or a structured retrieval over multiple fields. Failure mode: schema rigidity; workloads with task types the schema didn’t anticipate get key-collisions or are forced into the wrong cell. Mitigated by a flexible “free-text annotations” field that captures schema-overflow without breaking the structured retrieval.

Hierarchical procedure embedding. A two-level key: a coarse-grained domain embedding (e.g., “deployment-related”) plus a fine-grained sub-task embedding (“deploy the billing service”). Retrieval is a two-step rerank; filter to the coarse cluster, then rank within. Failure mode: cluster-boundary mismatch; a task that’s near the boundary of two coarse clusters gets routed to one and misses procedures in the other. Mitigated by querying both adjacent coarse clusters when the top-1 confidence is low, and by recalibrating cluster boundaries periodically against the empirical task distribution. The 2025 Procedural Memory Retrieval Benchmark (Liu et al.) reports that LLM-generated procedural abstractions outperform embedding-based methods on cross-context transfer; the cleanest evidence yet for the hierarchical-key approach.

In practice the right scheme is workload-dependent. Voyager runs on (3) and gets away with it because Minecraft’s task vocabulary is small and stable. AWM uses (3) augmented with the workflow’s task-class description. LangMem’s prompt-rule shape sidesteps the question (the “key” is implicit in which prompt section the rule lives in). Agent Skills uses (4); the name and description are structured fields. A production system in a fuzzy domain (general-purpose assistants, enterprise knowledge work) usually ends up at (4) or (5) after one rebuild.

Code: Python; a success-gated skill library with task-shape keying

The smallest interesting build: a procedural store with explicit success-gating, generated-description keying, retrieval-at-task-start, and graceful fallback when a retrieved procedure fails. The example uses Chroma for the embedding store and the Anthropic SDK for the description generation and the agent loop. Install: pip install anthropic chromadb.

python

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
import json
import time
import uuid
from dataclasses import dataclass, field
from anthropic import Anthropic
import chromadb

client = Anthropic()
chroma = chromadb.Client()
procedures = chroma.get_or_create_collection("procedures")

MODEL = "claude-sonnet-4-5"

@dataclass
class Procedure:
    proc_id: str
    description: str           # normalized task-shape description
    steps: list[str]           # concrete action sequence
    preconditions: list[str]   # what must be true for this to apply
    confidence: float          # promotion tier: 0.3 candidate / 0.7 eligible / 0.95 high
    success_count: int         # number of verified successes
    last_used: float           # for staleness checks
    env_version: str = ""      # dependency stamp; invalidates on env change


def generate_description(task: str, steps: list[str]) -> tuple[str, list[str]]:
    """Distill a normalized task-shape description and preconditions from a successful trace.

    This is the load-bearing step: the description becomes the retrieval key,
    and a poorly-generated one is the entire reason the store doesn't work.
    """
    resp = client.messages.create(
        model=MODEL,
        max_tokens=400,
        messages=[{
            "role": "user",
            "content": f"""I successfully completed this task:
TASK: {task}
STEPS: {json.dumps(steps)}

Generate:
1. A normalized, abstract description of WHAT KIND OF TASK this is
   (e.g., "deploy a service to production environment", not "deploy
   the user service to prod on Monday").
2. A list of preconditions that need to hold for this procedure to apply.

Return JSON: {{"description": "...", "preconditions": ["...", "..."]}}"""
        }],
    )
    text = resp.content[0].text
    parsed = json.loads(text[text.find("{"):text.rfind("}") + 1])
    return parsed["description"], parsed["preconditions"]


def write_procedure_candidate(task: str, steps: list[str], env_version: str = "") -> str:
    """Admit a successful trace as a CANDIDATE procedure.

    Success-gated: only called after the harness has verified the trace succeeded.
    Confidence starts at 0.3 (candidate); promotion to 0.7 happens after 2 more successes.
    """
    desc, preconds = generate_description(task, steps)
    proc_id = f"proc-{uuid.uuid4().hex[:8]}"
    procedures.add(
        documents=[desc],
        metadatas=[{
            "proc_id": proc_id,
            "steps": json.dumps(steps),
            "preconditions": json.dumps(preconds),
            "confidence": 0.3,
            "success_count": 1,
            "last_used": time.time(),
            "env_version": env_version,
        }],
        ids=[proc_id],
    )
    return proc_id


def promote_procedure(proc_id: str):
    """Bump confidence after additional verified successes."""
    existing = procedures.get(ids=[proc_id])
    if not existing["ids"]:
        return
    meta = existing["metadatas"][0]
    meta["success_count"] += 1
    if meta["success_count"] == 3:
        meta["confidence"] = 0.7   # eligible-for-injection
    if meta["success_count"] >= 10:
        meta["confidence"] = 0.95  # high-confidence; skip verification on retrieval
    meta["last_used"] = time.time()
    procedures.update(ids=[proc_id], metadatas=[meta])


def retrieve_procedures(task: str, env_version: str, k: int = 3,
                        min_confidence: float = 0.7) -> list[Procedure]:
    """Recognition-triggered retrieval: runs automatically at task start.

    Filters out stale procedures (env_version mismatch) and procedures below
    the eligibility threshold. Returns the top-K most relevant.
    """
    if procedures.count() == 0:
        return []
    hits = procedures.query(query_texts=[task], n_results=min(k * 3, procedures.count()))
    out: list[Procedure] = []
    for doc, meta, dist in zip(
        hits["documents"][0], hits["metadatas"][0], hits["distances"][0]
    ):
        # cache-coherence check: env mismatch invalidates the entry
        if env_version and meta.get("env_version") and meta["env_version"] != env_version:
            continue
        if meta["confidence"] < min_confidence:
            continue
        out.append(Procedure(
            proc_id=meta["proc_id"],
            description=doc,
            steps=json.loads(meta["steps"]),
            preconditions=json.loads(meta["preconditions"]),
            confidence=meta["confidence"],
            success_count=meta["success_count"],
            last_used=meta["last_used"],
            env_version=meta.get("env_version", ""),
        ))
        if len(out) >= k:
            break
    return out


def run_with_skill_library(task: str, env_version: str = "v1") -> str:
    """Agent loop that consults procedural memory before reasoning from scratch."""
    candidates = retrieve_procedures(task, env_version, k=3)

    if candidates:
        # Inject top procedures as candidate plans
        plans = "\n\n".join(
            f"[procedure {p.proc_id} | confidence {p.confidence:.2f}]\n"
            f"Description: {p.description}\n"
            f"Preconditions: {p.preconditions}\n"
            f"Steps: {p.steps}"
            for p in candidates
        )
        system = (
            "You are an agent with a procedural-memory skill library. "
            "Before reasoning from scratch, consider whether any of the "
            "retrieved procedures matches the task shape. If one does, follow "
            "its steps with appropriate adaptation. If none match, derive from "
            "first principles and the harness will record the new procedure on "
            "verified success.\n\n"
            f"Retrieved procedures:\n{plans}"
        )
    else:
        system = (
            "You are an agent. No procedural matches found for this task; "
            "derive the solution from first principles."
        )

    resp = client.messages.create(
        model=MODEL,
        max_tokens=1024,
        system=system,
        messages=[{"role": "user", "content": task}],
    )
    return resp.content[0].text


# Example usage
if __name__ == "__main__":
    # Day 1: agent derives the deploy procedure from scratch and succeeds
    deploy_steps_v1 = [
        "checkout main",
        "run pytest -q",
        "build docker image",
        "push to staging registry",
        "wait 45s for staging healthcheck",
        "promote to prod",
        "tag git release",
    ]
    proc_id = write_procedure_candidate(
        task="Deploy the billing service to production",
        steps=deploy_steps_v1,
        env_version="v1",
    )

    # Days 2-4: agent retrieves and re-executes; promote on each success
    for _ in range(2):
        promote_procedure(proc_id)

    # Day 5: a new deploy task arrives; the harness retrieves the cached procedure
    out = run_with_skill_library(
        task="Deploy the analytics service to production",
        env_version="v1",
    )
    print(out)

The write path is success-gated: write_procedure_candidate is called only after the task result has been verified. Retrieval occurs at task start, before prompt construction, so candidate plans are available without another tool decision. The environment-version stamp invalidates procedures after a relevant platform change. Confidence rises gradually with repeated success rather than after a single run.