$ cat ai-engineering/short-term-memory.md

Short-Term Memory: Managing the Conversation Buffer

How token budgets, protected messages, and head-tail eviction manage conversation context.

Jatin Bansal@blog:~/ai-engineering$ open short-term-memory

Short-term memory is the bounded message buffer sent on each model call. When it fills, deleting an arbitrary prefix can remove the system prompt, task definition, or unmatched tool messages. Eviction therefore needs explicit invariants, not a retry handler for context_length_exceeded.

The buffer contract

Short-term memory is the bounded message list that the harness sends to the model on each turn. It survives across turns in one session and disappears across sessions by default. A token, message, or application budget sets its size, so the harness must evict content before the next request exceeds that bound. Because this list is the prompt the model sees, eviction also determines which evidence remains available for reasoning.

The database may retain the complete conversation log. Short-term memory is the bounded working set that the harness reconstructs from that log for the next call. The log may grow; the working set must fit the prompt budget.

What belongs in the buffer

Every turn, the harness reconstructs a message array of the shape every modern Chat-Completions-style API accepts:

System prompt(s). One or more, but always at index 0. Sets behavior, injects pinned semantic facts (from the semantic memory tier), declares tools, declares output schemas.
Task anchor. The first user message of the session. The one that defines what the agent is trying to accomplish. Treated as protected in any non-trivial harness.
Retrieved context. RAG hits, memory-store retrievals, tool definitions injected just-in-time. Lives near the system prompt or just before the latest user turn, depending on the JIT-vs-AOT context engineering policy.
Conversation tail. The most recent N user/assistant turns and any tool-call/tool-result pairs they generated.
The current user turn. Always at the end. Always protected.

Eviction operates on the middle: the conversation between the task anchor and the recent tail. Within that middle, the harness chooses what to drop. Below are the policies that actually ship.

Policy 1: Hard sliding window

Keep only the last K messages. Drop the oldest. Linear, simple, and the default in nearly every framework’s getting-started tutorial.

text

1
buffer = system_prompts + history[-K:]

When this is right. Single-purpose assistants with tight token budgets where the next turn is overwhelmingly the most important context. Customer-support bots that handle one issue per session. Single-prompt chat UIs.

When this is wrong. Any session where the original user message matters past turn K. Coding agents (the task description matters at turn 200). Multi-step research agents (the original question matters even when the current step is debugging the JSON parser). The fix is the next policy.

Policy 2: Head-tail (task-anchored sliding window)

Keep the system prompt, keep the first user message, keep the last K messages. Drop the middle.

text

1
buffer = system_prompts + history[:1] + history[-K:]

When this is right. Any agent whose system prompt is the single most important context (almost all of them) and whose task description is the second most important context (most of them). This is the policy LangGraph’s trim_messages with include_system=True and strategy="last" defaults to.

The subtle bug. Hard cuts at the boundary between “first message” and “last K messages” can break message-pair integrity; you can end up with a tool result for a tool call that’s no longer in the buffer. Every production buffer manager must respect the tool-call/tool-result pairing invariant: if you drop a tool call, drop its result; if you drop a result, drop its call. Otherwise the model sees orphan messages and behavior gets weird.

Policy 3: Token-budget eviction

Same head-tail shape, but measured in tokens instead of messages. Count tokens for each message; keep adding from the tail backwards until you hit your budget, then add the head.

text

1
2
3
4
5
6
7
8
budget = model_context_window - max_reply_tokens - headroom
buffer = system_prompts + history[:1]
remaining = budget - tokens(buffer)
for msg in reversed(history[1:]):
    if tokens(msg) <= remaining:
        buffer.append(msg); remaining -= tokens(msg)
    else:
        break

Why this beats message-count. Messages have wildly variable sizes. A tool result returning 50KB of JSON is one message but eats more tokens than 200 short chat turns. Message-count eviction silently lets one fat tool result push the system prompt out of token budget; token-count eviction sees the size and evicts the fat result. This is the policy every production-grade harness ends up at.

How to count. Use the official tokenizer where available. Anthropic exposes a free count_tokens endpoint that accepts the exact message payload and returns the exact token count Anthropic will charge for. OpenAI ships tiktoken. Approximating with len(text) // 4 is fine for budget estimates but unsafe for hard cutoffs; overshoot the limit by 100 tokens because your estimate was off and the call fails. The cost of one tokenizer round-trip is dwarfed by the cost of one failed request.

Policy 4: Salience-based eviction

Instead of evicting the middle uniformly, drop messages tagged as low-importance first. Importance can come from a write-time classifier (“this is a routine clarification”) or from a learned salience score (lifted from the episodic-memory literature; the Generative Agents importance score is the canonical formulation, and the episode segmentation and salience scoring article works the anchored 1-10 prompt and its failure modes in detail). The classifier returns a number per message; the eviction pass drops the lowest-scored messages first until budget fits.

When this pays off. Long-running agents (research, coding, multi-day workflows) where some intermediates are critical and others are scratch. Pays the cost of a per-message importance call to get back token budget at the eviction step.

When this hurts. Short sessions where the classifier cost exceeds the value. A salience pass that calls the model is 100ms+ per message; for a 5-turn customer-support session you’ve spent more than the budget you saved.

Policy 5: Summarization-and-replace (preemptive compaction)

When token usage crosses a watermark (say 60% of the model’s context window), pause, summarize the oldest M messages into a single “summary” message, and replace them. The buffer keeps the system prompt + summary + recent tail. Repeat on every watermark crossing.

text

1
2
3
if tokens(buffer) > watermark:
    summary = summarize(buffer[1:-keep_tail])
    buffer = buffer[:1] + [summary_message(summary)] + buffer[-keep_tail:]

This is a summarization policy; the deep version belongs in the context-compression article in this Memory subtree. For short-term memory the relevant point: summarization is the more aggressive sibling of plain truncation. It preserves more semantic content per token but burns model calls to do so, and the summary itself is a lossy compression that you cannot undo. Run it when you have an actual reason to compress (long sessions, tight model budget); skip it when you don’t (short sessions where simple truncation works).

The frontier frameworks have all converged on this pattern as the production default. The OpenAI Agents SDK ships OpenAIResponsesCompactionSession as a decorator over any session backend, with a should_trigger_compaction hook that fires on a budget threshold. Anthropic exposes both server-side context editing with a trigger parameter (the model handles eviction transparently with the clear_tool_uses strategy) and an SDK-side compaction feature that summarizes the buffer when token usage hits the trigger. LangGraph composes the same idea with trim_messages plus a summarize_node you add to your graph. Different APIs, same algorithm.

Reserve headroom before eviction

The single most violated rule in real harnesses: never fill the context window to its stated limit, because the model has to output tokens too. A Claude Opus 4.7 1M-token window does not mean you can pack 1M input tokens; it means input + output must total ≤ 1M. If you pack 999K of input, the model has 1K to respond with, and any response longer than that gets truncated mid-token, often in the middle of a tool-call JSON, which breaks downstream parsing.

The defensible formula:

text

1
2
3
4
input_budget = context_window
               - max_reply_tokens         # what you set in max_tokens
               - safety_headroom          # for tokenizer estimation error
               - tool_result_headroom     # tools may return mid-call material

For a 200K-token Claude Sonnet 4.6 window with max_tokens=4096, a sane setup is:

max_reply_tokens = 4096 (matches max_tokens)
safety_headroom = 2048 (~1% buffer for estimation drift)
tool_result_headroom = 8192 (if this turn might call tools)
→ input_budget = 200,000 - 4,096 - 2,048 - 8,192 = 185,664 tokens

When the buffer crosses input_budget, evict or compact. When it crosses the hard limit, the call fails. The watermark is the place to act; the limit is the place to panic.

Account for lost-in-the-middle behavior

A 2023 paper from Nelson Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang; Lost in the Middle: How Language Models Use Long Contexts; quantified what most engineers had a gut feel for: model accuracy on retrieval-from-context tasks follows a U-shaped curve over the position of the relevant content. Performance is highest when the relevant content is at the beginning (primacy bias) or end (recency bias) of the prompt, and degrades by 20–30%+ when the relevant content is in the middle. The follow-up work Found in the Middle attributes the curve to positional-encoding decay in RoPE-based architectures, which is most modern LLMs.

Two design implications for short-term memory.

Putting the recently evicted summary in the middle matches the model’s attention pattern. The head and tail receive more attention, so the compressed middle does not occupy a privileged slot.

Second, the more you stuff into the middle, the less reliably the model reads it. Some teams “compress” by leaving lots of low-importance middle material in and trusting the model to filter. The model does not reliably filter. If you wouldn’t bet the agent’s behavior on a particular middle message being read, evict it. The middle is not a free zone; it’s an attention-degraded zone.

Code: a token-budget head-tail buffer in Python

The smallest production-shaped buffer manager. Uses the Anthropic SDK for both the model call and (importantly) the official count_tokens endpoint so the budget math matches what the model actually charges. Install: pip install anthropic.

python

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
import os
from anthropic import Anthropic

client = Anthropic()
MODEL = "claude-opus-4-7"

# Window math
CONTEXT_WINDOW = 200_000        # safe cap; raise if using the 1M tier
MAX_REPLY_TOKENS = 4_096
SAFETY_HEADROOM = 2_048
TOOL_HEADROOM = 8_192
INPUT_BUDGET = CONTEXT_WINDOW - MAX_REPLY_TOKENS - SAFETY_HEADROOM - TOOL_HEADROOM

def count_tokens(system: str, messages: list[dict]) -> int:
    """Authoritative token count via Anthropic's count_tokens endpoint."""
    resp = client.messages.count_tokens(
        model=MODEL, system=system, messages=messages,
    )
    return resp.input_tokens

def build_buffer(
    system: str,
    history: list[dict],
    user_msg: str,
    budget: int = INPUT_BUDGET,
) -> list[dict]:
    """
    Head-tail token-budget eviction.
    - System prompt is sent as the API's `system` param, not in messages.
    - The first user message (task anchor) is always preserved if it fits.
    - We grow the tail backwards atomic-pair by atomic-pair until budget breach.
    - tool_use / tool_result pairs are kept atomic (never orphaned).
    """
    current_turn = [{"role": "user", "content": user_msg}]
    if not history:
        return current_turn

    task_anchor = history[0]
    pairs = group_into_atomic_pairs(history[1:])  # everything after the anchor
    tail: list[dict] = []

    # Grow tail backwards, pair-by-pair, while budget allows.
    for pair in reversed(pairs):
        trial = [task_anchor] + flatten([pair]) + tail + current_turn
        if count_tokens(system, trial) > budget:
            break
        tail = flatten([pair]) + tail

    # Try the buffer with the task anchor; fall back to a summary line if too big.
    with_anchor = [task_anchor] + tail + current_turn
    if count_tokens(system, with_anchor) <= budget:
        return with_anchor

    summary = {
        "role": "user",
        "content": "[original task: " + truncate_for_summary(task_anchor["content"]) + "]",
    }
    return [summary] + tail + current_turn

# Atomic pair helpers — tool_use/tool_result must travel together
def group_into_atomic_pairs(history: list[dict]) -> list[list[dict]]:
    """Group adjacent tool_use → tool_result messages into single atomic units."""
    pairs, i = [], 0
    while i < len(history):
        msg = history[i]
        # Heuristic: an assistant turn that triggers a tool gets paired
        # with the next user message that holds the tool_result.
        if (msg["role"] == "assistant" and is_tool_use(msg)
                and i + 1 < len(history) and is_tool_result(history[i + 1])):
            pairs.append([msg, history[i + 1]])
            i += 2
        else:
            pairs.append([msg])
            i += 1
    return pairs

def flatten(pairs: list[list[dict]]) -> list[dict]:
    return [m for pair in pairs for m in pair]

def is_tool_use(msg: dict) -> bool:
    c = msg.get("content")
    if isinstance(c, list):
        return any(b.get("type") == "tool_use" for b in c)
    return False

def is_tool_result(msg: dict) -> bool:
    c = msg.get("content")
    if isinstance(c, list):
        return any(b.get("type") == "tool_result" for b in c)
    return False

def truncate_for_summary(content) -> str:
    text = content if isinstance(content, str) else str(content)
    return text[:200] + ("…" if len(text) > 200 else "")

# Demo: short-term memory turn
def turn(system: str, history: list[dict], user_msg: str) -> str:
    messages = build_buffer(system, history, user_msg)
    resp = client.messages.create(
        model=MODEL,
        max_tokens=MAX_REPLY_TOKENS,
        system=system,
        messages=messages,
    )
    text = "".join(b.text for b in resp.content if b.type == "text")
    history.append({"role": "user", "content": user_msg})
    history.append({"role": "assistant", "content": text})
    return text

the system prompt is the API’s system parameter, not message index 0; both Anthropic and OpenAI separate it out, so eviction logic never has to worry about accidentally dropping it. count_tokens is the authoritative source of truth; using a local estimator (len(text) / 4) is fine for development but a recipe for production context_length_exceeded errors. The task anchor is the first user message and is added last, not first; if it doesn’t fit, it falls back to a short summary line rather than evicting recent turns; “preserve the anchor” must not silently become “preserve nothing else.” Tool-use and tool-result pairs travel atomically; never break a pair, the model crashes hard on orphan tool messages. This is the same invariant Anthropic’s server-side clear_tool_uses strategy maintains.

The code is deliberately framework-light. A real LangGraph deployment would lean on trim_messages; an Agents SDK deployment would use OpenAIResponsesCompactionSession and let the runner handle the watermark. The principles are the same: a budget, a watermark, head-tail preservation, atomic pairs, authoritative token counting.

Short-Term Memory: Managing the Conversation Buffer

The buffer contract

What belongs in the buffer

Policy 1: Hard sliding window

Policy 2: Head-tail (task-anchored sliding window)

Policy 3: Token-budget eviction

Policy 4: Salience-based eviction

Policy 5: Summarization-and-replace (preemptive compaction)

Reserve headroom before eviction

Account for lost-in-the-middle behavior

Code: a token-budget head-tail buffer in Python

Further reading