Summarization and Context Compression

A coding agent has been working on the same refactor for three hours. The conversation buffer is at 180,000 tokens: sixty tool calls, forty file reads, twenty edit attempts, the original task brief, and the user’s incremental clarifications scattered across the timeline. The harness fires its compaction pass at the watermark, drops the buffer to 30,000 tokens of summary, and continues. Five turns later the agent calls Read on src/auth/handler.ts, reads the file fresh, and announces it will “now implement the change requested earlier.” The user, watching, types: you already implemented that change two hours ago, in the commit you summarized as “auth refactor.” The agent has no record of the commit. The summary said “implemented auth refactor”; the file path is gone, the diff is gone, the decision rationale is gone. The conversation lost its grip on its own history at the moment it tried to keep it. Context compression must preserve enough detail to prevent that failure.

The compression contract

Context compression replaces a span of conversation history or memory entries with a shorter summary, structured note, compressed token stream, or curated subset. The replacement becomes the model’s live context, information is necessarily lost, and the operation is triggered by token, cost, or latency pressure.

Truncation drops messages whole, while compression rewrites them. Reflection derives higher-order claims across episodes; compression retains the episodes in shorter form. Chunking sets retrieval units at write time, whereas compression acts on the runtime buffer.

Compression strategies

The market has converged on four distinct strategies. Knowing them by name is what makes the framework choices downstream legible.

Strategy 1: Recursive (hierarchical) summarization

The seminal pattern, formalized in Wang et al.’s “Recursively Summarizing Enables Long-Term Dialogue Memory in Large Language Models” (2023, journal version 2025). The mechanism: maintain a running summary; on every N turns, prompt the model to regenerate the summary by combining the previous summary with the new turns; the regenerated summary replaces the old one. The recursion is in the update step: each new summary is a function of (previous_summary, new_turns), and the previous_summary is itself the recursive result of all earlier compressions.

The pattern’s claim to fame is holistic regeneration. Hard append-or-delete summary updates fragment the summary over many cycles; a regeneration update produces a coherent rewrite each cycle. The cost is one full model call per update, with the summary token count growing slowly over time (each new event adds a few tokens to the summary, then the regeneration prunes the older now-less-relevant parts).

When it’s right. Long-running conversations where the gist matters more than the verbatim history. Personal AI companions, customer-service follow-ups, the kind of agent where “what did we agree to do?” is the dominant read query, and “what was the exact line you said?” is rare.

When it’s wrong. Coding agents, debugging agents, anywhere the verbatim content of an earlier turn is what the next call needs (file paths, error messages, exact configuration values). A recursive summary will paraphrase src/auth/handler.ts:142 into “the authentication handler” and the agent will spend the next ten turns re-reading the codebase to find what it once knew.

Strategy 2: Structured note-taking (anchored sections)

The 2026 production answer to recursive summarization’s paraphrasing problem. Productized by Factory.ai’s anchored iterative summarization and adopted in shape by every coding agent’s harness layer. The mechanism: define a fixed schema for the summary: sections for session_intent, files_modified, decisions_made, pending_questions, next_steps: and prompt the summarizer to populate each section explicitly. The schema is the structure-vs-prose distinction: a prose summary can silently drop a file path; a section labeled files_modified cannot, because the model has to either fill it or explicitly leave it empty.

Factory’s evaluation on 36,000 production messages gave structured summarization 3.70/5 for retention and accuracy, versus 3.44 for Anthropic’s automatic compaction and 3.35 for OpenAI’s /compact endpoint. It also retained technical details such as file paths, error codes, and line numbers better than prose-only approaches.

When it’s right. Any agent where the critical context is typed and enumerable: coding agents (files, lines, errors, decisions), data-analysis agents (datasets, columns, transformations), customer-support agents (issue, attempted fixes, escalations). The vast majority of production agents land here.

When it’s wrong. Open-ended conversational agents where the schema doesn’t pre-exist. A therapy-style companion or an open-domain Q&A doesn’t have natural section boundaries; forcing a schema produces awkward “Section: Intent: N/A” outputs that waste tokens. Recursive summarization or no-compression-with-aggressive-truncation is the better fit.

Strategy 3: Verbatim compaction (curated subset)

Morph’s verbatim compaction is the productized version: drop low-value tokens but keep high-value ones character-for-character. No rewriting, no paraphrasing, no model-generated text: the surviving content is byte-identical to the original input. The compression ratios are modest (50-70% vs the 80-90% of prose summarization) but the fidelity is perfect on the parts that survive. Reported numbers: 3,300+ tokens/sec and 98% verbatim accuracy.

The trade-off is in the deletion oracle. The pass needs to decide which tokens are low-value, and that decision is itself a model call: typically a smaller model classifying each message as “drop” or “keep.” The pass is fast in aggregate because the per-message decision is cheap, but the model has to be calibrated; a bad oracle deletes the wrong half.

When it’s right. Coding agents (the dominant use case Morph targets), debugging sessions, anywhere the agent will re-reference specific lines from the conversation later. The verbatim-preservation property is what makes the “re-reading loop” go away: the agent doesn’t have to re-discover what was already established.

When it’s wrong. Conversations dominated by long-form natural language where the gist matters more than any individual sentence. A long meeting transcript compressed by verbatim deletion still produces a long, awkward sequence of disconnected verbatim passages; a prose summary of the same transcript is much more useful as a working-set substitute.

Strategy 4: Opaque (model-internal) compression

OpenAI’s /v1/conversations/{id}/compact endpoint and similar opaque-compression interfaces produce a server-side compressed representation that the client never sees in human-readable form. The compression ratios are dramatic: Morph’s reporting cites 99.3% for OpenAI: because the representation isn’t bound by being human-legible. The compressed state is then attached to the next request as an opaque token.

When it’s right. When you don’t need to inspect what was kept and dropped, and you trust the vendor’s compression model. Extremely long contexts where the gain from 99% compression dominates the loss from inspection.

When it’s wrong. Auditable systems, multi-vendor portability, debugging-sensitive workflows. Opaque compression is the vendor-lock-in version of the trade-off: the operator gives up inspection and portability in exchange for raw compression ratio. Most production teams that have evaluated it (Factory’s published score: 3.35/5) end up preferring structured summarization for the inspectability.

API-level compaction

The cleanest reference implementation in 2026 is Anthropic’s compaction strategy. The strategy name is compact_20260112; the beta header is compact-2026-01-12; the trigger defaults to 150,000 input tokens. When the threshold is breached, the API runs the compaction inline, emits a compaction block in the response, and on the next request automatically drops all message blocks prior to the compaction block. The pattern is straightforward log-compaction with a server-managed checkpoint, exposed to the client as a single header and a configuration object. Custom summarization instructions can be passed in instructions to override the default prompt.

The shape worth internalizing: and the reason Anthropic ships this as a beta strategy rather than letting the client implement it: is that the server-managed checkpoint eliminates the client’s risk of getting the boundary wrong. A client-side compaction has to maintain its own “compacted prefix” pointer; if the pointer drifts (or the client retries a request and the retry runs on the un-compacted version), the model sees two copies of the same content or loses the compaction entirely. The server-managed version eliminates that bug class.

When compaction runs

The single biggest cost lever in compression is when it runs.

Reactive (watermark-triggered). The harness counts tokens before each turn; if the count exceeds a watermark, run the compaction pass before the next model call. Pros: only runs when needed, minimal wasted model calls. Cons: the user-facing turn that triggers the watermark pays the compaction latency (typically 1-5 seconds with a small model). This is the default in LangGraph’s SummarizationNode (compresses when max_tokens_before_summary is exceeded) and in Anthropic’s beta.

Preemptive (budget-aware). Before each turn, predict whether the current turn will breach the budget after the model’s reply, and if so, run the compaction before the turn rather than after. Requires a token-count estimate for the model’s output (use a conservative estimate: the model’s max_tokens parameter, or a 95th-percentile output length). Pros: the user-facing turn never gets surprise-latency from compaction. Cons: extra model calls when the prediction is wrong (compaction runs but wasn’t actually needed). Best for high-stakes interactive agents where the unexpected latency is more expensive than the occasional wasted compaction.

Deferred (session-end or background). Compaction runs at session close or in a sleep-time / background pass over recorded sessions. Best when the conversations are bounded and the compressed state is consumed by future sessions, not the current one. The classic use case is multi-session memory: the live session uses the raw buffer; the cross-session memory layer pulls the compacted version. This is the pattern LangMem’s summarize_messages is designed for.

Semantic (topic-shift-triggered). Compress when the topic shifts: when a tool’s output indicates a task transition, when the user introduces a new objective, when the agent itself signals “starting fresh.” The trigger is content-aware rather than budget-aware. Pros: compaction lands at natural boundaries; the compressed summary is a coherent block. Cons: requires a topic-detection signal that itself costs something. Useful as a secondary trigger on top of a watermark: if the watermark fires and a topic shift is detected, compress; if only one fires, hold.

The defensible production answer is watermark as primary, semantic as secondary: maintain a token watermark at 70-80% of the model’s effective context, fire compaction when crossed, and use semantic signals to nudge the boundary to a coherent point. The conversation-compaction article works the harness-level orchestration design: reactive vs preemptive triggers, cache-aware deletion, circuit breakers, snapshot-and-rollback: in more detail; the takeaway here is that compression is a budget-driven operation, and the trigger is the dominant lever.

Implementations

Both implementations realize the same recursive-summarization-with-structured-output pattern: a running summary stored separately from the buffer, an LRU-style tail of recent messages kept verbatim, and a structured-section prompt that forces the summarizer to populate each critical slot explicitly. The patterns scale to richer schemas (Factory’s full set is intent, files_modified, decisions_made, pending_questions, next_steps); a five-section schema is a defensible starting point.

Python: recursive structured-section compaction

Uses the Anthropic SDK for the summarization call. The implementation is hot-path-blocking but the same shape ports to a background queue.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96

# pip install "anthropic>=0.40.0"
import json
from typing import Any
from anthropic import Anthropic

client = Anthropic()
SUMMARIZER_MODEL = "claude-haiku-4-5"  # $1/$5 per million tokens

SUMMARY_PROMPT = """\
You are compacting an agent conversation. Read the prior summary (if any) and the new messages, then emit an updated structured summary with EXACTLY these JSON keys. Every key must be present. Use [] for empty lists, "" for empty strings, never omit a key. Never paraphrase technical identifiers (file paths, error codes, function names): copy them verbatim from the source.

{{
  "session_intent": str,            // one sentence: the user's overall goal
  "files_modified": [str],          // list of file paths touched, in order
  "decisions_made": [str],          // bulleted decisions with rationale
  "pending_questions": [str],       // unresolved questions from the user
  "next_steps": [str]               // what the agent plans to do next
}}

PRIOR SUMMARY:
{prior_summary}

NEW MESSAGES:
{new_messages}

Respond with the JSON object only, no prose.
"""

def render_messages(messages: list[dict]) -> str:
    return "\n\n".join(
        f"[{m['role'].upper()}] {m['content']}" for m in messages
    )

def compact(prior_summary: dict | None, new_messages: list[dict]) -> dict:
    prior_text = json.dumps(prior_summary or {
        "session_intent": "", "files_modified": [],
        "decisions_made": [], "pending_questions": [], "next_steps": [],
    }, indent=2)
    prompt = SUMMARY_PROMPT.format(
        prior_summary=prior_text,
        new_messages=render_messages(new_messages),
    )
    resp = client.messages.create(
        model=SUMMARIZER_MODEL,
        max_tokens=1024,
        messages=[{"role": "user", "content": prompt}],
    )
    text = resp.content[0].text.strip()
    # Strip code-fences if the model wrapped the JSON
    if text.startswith("```"):
        text = text.split("```")[1].lstrip("json\n").rstrip("`\n")
    return json.loads(text)

# ----- The buffer manager that uses it -----
TAIL_KEEP = 8         # last N messages kept verbatim
WATERMARK_TOKENS = 100_000  # compress when this is exceeded

class CompactingBuffer:
    def __init__(self) -> None:
        self.system_prompt: dict | None = None
        self.summary: dict | None = None
        self.tail: list[dict] = []   # uncompacted recent messages

    def append(self, message: dict) -> None:
        self.tail.append(message)
        if self._approx_tokens() > WATERMARK_TOKENS:
            self._compact()

    def _approx_tokens(self) -> int:
        # ~4 chars per token is a defensible rough estimate
        body = json.dumps(self.summary or {}) + render_messages(self.tail)
        return len(body) // 4

    def _compact(self) -> None:
        # Compress everything except the last TAIL_KEEP messages
        if len(self.tail) <= TAIL_KEEP:
            return
        to_compress = self.tail[:-TAIL_KEEP]
        self.summary = compact(self.summary, to_compress)
        self.tail = self.tail[-TAIL_KEEP:]

    def render_for_model(self) -> list[dict]:
        msgs: list[dict] = []
        if self.system_prompt:
            msgs.append(self.system_prompt)
        if self.summary:
            msgs.append({
                "role": "user",
                "content": (
                    "PRIOR SESSION SUMMARY (structured):\n"
                    + json.dumps(self.summary, indent=2)
                    + "\n\nContinue the conversation below."
                ),
            })
        msgs.extend(self.tail)
        return msgs

A few decisions worth noting. The summarizer is a small model (Claude Haiku 4.5 at $1/$5 per million tokens); using the foreground agent’s model for compaction is the most common cost bug in production harnesses. The watermark check is on every append, but the compaction only runs when the buffer actually exceeds the threshold and there are enough tail messages to compress. The summary is rendered into the next prompt as a user message containing structured JSON, not as a system-prompt mutation: keeping the system prompt static preserves the prompt cache prefix, which is the second-most-common cost bug.

Measuring compression-induced quality loss

Compression is the operation that’s most-often shipped untested, and the failure mode is silent. Three diagnostics worth running before any compression strategy goes to production.

The probe test. Before compression, the buffer contains a set of probes: verifiable facts that the next call could plausibly need (file paths, decisions, configuration values, user-stated preferences). After compression, query the resulting buffer (via a small model call) and check whether each probe is recoverable. The probe-recovery rate is the headline metric. Factory’s published evaluation uses probe-based scoring across four dimensions (factual retention, file tracking, task planning, reasoning chains); the framework is general and worth porting to any custom summarizer.

The re-reading-loop diagnostic. After compression, count how many turns it takes the agent to re-discover facts that were in the pre-compression buffer. If the agent calls Read on a file it has already edited, or asks the user for a clarification it has already received, the compression dropped a critical piece. The instrumentation is cheap (compare post-compression tool calls to pre-compression tool calls for the same agent on the same task) and the signal is direct. Morph’s pitch for verbatim compaction is grounded in this metric: a re-reading loop is what you avoid when you preserve byte-for-byte content.

The confabulation diagnostic. Run two versions of the next call: one with the compressed buffer, one with the uncompressed buffer: and compare the responses. Disagreements on the facts (which file, what value, which decision) are confabulation; disagreements on the style are acceptable. The diff-based diagnostic is what catches the failure mode where the agent confidently says the wrong thing because the summary said the wrong thing. Run it offline on logged sessions; the cost is one extra model call per probe session, and the signal is high.

These three together: probe-recovery, re-reading-loop, confabulation diff: are the closest the compression layer comes to having a unit-test discipline. The RAG evaluation article covers the broader testing framework for memory-side retrieval; the same pattern (golden set + probe queries + scored outputs) ports over.

When to use compression

Compression wins when the conversation contains a long mid-section of low-value turns surrounded by high-value endpoints. Coding sessions with extensive tool-call cycles, debugging sessions with many false starts, exploratory analysis sessions: all have the shape “the recent tail and the original task are critical, the middle is mostly retrieved-but-unused context.” A summary of the middle preserves the relevant facts at a fraction of the token cost, and the working set fits comfortably.

Compression wins when the next call’s working set is a small subset of the full history. If the agent only needs to remember “what we decided” and “where we are now,” the full history is overkill. A small structured summary substitutes for the history at one or two orders of magnitude lower token cost.

Truncation beats compression for short sessions. If the conversation is going to end in another 5-10 turns regardless, the cost of a compression pass (a model call, plus the cache invalidation, plus the latency) outweighs the cost of just truncating. Compression is a long-session strategy; in a short session, simple head-tail truncation wins.

Verbatim compaction beats summary compression when verbatim content is critical. Coding agents are the standard example. Most teams running coding agents in production have converged on verbatim or structured compaction for this reason; the prose-summary approach is the simpler-and-prettier loser.

Long-context models reduce the urgency, but don’t eliminate it. Models now ship with 1M-token contexts; the immediate budget pressure for compression is lower. But the cost per call scales linearly with context size, and the lost-in-the-middle effect means an over-stuffed long context is worse than a well-curated short context. Compression remains the right operation even when the hard limit is no longer binding: the soft limit (attention dilution, cost per call, latency) shows up sooner than the hard one.