$ cat ai-engineering/conversation-compaction.md

Conversation Compaction: Keeping Long Sessions Alive

How long agent sessions compact context using preemptive triggers, cache-aware deletion, rollback, and journals.

Jatin Bansal@blog:~/ai-engineering$ open conversation-compaction

A large tool result can push an agent beyond its context limit before a reactive compactor runs. If the summarizer has a smaller window, it may reject the same buffer and leave the session unable to continue. A reliable harness compacts with headroom, commits changes atomically, and keeps a degraded truncation path for summarizer failure.

When compaction runs

Conversation compaction is the harness operation that, when triggered, reduces the size of the live conversation buffer in place so that the next model call fits. Three properties distinguish conversation compaction in a long session from the broader compression operation. It is in-place mutation of the live buffer, not an offline pass; the next call must succeed against the new buffer. It is non-optional once triggered: there is no path forward if it fails. And it is cache-disruptive by default: any rewrite invalidates the prompt-cache prefix from the rewrite boundary onward, so a naive implementation pays full prefill on every subsequent call. The orchestration question is how to fire it as rarely and cheaply as possible, and how to keep the session alive when it fails.

Reactive vs preemptive triggers

Two philosophies, and most production harnesses end up running both.

Reactive compaction fires after the buffer crosses a watermark. Anthropic’s compaction beta is the cleanest example; the server detects when input tokens exceed the configured trigger (default 150,000), runs compaction inline, emits a compaction block, and continues. OpenAI’s server-side compaction via the Responses API (shipped February 11, 2026) is conceptually identical: pass context_management.compact_threshold and the server fires compaction when the rendered token count crosses it. Claude Code’s auto-compact at ~95% is the client-side version. The pros are operational simplicity: a single number, a single conditional. The cons: the user-facing turn that crosses the watermark pays 1-5s of compaction latency, and there’s no head-room; if the model emits a 50K-token tool result at 90K of a 100K budget, the compaction must succeed on the first try because there’s no room for retry tokens.

Preemptive compaction fires before the buffer is dangerous, by projecting whether the next turn will breach the budget after the model’s reply. The estimate is the current buffer size plus a conservative max_tokens bound on output plus the expected tool-result size from the most recent tool_use block. The pros: the user-facing turn never gets surprise latency, and there is always head-room for the model to reply. The cons: some compactions are wasted (the prediction was conservative), and the trigger requires a reasonably accurate token estimator plus policy about max output size, which often lives outside the core compaction module.

The defensible production pattern is preemptive as primary, reactive as fallback. Run preemptive at ~70% of effective context; keep reactive as a backstop at ~95% for cases where the estimator misses a heavy-tailed tool result. This mirrors generational garbage collection: frequent cheap cleanup with a less frequent full fallback.

Cache-aware compaction: surgical deletion

Compaction should preserve stable cached prefixes. Rewriting the whole history invalidates prompt-cache reuse after the first changed token. Prefer removing superseded tool results or replacing an old range with an immutable summary block.

Three patterns ship in production:

Surgical tool-result deletion, system prompt and assistant turns preserved. The cheapest operation: identify tool-call/tool-result pairs where the result has since been superseded (e.g., five Read calls on the same file; keep the latest, replace the earlier ones with [result superseded; see turn N]). The system prompt, assistant reasoning, and the recent tail are unchanged; the cacheable prefix grows monotonically. This is what Anthropic’s clear_tool_uses_20250919 does at the API level and what Claude Code’s micro-compact does client-side. Compression ratio is modest (10-30%), but the cache hit rate stays high, and the operation can run frequently.

Append-only summary block, untouched tail. When micro-compaction isn’t enough: write a new “session summary” block at a stable position (right after the system prompt), and drop the messages it summarizes. The summary, once written, is treated as immutable for the next K turns; it doesn’t get re-summarized on every turn; the cache treats it as a stable prefix. The conversation tail after the summary keeps growing and benefits from incremental caching. The Anthropic beta’s pause_after_compaction option exists for this pattern: pause after the summary is generated, let the client preserve any instruction-oriented messages, then continue. Compression ratio is high (80-95%); cache invalidation cost is paid once per K turns rather than every turn.

Anti-pattern: summary-as-system-prompt mutation. Some harnesses fold the running summary into the system prompt, mutating it every cycle. This is the worst thing you can do for the cache; the system prompt is the most-cached prefix, and rewriting it every turn recomputes the entire prefix on every turn. The cost graph reads “we turned on caching” but the bill stays flat. The harness anatomy article flagged this; it bears repeating because every team seems to invent it independently.

Rule of thumb: compress as far from the cache-hit prefix as possible; rewrite as little of the prefix as possible; treat the summary block as an immutable record between compactions, not a running state.

Error recovery: circuit breakers and snapshot-rollback

Compaction can fail because the summarizer is unavailable, its output violates the schema, required fields are empty, its own context limit is exceeded, or the compacted buffer remains too large. Each case needs a typed recovery path.

Take a snapshot before mutating the live buffer. Run the summarizer against that snapshot, validate the schema and required fields, then swap the compacted buffer atomically. On failure, retain the original. Long-horizon checkpointing applies the same commit discipline at a larger scope.

Circuit breaker around compaction failures. Maintain a per-session counter of consecutive failures. Trip at N=3 (a defensible default; lower if throughput-bounded, higher if the summarizer is flaky). When tripped, do not retry compaction; fall back to a degraded strategy and emit a structured failure. The breaker’s purpose is to break the infinite-failure loop: a session that fails once is recoverable; a session that fails repeatedly while burning the full timeout window each time is a service outage. After M turns of cool-down, reset.

Lossy truncation as the last-resort fallback. When the breaker is tripped or the summarizer is unreachable, fall back to unconditional head-tail truncation: keep the system prompt, keep the most recent K turns, drop everything in the middle. This is the default eviction policy from short-term memory, used here as the fallback when the smarter policy fails. The agent loses semantic context but the session stays alive; and a degraded session is recoverable; a crashed one is not. The fallback should be loud: log the event, expose a UI indicator, mark the failure for downstream review.

The wedged-buffer escape hatch. For the case where the summarizer itself can’t fit the buffer in its context: either keep a secondary summarizer with a larger context window, or run a chunked-and-merge pass that splits the buffer in half, summarizes each half independently, then summarizes the summaries. The chunked-and-merge pattern is the same shape as map-reduce summarization and is the right recovery primitive when the foreground model’s context exceeds the summarizer’s.

Append-only memory journals: the architectural alternative

A radically different design ports the database log-vs-LSM-tree decision: instead of compacting the live buffer, run the live buffer at near-zero retention and write everything important to an external append-only memory journal, queried at retrieval time rather than loaded wholesale.

Mechanically: every turn, before the model call, the harness extracts decisions, files, errors, or load-bearing facts and appends them to a per-session journal (JSONL file, Postgres table, vector store; substrate-agnostic). The live buffer stays short; last 10-20 turns; by aggressive truncation. When the model needs earlier context, it queries the journal via a recall tool the harness exposes. The journal grows linearly; the live buffer is bounded by design.

The journal model replaces the orchestration-failure surface with a retrieval-quality surface: no compaction to fail, but a recall query whose quality determines whether the agent finds the relevant fact. It makes the trade-off explicit: the operator can inspect, query, and audit the journal; versus a summary block whose contents are at the summarizer’s discretion. And it plays well with sleep-time compute: the journal is exactly the artifact an offline consolidation pass needs.

The pattern shows up under several names in 2026 production systems. Doug Turnbull’s “give your coding agent a journal” is the cleanest articulation; an agent maintains a journal file in the working directory, one entry per significant action, queried when it needs to remember. OpenCode’s append-only journal blocks productize the idea with semantic search. LangGraph’s checkpoint-and-store split implements the architectural distinction at the framework level. The unifying claim: aggressive truncation plus a queryable external journal beats summarization-in-place for any task where audit and reproducibility matter more than the gist surviving in prose.

The journal wins for coding agents (audit trail needs to survive verbatim), compliance workflows (regulators want exact actions, not paraphrases), long-horizon research (the journal is the research log), and multi-session agents (journals are naturally cross-session). Compaction still wins for open-ended conversational agents where the gist is what matters, latency-sensitive interactive agents where the journal round-trip dominates, and single-session ephemeral workflows. The patterns are complementary; the production answer for most agents is both; a journal for durable state, compaction for the live buffer’s working set.

A preemptive, cache-aware compactor in Python

Realizes the full orchestration shape: preemptive triggering, snapshot-and-rollback, circuit-breaker recovery, cache-aware mutation. Uses the Anthropic SDK.

python

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
# pip install "anthropic>=0.89.0"
import copy
import json
import logging
import time
from dataclasses import dataclass, field
from typing import Any
from anthropic import Anthropic

log = logging.getLogger("compactor")
client = Anthropic()

SUMMARIZER_MODEL = "claude-haiku-4-5"
SUMMARY_PROMPT = """You are compacting an agent conversation. Read the messages and emit a JSON object with these keys exactly. Never paraphrase file paths, error codes, function names, or version numbers — copy them verbatim.

{"session_intent": "", "files_touched": [], "decisions": [], "pending_questions": [], "next_steps": []}

MESSAGES:
{messages}

Respond with the JSON object only.
"""

@dataclass
class CompactionConfig:
    soft_watermark: float = 0.70   # preemptive trigger
    hard_watermark: float = 0.95   # reactive fallback
    tail_keep: int = 10            # messages preserved verbatim after compaction
    max_failures: int = 3          # circuit-breaker trip threshold
    cooldown_turns: int = 5        # cool-down before breaker resets
    summarizer_max_tokens: int = 1024
    output_headroom_tokens: int = 4096  # reserved for the foreground reply

@dataclass
class CompactorState:
    consecutive_failures: int = 0
    breaker_tripped_at_turn: int | None = None
    last_summary: dict | None = None
    last_summary_at_turn: int = 0
    turn: int = 0

def estimate_tokens(messages: list[dict]) -> int:
    # ~4 chars per token; replace with the provider's tokenizer in production.
    return sum(len(json.dumps(m)) for m in messages) // 4

def project_next_turn(
    messages: list[dict],
    pending_tool_result_estimate: int,
    cfg: CompactionConfig,
) -> int:
    return (
        estimate_tokens(messages)
        + pending_tool_result_estimate
        + cfg.output_headroom_tokens
    )

class CompactorError(Exception): ...
class WedgedBufferError(CompactorError): ...
class BreakerTrippedError(CompactorError): ...

def _run_summarizer(messages: list[dict]) -> dict:
    # Validate output shape; raise on any malformed result so the caller can
    # snapshot-rollback rather than commit a broken summary.
    body = json.dumps(messages, ensure_ascii=False)
    resp = client.messages.create(
        model=SUMMARIZER_MODEL,
        max_tokens=1024,
        messages=[{"role": "user", "content": SUMMARY_PROMPT.format(messages=body)}],
    )
    text = resp.content[0].text.strip()
    if text.startswith("```"):
        text = text.split("```", 2)[1].lstrip("json\n").rstrip("`\n")
    obj = json.loads(text)  # malformed JSON raises here
    required = {"session_intent", "files_touched", "decisions",
                "pending_questions", "next_steps"}
    if not required.issubset(obj.keys()):
        raise CompactorError(f"summarizer missing keys: {required - obj.keys()}")
    # Empty-load-bearing-field check: a summary with no decisions and no files
    # touched after a 50-turn coding session is almost certainly a silent failure.
    if not (obj["decisions"] or obj["files_touched"]) and len(messages) > 20:
        raise CompactorError("summarizer returned empty load-bearing fields")
    return obj

def _chunked_and_merge(messages: list[dict]) -> dict:
    # Wedged-buffer escape hatch: split, summarize halves, summarize summaries.
    mid = len(messages) // 2
    left = _run_summarizer(messages[:mid])
    right = _run_summarizer(messages[mid:])
    merged_input = [
        {"role": "user", "content": "PARTIAL SUMMARY (first half):\n" + json.dumps(left)},
        {"role": "user", "content": "PARTIAL SUMMARY (second half):\n" + json.dumps(right)},
    ]
    return _run_summarizer(merged_input)

def _lossy_truncate(messages: list[dict], tail_keep: int) -> list[dict]:
    # Last-resort fallback when the circuit breaker is tripped or summarizer fails.
    system = [m for m in messages if m.get("role") == "system"]
    tail = [m for m in messages if m.get("role") != "system"][-tail_keep:]
    return system + [{"role": "user", "content": "[earlier history truncated; "
                                                 "compactor unavailable]"}] + tail

class Compactor:
    def __init__(self, cfg: CompactionConfig | None = None,
                 model_context_limit: int = 200_000):
        self.cfg = cfg or CompactionConfig()
        self.state = CompactorState()
        self.limit = model_context_limit

    def maybe_compact(
        self,
        messages: list[dict],
        pending_tool_result_estimate: int = 0,
    ) -> list[dict]:
        self.state.turn += 1
        projected = project_next_turn(messages, pending_tool_result_estimate, self.cfg)

        # Breaker cool-down: reset after the configured number of turns.
        if (self.state.breaker_tripped_at_turn is not None and
                self.state.turn - self.state.breaker_tripped_at_turn
                >= self.cfg.cooldown_turns):
            log.info("compactor.breaker_reset turn=%d", self.state.turn)
            self.state.breaker_tripped_at_turn = None
            self.state.consecutive_failures = 0

        if projected < self.limit * self.cfg.soft_watermark:
            return messages  # no compaction needed

        # If the breaker is tripped, go straight to lossy truncation.
        if self.state.breaker_tripped_at_turn is not None:
            log.warning("compactor.breaker_open fallback=truncate turn=%d",
                        self.state.turn)
            return _lossy_truncate(messages, self.cfg.tail_keep)

        # Snapshot the live buffer before any mutation.
        snapshot = copy.deepcopy(messages)
        try:
            if estimate_tokens(messages) > self.limit:  # wedged-buffer escape
                summary = _chunked_and_merge(messages[:-self.cfg.tail_keep])
            else:
                summary = _run_summarizer(messages[:-self.cfg.tail_keep])
        except Exception as e:
            self.state.consecutive_failures += 1
            log.error("compactor.failure n=%d err=%s",
                      self.state.consecutive_failures, e)
            if self.state.consecutive_failures >= self.cfg.max_failures:
                self.state.breaker_tripped_at_turn = self.state.turn
                log.error("compactor.breaker_tripped turn=%d", self.state.turn)
            # Atomic rollback: original buffer is unchanged.
            return _lossy_truncate(snapshot, self.cfg.tail_keep)

        # Success: build the new buffer atomically.
        self.state.consecutive_failures = 0
        self.state.last_summary = summary
        self.state.last_summary_at_turn = self.state.turn
        system = [m for m in messages if m.get("role") == "system"]
        tail = messages[-self.cfg.tail_keep:]
        summary_block = {
            "role": "user",
            "content": "PRIOR SESSION SUMMARY (immutable until next compaction):\n"
                       + json.dumps(summary, indent=2),
        }
        return system + [summary_block] + tail

The interesting parts aren’t the summarizer call; that’s the context-compression article’s domain; but the orchestration. The snapshot-and-rollback cannot leave the buffer half-rewritten. The breaker prevents the infinite-failure loop. The lossy-truncation fallback keeps the session moving. The chunked-and-merge escape hatch handles the wedged case. The preemptive trigger gates on projected, not on current buffer size, so head-room is reserved before the call.