$ cat ai-engineering/long-horizon-reliability.md

Long-Horizon Task Reliability

How checkpointing, abort policies, and recovery limit drift in long-running agents.

Jatin Bansal@blog:~/ai-engineering$ open long-horizon-reliability

Long-running agents accumulate state, assumptions, and side effects. A coding agent can complete hundreds of valid edits, form one bad hypothesis, then damage shared infrastructure while trying to recover. Model capability did not change during the run; the loop crossed into a less reliable operating regime.

Separate capability from reliability

The cleanest framing is the reliability-vs-capability split from the late-2026 “Beyond pass@1” framework: capability is whether the model can do a task on its best attempt (pass@1, the standard benchmark axis); reliability is whether it can do the task consistently across attempts of varying duration. A model that scores 90% on a 5-minute task and 44% on a 4-hour version isn’t a worse model; it’s the same model in a different failure regime.

Two headline findings. Frontier models exhibit the highest meltdown rates (DeepSeek V3 at 19%, MiniMax M2.5 at 13% at very-long horizons) precisely because they attempt ambitious multi-step strategies; weaker models hover around 0-4% because they fall back on rote, shallow strategies that don’t have anywhere to drift to. And the failure shape is positive error correlation across steps: once the agent forms a wrong hypothesis it persists rather than recovers, and the reliability curve decays super-linearly relative to an i.i.d. Bernoulli baseline. Capability is the model’s job; reliability is the harness’s job.

Interpret time-horizon benchmarks carefully

The empirical anchor is METR’s time-horizon benchmark, which measures the task duration (calibrated against human completion time) that an agent can do with 50% reliability. As of METR Time Horizon 1.1 in January 2026, Claude Opus 4.5 sits at 320 minutes (5.3 hours), GPT-5 at 214, o3 at 121, Claude Opus 4 at 101. The trend since 2024 is a doubling every 89-131 days, with later updates putting Opus 4.6 at roughly 14.5 hours by February 2026. The “Moore’s law for agents” framing is real and the curve is steep.

The published 50% horizon is an inflection point, not a safe operating target. Horizons at 80% or 95% reliability are much shorter, and failures beyond the measured range can be abrupt rather than gradual.

How long runs drift

Four mechanisms drive the super-linear decay, and they compound. The “Beyond pass@1” framework and the Wire blog’s anatomy of agent drift converge on this list:

Context drift. History grows; older turns become decorations the model attends to less; critical details from turn 10 are diluted by 200 turns of subsequent noise. The lost-in-the-middle failure from the context-engineering article at a longer time scale. Mitigation: aggressive compaction and JIT-only context fetches.
Hallucination cascades. A wrong fact at turn 15 gets cited at turn 30, becomes part of the agent’s “known” state, and by turn 100 the agent is reasoning from a corrupted premise it can’t distinguish from primary observation. Mitigation: explicit provenance tracking: every working-state claim carries a pointer to its source turn, the same shape as the memory-provenance article describes.
Goal drift. “Migrate 400 stored procedures” gets gradually re-interpreted as “improve the test harness” or “refactor the migration script”: adjacent activities that look like progress but aren’t the task. Mitigation: an explicit goal artifact (a typed object, not a paragraph) re-injected at every step.
Meltdown. The pathological end state: the agent transitions from “coherent but wrong” to “incoherent looping, self-contradiction, hallucinated tool outputs.” MOP (meltdown-of-progress) precursors are entropy spikes, repeated tool calls with slightly varied args, contradictions across consecutive turns. Mitigation: detect early, save state and restart with a fresh context window; not just compressing, but saying “this run is poisoned, resume from a checkpoint that wasn’t.”

Context drift makes cascades easier to seed; cascades make goal drift undetectable from inside the loop; goal drift sets up meltdown when the wrong action contradicts the world hard enough.

Use durable checkpoints

Once you accept the run will fail somewhere past the 50% horizon, the engineering question is how cheaply you can resume. Production frameworks disagree on what “checkpoint” means.

Step-level (between-node) checkpointing. LangGraph’s persistence layer is the standard example: a checkpointer saves graph state between nodes. On crash, the next run reads the most recent checkpoint and resumes from the next node. Cheap, human-debuggable, the right primitive for most agents.

The gotcha is what it doesn’t checkpoint. State inside a node is not saved. If one node is doing 200 iterations of work and crashes at iteration 47, the next run restarts that node at iteration 0. Diagrid’s critique of checkpoint-based frameworks is the pointed take. The cost shows up as wasted tokens; 47 API calls re-issued, 47 mutating writes re-attempted (why the idempotency keys above are mandatory), and as latency on recovery.

Durable-execution checkpointing. Temporal, Restate, and similar frameworks invert the model. The engine records every I/O operation into a durable event log. On crash, the workflow replays from the start, but every recorded I/O returns its previously-cached result instead of re-issuing. The agent never knows it crashed. Trade-off: more infrastructure and a constrained programming model (every side effect wrapped as an “activity”) in exchange for dramatically stronger recovery semantics.

The 2026 production hybrid: LangGraph for control flow with Postgres checkpointing, Temporal (or equivalent) wrapping the outer harness when nodes can take minutes or contain many side effects. The AWS DynamoDB write-up and the LangGraph-vs-Temporal architecture pieces make this case directly. A third primitive worth flagging: event-sourcing the conversation itself; persist every turn with a monotonic sequence number and you can rebuild state at any point by replaying turns up to that number, the continuation-passing-style frame from the agent-loop article made durable.

Code: a checkpointed agent loop in Python with LangGraph

A migration-assistant pattern with explicit drift detection and recovery. The loop checkpoints to SQLite between steps, tracks an explicit goal, and aborts on meltdown precursors. Install: pip install langgraph langgraph-checkpoint-sqlite anthropic. Uses LangGraph and the Anthropic SDK.

python

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
import json, hashlib
from typing import TypedDict, Annotated
from operator import add
from anthropic import Anthropic
from langgraph.graph import StateGraph, START, END
from langgraph.checkpoint.sqlite import SqliteSaver

client = Anthropic()

class AgentState(TypedDict):
    goal: str
    messages: Annotated[list, add]
    step: int
    tool_sigs: Annotated[list, add]
    drift_score: float
    completed_ids: list[str]
    aborted: bool

MELTDOWN_THRESHOLD = 0.7
MAX_STEPS = 500

TOOLS = [
    {"name": "migrate_proc", "description": "Migrate one stored procedure. Idempotent.",
     "input_schema": {"type": "object", "properties": {"path": {"type": "string"}},
                      "required": ["path"]}},
    {"name": "run_tests", "description": "Run the test suite. Read-only.",
     "input_schema": {"type": "object", "properties": {"path": {"type": "string"}},
                      "required": ["path"]}},
    {"name": "submit_final", "description": "Submit final results. Terminates.",
     "input_schema": {"type": "object",
                      "properties": {"summary": {"type": "string"},
                                     "completed": {"type": "integer"}},
                      "required": ["summary", "completed"]}},
]


def call_model(state: AgentState) -> dict:
    # Re-inject the goal at the top of every call. Cheapest mitigation
    # for goal drift. The model sees the original task on every turn,
    # not just at turn 0.
    sys = (
        f"GOAL (immutable): {state['goal']}\n\n"
        f"Step {state['step']} of at most {MAX_STEPS}. Before each tool call, "
        f"confirm it serves the goal. If finished, call submit_final."
    )
    resp = client.messages.create(
        model="claude-opus-4-7", max_tokens=2048,
        system=sys, tools=TOOLS, messages=state["messages"],
    )
    return {"messages": [{"role": "assistant", "content": resp.content}]}


def dispatch_tools(state: AgentState) -> dict:
    last = state["messages"][-1]["content"]
    tool_blocks = [b for b in last if b.type == "tool_use"]
    if not tool_blocks:
        return {"step": state["step"] + 1}

    results, sigs, drift_delta = [], [], 0.0
    new_completed = list(state["completed_ids"])
    for b in tool_blocks:
        sig = hashlib.sha1(
            f"{b.name}:{json.dumps(b.input, sort_keys=True)}".encode()
        ).hexdigest()
        sigs.append(sig)
        # No-progress detection: same call in last 5 → drift signal.
        if sig in state["tool_sigs"][-5:]:
            drift_delta += 0.3

        if b.name == "submit_final":
            return {"messages": [{"role": "user", "content": [
                {"type": "tool_result", "tool_use_id": b.id, "content": "ok"}]}],
                "step": state["step"] + 1}

        # Idempotency receipt for mutating tools — refuse re-execution.
        if b.name == "migrate_proc" and sig in new_completed:
            results.append({"type": "tool_result", "tool_use_id": b.id,
                            "content": "skipped: already completed"})
            continue

        out = execute_tool(b.name, b.input)   # real dispatch here
        results.append({"type": "tool_result", "tool_use_id": b.id,
                        "content": json.dumps(out),
                        "is_error": not out.get("ok", True)})
        if b.name == "migrate_proc" and out.get("ok"):
            new_completed.append(sig)
        if not out.get("ok", True):
            drift_delta += 0.1

    return {"messages": [{"role": "user", "content": results}],
            "tool_sigs": sigs,
            "drift_score": state["drift_score"] + drift_delta,
            "completed_ids": new_completed,
            "step": state["step"] + 1}


def execute_tool(name: str, args: dict) -> dict:
    if name == "migrate_proc": return {"ok": True, "path": args["path"]}
    if name == "run_tests":    return {"ok": True, "passed": True}
    return {"ok": False, "error": f"unknown tool: {name}"}


def should_continue(state: AgentState) -> str:
    if state["step"] >= MAX_STEPS:
        return END
    if state["drift_score"] >= MELTDOWN_THRESHOLD:
        # Meltdown precursor crossed. Bail; caller restarts from the
        # last good checkpoint with a fresh, compacted context.
        return END
    return "call_model"


def build_graph():
    g = StateGraph(AgentState)
    g.add_node("call_model", call_model)
    g.add_node("dispatch", dispatch_tools)
    g.add_edge(START, "call_model")
    g.add_edge("call_model", "dispatch")
    g.add_conditional_edges("dispatch", should_continue,
                            {"call_model": "call_model", END: END})
    return g.compile(checkpointer=SqliteSaver.from_conn_string("agent.sqlite"))


def run(goal: str, *, thread_id: str):
    graph = build_graph()
    initial = {"goal": goal,
               "messages": [{"role": "user", "content": goal}],
               "step": 0, "tool_sigs": [], "drift_score": 0.0,
               "completed_ids": [], "aborted": False}
    # thread_id binds this run to a checkpoint stream. Re-invoking with
    # the same thread_id after a crash resumes from the last checkpoint.
    return graph.invoke(initial, config={"configurable": {"thread_id": thread_id}})

The system prompt re-injects the goal on every turn so it cannot disappear under a long history. When the drift score crosses the threshold, the graph routes to END; the caller then restarts from the last checkpoint with a compacted prefix, following the MOP-triggered context-resetting pattern. The dispatch layer also makes repeated side effects idempotent: a second migrate_proc call returns skipped instead of executing again. thread_id binds each invocation to its checkpoint stream. A crash inside dispatch_tools can still lose work, so nodes with many side effects need a durable workflow engine such as Temporal beneath LangGraph.

Abort when retries cannot help

The wrong instinct on a long-horizon failure is “retry from the failure point.” The failure is rarely the failure; it’s the symptom of accumulated drift, and retrying from the same drift-poisoned context produces the same shape of failure with new noise. The right instinct is more often abort, salvage, restart.

A checklist for the abort decision:

Drift score over threshold. Abort, save state, restart with a compacted context.
Goal drift detected. A second model (or a deterministic check) audits whether recent actions still serve the goal. If the audit fires, abort and re-plan from the original goal.
Compensable side effects accumulating. If you’ve shipped 312 PRs and the next 88 start failing in correlated ways, stop. Salvage the 312.
Non-compensable side effects imminent. Email send, payment capture, irrevocable account changes. If drift is climbing and the next predicted action is non-compensable, abort before the action. Better to lose 5 minutes of work than send the wrong email to 200 customers.
Wall-clock or budget breach. The agent-loop budgets still apply: they just fire later.

The aborted-with-partial-results outcome is a first-class success state, not a failure. A migration agent that finished 312 of 400 procedures and stopped cleanly is a win; one that finished 400 of 400 with a broken test harness is a loss. The harness must expose partial state cleanly; completion list, abort reason, recovery hints; so the next run or the human operator can pick up without re-doing work.