Context Engineering: JIT vs AOT Context Loading

A large context window permits more input but does not guarantee that the model will use every part reliably. Sending an entire knowledge base can increase cost and reduce answer quality relative to a smaller, better selected working set. Context engineering decides which material is loaded before a call and which material is fetched only when the task requires it.

What context engineering actually is

Context engineering covers instructions, user state, conversation history, retrieved evidence, tool schemas, and tool results. Anthropic’s context-engineering article uses the term for curating and maintaining tokens during inference. Prompt writing is one part of that job; retrieval and compaction are others.

The relevant limit is the model’s effective working set for a task, not only the API’s maximum token count. Irrelevant material adds input cost and competes with evidence the answer needs. The policy should therefore optimize task quality under token, latency, and tool-call budgets.

Mechanics of AOT context assembly

AOT assembly retrieves and packages context before the generation call:

1. Run the retrieval cascade against the user’s query.
2. Select top-ranked chunks using an evaluation set to choose K.
3. Concatenate into a prompt template: instructions, retrieved context, user question.
4. Call the model. Single round-trip.

Ordering is part of retrieval policy. Lost in the Middle found that position affects use of relevant information, so evaluate arrangements rather than assuming rank order alone is enough. Keep source identifiers attached to passages so the model can cite evidence and the evaluator can check it.

Use structured fields for typed application state such as plan, locale, and account permissions. Keep source prose as prose when wording and citations matter. Structure does not make a fact true; it makes field boundaries explicit. Structured output applies the corresponding discipline to responses.

Mechanics of JIT context retrieval

JIT retrieval gives the model tools and lets the agent loop request evidence during execution:

1. The model receives a thin context: instructions, available tool schemas, the user’s question, no corpus material.
2. The model emits a tool call: search_docs(query="dashboard refresh interval").
3. The harness executes the tool, returns the result into the conversation.
4. The model decides whether it has enough to answer, or emits another tool call.
5. Loop until the model produces a final answer or a step limit is hit.

AOT pays for selected chunks in one model call. JIT may load fewer passages but pays for additional model turns, tool latency, and repeated prefixes. The break-even point depends on task variability, average retrieval hops, cache reuse, and latency limits. Single-turn question answering often favors AOT. Exploratory tasks benefit from JIT because each result can change the next query.

JIT also needs stop conditions. Cap tool calls, detect repeated queries, and require the model to explain what missing evidence another search should obtain. Without those controls, the loop can spend tokens retrieving material it never uses.

Code: AOT in Python with the Anthropic SDK

Install the Anthropic SDK with pip install anthropic. The example assembles typed facts and ranked source passages before one model call.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43

import json
import anthropic

client = anthropic.Anthropic()

SYSTEM = """You are a support assistant. Answer ONLY from the
<context> block. If the answer is not in the context, say so."""

def aot_answer(question: str, chunks: list[dict], facts: dict) -> str:
    # Alternate high-ranked passages between the start and end.
    front, back = [], []
    for index, chunk in enumerate(chunks):
        (front if index % 2 == 0 else back).append(chunk)
    ordered = front + list(reversed(back))

    seen, deduped = set(), []
    for c in ordered:
        if c["id"] not in seen:
            seen.add(c["id"])
            deduped.append(c)

    context_block = "\n\n".join(
        f"<doc id={c['id']} source={c['source']!r}>{c['text']}</doc>"
        for c in deduped
    )

    # Structured facts go as JSON, not prose: deterministic key-based recall.
    user_block = (
        f"<facts>{json.dumps(facts, ensure_ascii=False)}</facts>\n"
        f"<context>\n{context_block}\n</context>\n"
        f"<question>{question}</question>"
    )

    msg = client.messages.create(
        model="claude-sonnet-4-6",
        max_tokens=1024,
        system=[
            {"type": "text", "text": SYSTEM,
             "cache_control": {"type": "ephemeral"}},
        ],
        messages=[{"role": "user", "content": user_block}],
    )
    return msg.content[0].text

The system instructions use cache_control; prompt caching covers its usage counters and cost model. The facts are JSON-serialized rather than interpolated with Python’s representation. In production, escape or delimit untrusted document text and enforce access control before assembly.

Combining AOT and JIT

A hybrid policy loads stable instructions, task schema, essential user or workspace state, and a small set of broadly relevant evidence before the call. Retrieval tools handle facts that vary by request or depend on earlier observations. This keeps the reusable prefix stable while allowing the loop to investigate an unexpected branch.

Define the boundary from traces. A field belongs in AOT context when most eligible requests use it and its value remains stable for a useful cache interval. A corpus belongs behind JIT retrieval when only a small fraction is relevant to any request. Sensitive data must also pass authorization before either route exposes it to the model.

Operational risks

Long windows can hide poor selection. NoLiMa, Context Rot, and Lost in the Middle examine different forms of degradation as context grows. Test the actual model and task at several context sizes; do not use the advertised maximum as a target.

AOT fails when weak passages dilute or contradict strong evidence. Measure answer quality as K changes, preserve retrieval scores and source IDs, and rerank before assembly. JIT fails when the model searches repeatedly without gaining evidence. Measure useful tool results per call, repeated-query rate, total hops, and end-to-end latency.

Stable AOT prefixes can benefit from prompt caching. Put volatile retrieved text and the current message after stable instructions where the API permits. Cache layout remains subordinate to semantic correctness and privacy boundaries.

Conversation history is another context source. Tool results and assistant messages can eventually dominate the working set. Apply truncation or conversation compaction before the hard limit, and preserve exact facts that summaries are likely to damage.

Run the RAG evaluation harness after changing K, chunk order, fact schema, tool descriptions, or JIT step limits. Report task success, citation support, input tokens, tool calls, TTFT, and total latency. The best policy is workload-specific and may route different query classes differently.

Choosing a route

Route with observable task properties before adding an LLM classifier. A request for a known account field can use structured AOT state. A question over a bounded document set can use one AOT retrieval pass. Debugging, research, and other tasks whose next question depends on the previous result are candidates for JIT tools.

A simple policy can inspect query type, corpus size, required freshness, and whether the task permits multiple round trips. Keep a fallback: an AOT request that lacks evidence may expose a search tool, while a JIT loop approaching its step limit can assemble the best evidence found so far and answer with uncertainty.

If a learned router is needed, label routes by measured outcome rather than intuition. For each training query, run both policies under the same model and budget, then label the lower-cost policy among those that pass the quality threshold. Include “abstain” when neither route has enough evidence.

Routing errors have asymmetric costs. Sending an exploratory task to AOT may produce an unsupported answer. Sending a simple lookup to JIT usually wastes latency. Set the router’s threshold according to the more serious failure for the product, and monitor manual overrides or fallbacks by route.

Context provenance and security

Every inserted passage should retain a source ID, retrieval score, timestamp where relevant, and access scope. The generation layer can cite source IDs, while the evaluator can test whether each claim is supported. Without provenance, a compact context may be cheap but difficult to audit.

Authorization must happen before retrieval results enter the model context. Filtering after generation cannot undo disclosure to the model or provider. In multi-tenant systems, enforce tenant and document permissions inside the retrieval query and verify them again during assembly.

Treat retrieved text and tool results as untrusted data. Delimit them from system instructions and tell the model which fields contain evidence. Prompt injection remains possible, so restrict tool authority independently of model instructions. JIT is especially sensitive because an injected document can influence the next tool call.

Freshness belongs in the context policy. Stable account facts may be loaded AOT from a versioned profile, while operational status should be fetched JIT from its source. Attaching timestamps without a validity rule only tells the model that evidence is old; it does not decide whether old evidence may be used.

Evaluation design

Build a test set with direct lookups, multi-hop questions, irrelevant near-neighbours, conflicting sources, missing answers, and tasks that require a second search. For each case, record the evidence that should be sufficient and any source that must not be exposed.

Measure retrieval recall before generation, citation precision after generation, and task success at the end. Add input tokens, cache-read tokens, tool calls, and latency to expose the quality-cost curve. A policy that gains one accuracy point with five extra model turns may be wrong for an interactive endpoint and acceptable for offline research.

Run ablations that remove AOT fields, reduce K, disable JIT, and change document order. Ablations reveal which context the model actually uses. They also find decorative fields that consume tokens without affecting outcomes. Re-run the suite after model changes because context-size and position sensitivity can differ across models.

Budgeting the working set

Reserve output capacity before allocating input. From the remaining window, subtract the system and tool schemas, then set separate budgets for stable facts, retrieved passages, conversation history, and tool results. The budgets can be token counts or percentages, but they need hard overflow behavior.

When AOT retrieval exceeds its budget, prefer reranking and deduplication to blind truncation. If a single source is too large, select passages while retaining document metadata. When JIT results exceed the remaining budget, summarize or page them inside the tool layer and tell the model that more results are available.

Log budget use by category on every call. This reveals whether conversation history, tool schemas, or retrieved evidence is consuming the window. It also makes regressions attributable: a deployment that adds ten verbose tool descriptions may reduce retrieval capacity even though the retrieval code did not change.

Further reading

• Anthropic: Effective context engineering for AI agents
• Chroma Research: Context Rot and its companion repository
• NoLiMa: Long-Context Evaluation Beyond Literal Matching
• Lost in the Middle