<?xml version="1.0" encoding="utf-8" standalone="yes"?><rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:dc="http://purl.org/dc/elements/1.1/"><channel><title>Jatin Bansal</title><link>https://jatinbansal.com/</link><description>Engineering Lead at Pluang. Notes on distributed systems, fintech infrastructure, Java, Kafka, and event-driven backends — drawn from production work.</description><language>en-US</language><generator>Hugo</generator><lastBuildDate>Sat, 10 Oct 2026 21:36:42 +0530</lastBuildDate><atom:link href="https://jatinbansal.com/index.xml" rel="self" type="application/rss+xml"/><item><title>Quantization and Distillation: Compression for Inference</title><link>https://jatinbansal.com/ai-engineering/quantization-distillation/</link><guid isPermaLink="true">https://jatinbansal.com/ai-engineering/quantization-distillation/</guid><pubDate>Tue, 26 May 2026 19:08:40 +0000</pubDate><dc:creator>Jatin Bansal</dc:creator><category>AI Engineering</category><description>Quantization and distillation methods for reducing inference memory, latency, and serving cost.</description><content:encoded>&lt;p&gt;A 70B-parameter model uses about 140 GB for 16-bit weights before activation memory. INT4 weights use about 35 GB. Quantization reduces numeric precision, while distillation trains a smaller student to reproduce a larger teacher&amp;rsquo;s behavior. Both trade some quality for lower serving cost, so deployment requires workload-specific evaluation.&lt;/p&gt;&#10;&lt;h2 id="quantization-and-distillation"&gt;Quantization and distillation&lt;/h2&gt;&#10;&lt;p&gt;Quantization is the process of representing a tensor with fewer bits per element. A 16-bit weight occupies 2 bytes; an INT8 weight occupies 1 byte; an INT4 weight occupies half a byte. The mapping from the original floating-point value to the quantized integer is a function &lt;code&gt;q = round((x - z) / s)&lt;/code&gt;, where &lt;code&gt;s&lt;/code&gt; is the &lt;em&gt;scale&lt;/em&gt; (the floating-point span each integer step covers) and &lt;code&gt;z&lt;/code&gt; is the &lt;em&gt;zero point&lt;/em&gt; (the integer that maps back to zero in the original space). Dequantization at inference time recovers an approximation: &lt;code&gt;x' = q * s + z&lt;/code&gt;. The two free parameters; &lt;code&gt;s&lt;/code&gt; and &lt;code&gt;z&lt;/code&gt;; are chosen per tensor (or per channel, or per group of channels) to minimize the round-trip error, and the choice of granularity is the first major axis of the design space.&lt;/p&gt;&#10;&lt;p&gt;Distillation is the process of training a smaller student model to imitate the output of a larger teacher model. The 2015 paper that named the technique, &lt;a href="https://arxiv.org/abs/1503.02531"&gt;Hinton, Vinyals, and Dean&amp;rsquo;s &amp;ldquo;Distilling the Knowledge in a Neural Network&amp;rdquo;&lt;/a&gt;, made the core observation that became the entire field: the teacher&amp;rsquo;s full &lt;em&gt;probability distribution&lt;/em&gt; over the output classes carries far more information than the hard label alone, because the relative ranking of the wrong answers encodes structure about the input that the student can learn from. A teacher that says &amp;ldquo;this is 80% cat, 19% dog, 1% airplane&amp;rdquo; tells the student something a teacher that just says &amp;ldquo;cat&amp;rdquo; doesn&amp;rsquo;t; namely that the input is more dog-like than airplane-like. Training the student to match the soft distribution rather than the hard label is the entire pitch, and it works.&lt;/p&gt;&#10;&lt;p&gt;Both techniques compress; the difference is &lt;em&gt;what&lt;/em&gt; they compress. Quantization keeps the same number of weights but stores each one cheaper. Distillation throws away weights entirely and recovers behavior through a different network shape. Most production stacks use both.&lt;/p&gt;&#10;&lt;h2 id="quantization-mechanics"&gt;Quantization mechanics&lt;/h2&gt;&#10;&lt;p&gt;The mechanical primitives are worth being precise about. The vocabulary is dense but each piece pulls weight in production.&lt;/p&gt;&#10;&lt;p&gt;Bit width. The headline number. The 2026 production lineup is INT8, INT4, FP8, and increasingly FP4. INT8 retains essentially full quality on most workloads at 2× memory savings vs FP16; INT4 cuts memory 4× but starts to show degradation on reasoning-heavy tasks; FP8 sits between the two on memory but pulls native hardware support on Hopper and Blackwell GPUs; FP4 is the Blackwell-era frontier and the place where current research is concentrated.&lt;/p&gt;&#10;&lt;p&gt;Quantization scheme. &lt;em&gt;Symmetric&lt;/em&gt; vs &lt;em&gt;asymmetric&lt;/em&gt;: symmetric maps &lt;code&gt;[-α, α]&lt;/code&gt; to &lt;code&gt;[-2^(b-1), 2^(b-1) - 1]&lt;/code&gt; with &lt;code&gt;z=0&lt;/code&gt;; asymmetric maps &lt;code&gt;[min, max]&lt;/code&gt; to &lt;code&gt;[0, 2^b - 1]&lt;/code&gt; with &lt;code&gt;z&lt;/code&gt; non-zero. Weights are usually symmetric (their distributions are roughly centered); activations are usually asymmetric (they&amp;rsquo;re often non-negative after ReLU/SiLU). &lt;em&gt;Uniform&lt;/em&gt; vs &lt;em&gt;non-uniform&lt;/em&gt;: uniform spaces the representable values evenly across the range; non-uniform (e.g. NF4, &lt;a href="https://arxiv.org/abs/2305.14314"&gt;the NormalFloat 4-bit format from QLoRA&lt;/a&gt;) spaces them according to a target distribution; for normally-distributed weights, NF4 packs values quantile-spaced over the normal so each representable value covers about the same density of actual weights. NF4 is the reason QLoRA works at 4 bits where naive INT4 doesn&amp;rsquo;t.&lt;/p&gt;&#10;&lt;p&gt;Granularity. Where the &lt;code&gt;(s, z)&lt;/code&gt; parameters are computed. &lt;em&gt;Per-tensor&lt;/em&gt;: one scale for the whole matrix. Fast, brutal, loses quality on anything but small models. &lt;em&gt;Per-channel&lt;/em&gt;: one scale per output channel of a linear layer. The current standard for weight quantization. &lt;em&gt;Per-group&lt;/em&gt;: one scale per group of N consecutive weights inside a channel (typically N=64 or N=128). The best quality-per-bit at INT4; the cost is slightly more metadata. &lt;em&gt;Per-token&lt;/em&gt;: one scale per input token for activations. Necessary for activation quantization because activation statistics change per token.&lt;/p&gt;&#10;&lt;p&gt;Weight-only vs weight-and-activation. &lt;em&gt;Weight-only&lt;/em&gt; (W4A16, W8A16): quantize only the weights, keep activations in FP16 or BF16. The cheap, safe, default. Used for AWQ and GPTQ. &lt;em&gt;Weight-and-activation&lt;/em&gt; (W8A8, W4A8): quantize both. Higher throughput on supporting hardware, but requires solving the activation-outlier problem. SmoothQuant, ZeroQuant, and FP8 fall here.&lt;/p&gt;&#10;&lt;p&gt;Static vs dynamic. &lt;em&gt;Static&lt;/em&gt; (calibration-based): run a small calibration dataset through the model once to determine &lt;code&gt;(s, z)&lt;/code&gt; per channel, freeze them. &lt;em&gt;Dynamic&lt;/em&gt;: compute &lt;code&gt;(s, z)&lt;/code&gt; per inference at runtime from the actual activation values. Dynamic is more accurate for activations but adds runtime overhead; static is the production default once a good calibration set exists.&lt;/p&gt;&#10;&lt;h2 id="quantization-algorithms"&gt;Quantization algorithms&lt;/h2&gt;&#10;&lt;p&gt;The interesting work isn&amp;rsquo;t in the bit width; it&amp;rsquo;s in the algorithm that picks the per-channel scales and decides how to compensate for the quantization error.&lt;/p&gt;&#10;&lt;p&gt;&lt;a href="https://arxiv.org/abs/2210.17323"&gt;GPTQ (Frantar et al., 2022)&lt;/a&gt; quantizes one transformer layer and one column at a time. After choosing the integer value for a column, it updates the remaining weights to compensate for the quantization error using second-order information from the layer&amp;rsquo;s input Hessian. This follows the &lt;a href="https://arxiv.org/abs/2208.11580"&gt;Optimal Brain Quantization&lt;/a&gt; approach: each greedy decision changes the remaining choices. GPTQ quantized OPT-175B and BLOOM-176B to 4 bits in about four GPU hours with little perplexity increase. It is slower than naive quantization, but retains more quality at 4 bits.&lt;/p&gt;&#10;&lt;p&gt;&lt;a href="https://arxiv.org/abs/2306.00978"&gt;AWQ (Lin et al., 2023)&lt;/a&gt;. The activation-aware lightweight. AWQ&amp;rsquo;s central observation: not all weights are equally important to quantize well. About 1% of weights; the ones connected to the highest-magnitude activations; carry most of the information; the other 99% can be quantized aggressively. AWQ finds the salient weights by looking at the activation magnitudes (not the weight magnitudes; that&amp;rsquo;s the &amp;ldquo;activation-aware&amp;rdquo; name) and applies a per-channel scaling that amplifies the salient weights before quantization, then divides the activations by the same factor at runtime to cancel out the amplification. The math is exact; it&amp;rsquo;s a similarity transform; but the salient weights now occupy a wider slice of the quantization range, which preserves their precision. AWQ skips GPTQ&amp;rsquo;s slow Hessian computation and runs much faster, and the quality at INT4 is consistently equal to or better than GPTQ. &lt;a href="https://github.com/mit-han-lab/llm-awq"&gt;AWQ is the production INT4 default in 2026&lt;/a&gt;; most pre-quantized 4-bit checkpoints on Hugging Face are AWQ.&lt;/p&gt;&#10;&lt;p&gt;&lt;a href="https://arxiv.org/abs/2211.10438"&gt;SmoothQuant (Xiao et al., 2022)&lt;/a&gt;. The W8A8 unlocker. The activation-outlier problem hits hard above 6.7B parameters: a small fraction of activation channels (often &amp;lt;1%) carry magnitudes 100× larger than the rest, and naive activation quantization either truncates those channels (catastrophic quality loss) or sets the scale to cover them (catastrophic precision loss on the other 99%). SmoothQuant&amp;rsquo;s fix: migrate the variance from activations to weights by an offline per-channel rescaling. Activations get smoother (easier to quantize), weights get spikier (still quantizable because they have headroom), and a similarity transform absorbs the rescaling so the model&amp;rsquo;s outputs are unchanged. The result is W8A8 quantization with quality at parity to W8A16, and on hardware with INT8 tensor cores the throughput doubles.&lt;/p&gt;&#10;&lt;p&gt;FP8. The hardware-native 8-bit float. Hopper and Blackwell GPUs include native FP8 tensor cores that compute matrix multiplications at 2× the rate of FP16. FP8 comes in two flavors: E4M3 (4 exponent bits, 3 mantissa bits; more precision, less range) and E5M2 (5 exponent bits, 2 mantissa bits; more range, less precision). Production use typically pairs E4M3 for weights and E5M2 for activations. The pitch: native hardware support means no special kernels, near-FP16 quality out of the box, and dramatic memory and throughput gains. &lt;a href="https://docs.nvidia.com/tensorrt-llm/"&gt;TensorRT-LLM doubles throughput on H100 with FP8&lt;/a&gt; at minimal accuracy cost, and on Blackwell the FP8 advantage compounds with the new FP4 path.&lt;/p&gt;&#10;&lt;p&gt;A summary table of the 2026 lineup:&lt;/p&gt;&#10;&lt;table&gt;&#10;&#9;&lt;thead&gt;&#10;&#9;&#9;&#9;&lt;tr&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;th&gt;Format&lt;/th&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;th&gt;Bits&lt;/th&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;th&gt;Weight/Act&lt;/th&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;th&gt;Calibration&lt;/th&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;th&gt;Quality at parity&lt;/th&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;th&gt;Hardware&lt;/th&gt;&#10;&#9;&#9;&#9;&lt;/tr&gt;&#10;&#9;&lt;/thead&gt;&#10;&#9;&lt;tbody&gt;&#10;&#9;&#9;&#9;&lt;tr&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;td&gt;FP16/BF16&lt;/td&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;td&gt;16&lt;/td&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;td&gt;W16A16&lt;/td&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;td&gt;;&lt;/td&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;td&gt;baseline&lt;/td&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;td&gt;universal&lt;/td&gt;&#10;&#9;&#9;&#9;&lt;/tr&gt;&#10;&#9;&#9;&#9;&lt;tr&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;td&gt;FP8 (E4M3/E5M2)&lt;/td&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;td&gt;8&lt;/td&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;td&gt;W8A8&lt;/td&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;td&gt;per-tensor or per-channel&lt;/td&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;td&gt;within ~0.1 pp&lt;/td&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;td&gt;Hopper, Blackwell&lt;/td&gt;&#10;&#9;&#9;&#9;&lt;/tr&gt;&#10;&#9;&#9;&#9;&lt;tr&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;td&gt;INT8 (SmoothQuant)&lt;/td&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;td&gt;8&lt;/td&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;td&gt;W8A8&lt;/td&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;td&gt;per-channel + smoothing&lt;/td&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;td&gt;within ~0.2 pp&lt;/td&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;td&gt;broad&lt;/td&gt;&#10;&#9;&#9;&#9;&lt;/tr&gt;&#10;&#9;&#9;&#9;&lt;tr&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;td&gt;INT8 (weight-only)&lt;/td&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;td&gt;8&lt;/td&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;td&gt;W8A16&lt;/td&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;td&gt;per-channel&lt;/td&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;td&gt;within ~0.1 pp&lt;/td&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;td&gt;universal&lt;/td&gt;&#10;&#9;&#9;&#9;&lt;/tr&gt;&#10;&#9;&#9;&#9;&lt;tr&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;td&gt;INT4 (AWQ)&lt;/td&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;td&gt;4&lt;/td&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;td&gt;W4A16&lt;/td&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;td&gt;activation-aware per-group&lt;/td&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;td&gt;within ~0.5–1 pp&lt;/td&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;td&gt;broad&lt;/td&gt;&#10;&#9;&#9;&#9;&lt;/tr&gt;&#10;&#9;&#9;&#9;&lt;tr&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;td&gt;INT4 (GPTQ)&lt;/td&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;td&gt;4&lt;/td&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;td&gt;W4A16&lt;/td&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;td&gt;Hessian-based per-group&lt;/td&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;td&gt;within ~0.5–1 pp&lt;/td&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;td&gt;broad&lt;/td&gt;&#10;&#9;&#9;&#9;&lt;/tr&gt;&#10;&#9;&#9;&#9;&lt;tr&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;td&gt;NF4 (QLoRA-style)&lt;/td&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;td&gt;4&lt;/td&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;td&gt;W4A16&lt;/td&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;td&gt;none (datatype is the trick)&lt;/td&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;td&gt;within ~0.5–1 pp&lt;/td&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;td&gt;broad&lt;/td&gt;&#10;&#9;&#9;&#9;&lt;/tr&gt;&#10;&#9;&#9;&#9;&lt;tr&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;td&gt;FP4 (Blackwell)&lt;/td&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;td&gt;4&lt;/td&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;td&gt;W4A4&lt;/td&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;td&gt;per-tensor or per-channel&lt;/td&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;td&gt;within ~1 pp on early benchmarks&lt;/td&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;td&gt;Blackwell only&lt;/td&gt;&#10;&#9;&#9;&#9;&lt;/tr&gt;&#10;&#9;&lt;/tbody&gt;&#10;&lt;/table&gt;&#10;&lt;p&gt;FP8 suits Hopper or Blackwell deployments that prioritize throughput and quality. AWQ INT4 suits memory-constrained or smaller-hardware deployments. SmoothQuant W8A8 targets INT8 throughput on broader hardware.&lt;/p&gt;&#10;&lt;h2 id="distillation-mechanics"&gt;Distillation mechanics&lt;/h2&gt;&#10;&lt;p&gt;The soft-target loss. The original distillation loss from Hinton et al. is a weighted combination of two terms: the standard hard-label cross-entropy and the soft-target KL divergence between the student and teacher distributions. The trick that makes the soft distribution useful is &lt;em&gt;temperature scaling&lt;/em&gt;: divide the logits by &lt;code&gt;T &amp;gt; 1&lt;/code&gt; before the softmax. Higher &lt;code&gt;T&lt;/code&gt; flattens the distribution, exposing the smaller probabilities the teacher assigns to the wrong answers. At &lt;code&gt;T=1&lt;/code&gt; the teacher&amp;rsquo;s confident answer drowns out everything else; at &lt;code&gt;T=4&lt;/code&gt; or &lt;code&gt;T=10&lt;/code&gt;, the relative ranking of the wrong answers becomes visible and trainable. The loss is &lt;code&gt;α · CE(student_T1, hard_label) + (1-α) · T² · KL(student_T || teacher_T)&lt;/code&gt;, with &lt;code&gt;T&lt;/code&gt; typically in &lt;code&gt;[2, 10]&lt;/code&gt; and &lt;code&gt;α&lt;/code&gt; typically around &lt;code&gt;0.5&lt;/code&gt;. The &lt;code&gt;T²&lt;/code&gt; factor exists because the gradients of the softened KL are smaller by &lt;code&gt;1/T²&lt;/code&gt;, and rescaling restores them to the same magnitude as the hard-label gradient.&lt;/p&gt;&#10;&lt;p&gt;Distillation regimes. &lt;em&gt;Logit-level distillation&lt;/em&gt; (the Hinton original): the student learns to match the teacher&amp;rsquo;s full output distribution. Needs access to teacher logits; easy with open-weight teachers, impossible with closed-API teachers that only return text. &lt;em&gt;Sequence-level distillation&lt;/em&gt; (the workhorse): generate text from the teacher, fine-tune the student on the (input, generated-output) pairs as standard SFT. Doesn&amp;rsquo;t need logits; works with any teacher you can query; but throws away most of the per-token distribution information. &lt;em&gt;On-policy distillation&lt;/em&gt; (&lt;a href="https://arxiv.org/abs/2306.08543"&gt;MiniLLM, Gu et al., 2023&lt;/a&gt;): student generates a response, teacher scores it, student updates via policy gradient with reverse-KL objective. Better calibration on long generations than forward-KL distillation, but needs teacher logits and is more expensive than offline SFT. The 2026 production default is sequence-level; closed-API teachers are too useful to give up; with on-policy used for capability transfer when access to teacher logits is available.&lt;/p&gt;&#10;&lt;p&gt;Dataset construction. The single most important variable. The distillation corpus needs to cover the student&amp;rsquo;s deployment distribution; if the corpus is off-distribution, the student will be off-distribution. The standard recipe: capture a representative sample of real user traffic, run it through the teacher, use the (input, teacher-output) pairs as training data. For reasoning tasks, capture the full chain-of-thought from the teacher; that&amp;rsquo;s what DeepSeek-R1 did with its &lt;a href="https://huggingface.co/deepseek-ai/DeepSeek-R1-Distill-Qwen-7B"&gt;800k reasoning trajectories&lt;/a&gt;, and the result was small students with non-trivial chunks of the teacher&amp;rsquo;s reasoning quality. The corpus size matters less than the corpus distribution: 50k well-curated examples typically beat 500k randomly sampled ones.&lt;/p&gt;&#10;&lt;p&gt;Production examples. &lt;a href="https://arxiv.org/abs/1910.01108"&gt;DistilBERT (Sanh et al., 2019)&lt;/a&gt; is the classic; 40% smaller than BERT, 60% faster, 95% of BERT&amp;rsquo;s quality on GLUE. The recipe (smaller architecture, distillation from the full BERT teacher) became the template the field has been iterating on since. The 2024-2026 evolution: distill smaller students from much larger teachers, on much larger corpora, with much more careful corpus construction. The &lt;a href="https://huggingface.co/collections/deepseek-ai/deepseek-r1-678e1e131c0169c0bc89728d"&gt;DeepSeek-R1-Distill family&lt;/a&gt;; 1.5B through 70B students distilled from the R1 teacher; is the current canonical demonstration that distillation can transfer reasoning capabilities, not just pattern-matching, when the corpus is rich enough. OpenAI&amp;rsquo;s &lt;a href="https://openai.com/index/gpt-4o-mini-advancing-cost-efficient-intelligence/"&gt;GPT-4o mini&lt;/a&gt; is widely understood to be a distillation of GPT-4o, though the recipe is undisclosed.&lt;/p&gt;&#10;&lt;h2 id="post-training-quantization-in-python"&gt;Post-training quantization in Python&lt;/h2&gt;&#10;&lt;p&gt;The 2026 idiomatic way to quantize a Hugging Face model for vLLM serving is &lt;a href="https://github.com/vllm-project/llm-compressor"&gt;&lt;code&gt;llm-compressor&lt;/code&gt;&lt;/a&gt;, the official toolkit maintained by the vLLM team. It supports INT4 (GPTQ-style), INT8 (weight-only and W8A8), and FP8 with a consistent API. Install: &lt;code&gt;pip install llmcompressor&amp;gt;=0.4 transformers&amp;gt;=4.46 datasets&amp;gt;=3&lt;/code&gt;.&lt;/p&gt;&#10;&lt;figure class="nb-code"&gt;&#10; &lt;header class="nb-code-chrome"&gt;&#10; &lt;span class="nb-code-label"&gt;python&lt;/span&gt;&#10; &#10; &lt;/header&gt;&#10; &lt;div class="nb-code-body"&gt;&lt;div class="highlight"&gt;&lt;div class="chroma"&gt;&#10;&lt;table class="lntable"&gt;&lt;tr&gt;&lt;td class="lntd"&gt;&#10;&lt;pre tabindex="0" class="chroma"&gt;&lt;code&gt;&lt;span class="lnt"&gt; 1&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 2&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 3&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 4&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 5&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 6&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 7&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 8&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 9&#10;&lt;/span&gt;&lt;span class="lnt"&gt;10&#10;&lt;/span&gt;&lt;span class="lnt"&gt;11&#10;&lt;/span&gt;&lt;span class="lnt"&gt;12&#10;&lt;/span&gt;&lt;span class="lnt"&gt;13&#10;&lt;/span&gt;&lt;span class="lnt"&gt;14&#10;&lt;/span&gt;&lt;span class="lnt"&gt;15&#10;&lt;/span&gt;&lt;span class="lnt"&gt;16&#10;&lt;/span&gt;&lt;span class="lnt"&gt;17&#10;&lt;/span&gt;&lt;span class="lnt"&gt;18&#10;&lt;/span&gt;&lt;span class="lnt"&gt;19&#10;&lt;/span&gt;&lt;span class="lnt"&gt;20&#10;&lt;/span&gt;&lt;span class="lnt"&gt;21&#10;&lt;/span&gt;&lt;span class="lnt"&gt;22&#10;&lt;/span&gt;&lt;span class="lnt"&gt;23&#10;&lt;/span&gt;&lt;span class="lnt"&gt;24&#10;&lt;/span&gt;&lt;span class="lnt"&gt;25&#10;&lt;/span&gt;&lt;span class="lnt"&gt;26&#10;&lt;/span&gt;&lt;span class="lnt"&gt;27&#10;&lt;/span&gt;&lt;span class="lnt"&gt;28&#10;&lt;/span&gt;&lt;span class="lnt"&gt;29&#10;&lt;/span&gt;&lt;span class="lnt"&gt;30&#10;&lt;/span&gt;&lt;span class="lnt"&gt;31&#10;&lt;/span&gt;&lt;span class="lnt"&gt;32&#10;&lt;/span&gt;&lt;span class="lnt"&gt;33&#10;&lt;/span&gt;&lt;span class="lnt"&gt;34&#10;&lt;/span&gt;&lt;span class="lnt"&gt;35&#10;&lt;/span&gt;&lt;span class="lnt"&gt;36&#10;&lt;/span&gt;&lt;span class="lnt"&gt;37&#10;&lt;/span&gt;&lt;span class="lnt"&gt;38&#10;&lt;/span&gt;&lt;span class="lnt"&gt;39&#10;&lt;/span&gt;&lt;span class="lnt"&gt;40&#10;&lt;/span&gt;&lt;span class="lnt"&gt;41&#10;&lt;/span&gt;&lt;span class="lnt"&gt;42&#10;&lt;/span&gt;&lt;span class="lnt"&gt;43&#10;&lt;/span&gt;&lt;span class="lnt"&gt;44&#10;&lt;/span&gt;&lt;span class="lnt"&gt;45&#10;&lt;/span&gt;&lt;span class="lnt"&gt;46&#10;&lt;/span&gt;&lt;span class="lnt"&gt;47&#10;&lt;/span&gt;&lt;span class="lnt"&gt;48&#10;&lt;/span&gt;&lt;span class="lnt"&gt;49&#10;&lt;/span&gt;&lt;span class="lnt"&gt;50&#10;&lt;/span&gt;&lt;span class="lnt"&gt;51&#10;&lt;/span&gt;&lt;span class="lnt"&gt;52&#10;&lt;/span&gt;&lt;span class="lnt"&gt;53&#10;&lt;/span&gt;&lt;span class="lnt"&gt;54&#10;&lt;/span&gt;&lt;span class="lnt"&gt;55&#10;&lt;/span&gt;&lt;span class="lnt"&gt;56&#10;&lt;/span&gt;&lt;span class="lnt"&gt;57&#10;&lt;/span&gt;&lt;span class="lnt"&gt;58&#10;&lt;/span&gt;&lt;span class="lnt"&gt;59&#10;&lt;/span&gt;&lt;span class="lnt"&gt;60&#10;&lt;/span&gt;&lt;span class="lnt"&gt;61&#10;&lt;/span&gt;&lt;span class="lnt"&gt;62&#10;&lt;/span&gt;&lt;span class="lnt"&gt;63&#10;&lt;/span&gt;&lt;span class="lnt"&gt;64&#10;&lt;/span&gt;&lt;span class="lnt"&gt;65&#10;&lt;/span&gt;&lt;span class="lnt"&gt;66&#10;&lt;/span&gt;&lt;span class="lnt"&gt;67&#10;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/td&gt;&#10;&lt;td class="lntd"&gt;&#10;&lt;pre tabindex="0" class="chroma"&gt;&lt;code class="language-python" data-lang="python"&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="kn"&gt;from&lt;/span&gt; &lt;span class="nn"&gt;datasets&lt;/span&gt; &lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;load_dataset&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="kn"&gt;from&lt;/span&gt; &lt;span class="nn"&gt;transformers&lt;/span&gt; &lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;AutoModelForCausalLM&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;AutoTokenizer&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="kn"&gt;from&lt;/span&gt; &lt;span class="nn"&gt;llmcompressor.transformers&lt;/span&gt; &lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;oneshot&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="kn"&gt;from&lt;/span&gt; &lt;span class="nn"&gt;llmcompressor.modifiers.quantization&lt;/span&gt; &lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;GPTQModifier&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;QuantizationModifier&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="n"&gt;MODEL&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s2"&gt;&amp;#34;meta-llama/Meta-Llama-3.1-8B-Instruct&amp;#34;&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="n"&gt;OUT_INT4&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s2"&gt;&amp;#34;./Llama-3.1-8B-INT4&amp;#34;&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="n"&gt;OUT_FP8&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s2"&gt;&amp;#34;./Llama-3.1-8B-FP8&amp;#34;&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="c1"&gt;# Load the base model and tokenizer in fp16 (compression converts in place).&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="n"&gt;model&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;AutoModelForCausalLM&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;from_pretrained&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;MODEL&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;torch_dtype&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;#34;auto&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;device_map&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;#34;auto&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="n"&gt;tokenizer&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;AutoTokenizer&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;from_pretrained&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;MODEL&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="c1"&gt;# Calibration set: a small slice of representative inputs. 512 samples is the&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="c1"&gt;# usual sweet spot — enough to determine per-channel statistics, small enough&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="c1"&gt;# to run on a single GPU in minutes.&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="n"&gt;calib&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;load_dataset&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;&amp;#34;HuggingFaceH4/ultrachat_200k&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;split&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;#34;train_sft[:512]&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="n"&gt;calib&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;calib&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;map&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="k"&gt;lambda&lt;/span&gt; &lt;span class="n"&gt;row&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="s2"&gt;&amp;#34;text&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;tokenizer&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;apply_chat_template&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;row&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;&amp;#34;messages&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt; &lt;span class="n"&gt;tokenize&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="kc"&gt;False&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="p"&gt;})&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="n"&gt;calib_texts&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;calib&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;&amp;#34;text&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="c1"&gt;# --------------------------------------------------------------------------&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="c1"&gt;# INT4 weight-only with GPTQ-style Hessian-based quantization.&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="c1"&gt;# This is the workhorse for memory-constrained serving — Llama-3.1-8B in INT4&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="c1"&gt;# fits comfortably on a 16GB GPU vs 18GB+ for fp16.&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="c1"&gt;# --------------------------------------------------------------------------&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="n"&gt;recipe_int4&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;GPTQModifier&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;targets&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;#34;Linear&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;scheme&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;#34;W4A16&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="c1"&gt;# 4-bit weights, 16-bit activations&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;ignore&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;&amp;#34;lm_head&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt; &lt;span class="c1"&gt;# skip the output projection&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;dampening_frac&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mf"&gt;0.1&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="c1"&gt;# Hessian regularization&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="p"&gt;)&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="n"&gt;oneshot&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;model&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;model&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;dataset&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;calib_texts&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;recipe&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;recipe_int4&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;max_seq_length&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;2048&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;num_calibration_samples&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;512&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;output_dir&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;OUT_INT4&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="p"&gt;)&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="c1"&gt;# --------------------------------------------------------------------------&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="c1"&gt;# FP8 weight-and-activation quantization for Hopper/Blackwell deployment.&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="c1"&gt;# Native hardware support — the FP8 tensor cores on H100/B200 double matmul&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="c1"&gt;# throughput vs FP16 with essentially no quality loss.&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="c1"&gt;# --------------------------------------------------------------------------&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="c1"&gt;# Reload fresh model since the previous call mutated weights in-place.&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="n"&gt;model&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;AutoModelForCausalLM&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;from_pretrained&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;MODEL&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;torch_dtype&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;#34;auto&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;device_map&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;#34;auto&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="n"&gt;recipe_fp8&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;QuantizationModifier&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;targets&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;#34;Linear&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;scheme&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;#34;FP8&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="c1"&gt;# E4M3 weights, E5M2 activations&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;ignore&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;&amp;#34;lm_head&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="p"&gt;)&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="n"&gt;oneshot&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;model&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;model&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;dataset&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;calib_texts&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;recipe&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;recipe_fp8&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;max_seq_length&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;2048&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;num_calibration_samples&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;512&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;output_dir&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;OUT_FP8&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="p"&gt;)&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="c1"&gt;# Resulting checkpoints can be served directly by vLLM:&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="c1"&gt;# vllm serve ./Llama-3.1-8B-INT4 --quantization compressed-tensors&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="c1"&gt;# vllm serve ./Llama-3.1-8B-FP8 --quantization compressed-tensors&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/td&gt;&lt;/tr&gt;&lt;/table&gt;&#10;&lt;/div&gt;&#10;&lt;/div&gt;&lt;/div&gt;&#10;&lt;/figure&gt;&#10;&lt;p&gt;Calibration data should match production traffic; code workloads need code samples, while chat workloads need conversational samples. The output projection often remains unquantized because errors there affect the token distribution across the vocabulary. INT4 and FP8 artifacts can share calibration data while serving different hardware and memory constraints.&lt;/p&gt;&#10;</content:encoded></item><item><title>LoRA and Parameter-Efficient Fine-Tuning</title><link>https://jatinbansal.com/ai-engineering/lora-peft/</link><guid isPermaLink="true">https://jatinbansal.com/ai-engineering/lora-peft/</guid><pubDate>Tue, 26 May 2026 19:05:00 +0000</pubDate><dc:creator>Jatin Bansal</dc:creator><category>AI Engineering</category><description>How LoRA and QLoRA reduce trainable parameters, GPU memory, and adapter deployment cost.</description><content:encoded>&lt;p&gt;Low-Rank Adaptation trains a small adapter while keeping the base model frozen. The resulting checkpoint can be a few megabytes rather than a copy of the full model, which lowers training memory and allows several adapters to share one loaded base model.&lt;/p&gt;&#10;&lt;h2 id="low-rank-weight-updates"&gt;Low-rank weight updates&lt;/h2&gt;&#10;&lt;p&gt;Parameter-Efficient Fine-Tuning adapts a model by training a small set of parameters while the original weights remain frozen. LoRA represents each adapted layer&amp;rsquo;s update as &lt;code&gt;BA&lt;/code&gt;, with &lt;code&gt;B ∈ R^(d×r)&lt;/code&gt;, &lt;code&gt;A ∈ R^(r×k)&lt;/code&gt;, and &lt;code&gt;r ≪ min(d, k)&lt;/code&gt;. At inference time the adapter can remain separate or be merged into &lt;code&gt;W&lt;/code&gt;.&lt;/p&gt;&#10;&lt;p&gt;The reason LoRA matters beyond the parameter-count win is the &lt;em&gt;empirical observation&lt;/em&gt; from the &lt;a href="https://arxiv.org/abs/2106.09685"&gt;original LoRA paper (Hu et al., 2021)&lt;/a&gt;: the weight update &lt;code&gt;ΔW&lt;/code&gt; that a full fine-tune installs &lt;em&gt;is itself approximately low-rank&lt;/em&gt;. Fine-tuning doesn&amp;rsquo;t need to rewrite every dimension of the weight matrix; it needs to nudge a handful of directions. So restricting the update to be exactly low-rank from the start doesn&amp;rsquo;t cost you much, because the unrestricted update was nearly low-rank anyway. This is the entire pitch: the constraint is matched to the structure of the problem.&lt;/p&gt;&#10;&lt;h2 id="tune-rank-scale-and-target-modules"&gt;Tune rank, scale, and target modules&lt;/h2&gt;&#10;&lt;p&gt;The math of LoRA is one equation, three hyperparameters, and one initialization trick. Worth being precise about each.&lt;/p&gt;&#10;&lt;p&gt;&lt;strong&gt;The equation.&lt;/strong&gt; For each adapted layer, the modified forward pass becomes:&lt;/p&gt;&#10;&lt;figure class="nb-code"&gt;&#10; &lt;header class="nb-code-chrome"&gt;&#10; &lt;span class="nb-code-label"&gt;text&lt;/span&gt;&#10; &#10; &lt;/header&gt;&#10; &lt;div class="nb-code-body"&gt;&lt;div class="highlight"&gt;&lt;pre tabindex="0" class="chroma"&gt;&lt;code class="language-text" data-lang="text"&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;h = Wx + (α/r) · BAx&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;/div&gt;&#10;&lt;/figure&gt;&#10;&lt;p&gt;where &lt;code&gt;W&lt;/code&gt; is the frozen pretrained weight, &lt;code&gt;B ∈ R^(d×r)&lt;/code&gt; and &lt;code&gt;A ∈ R^(r×k)&lt;/code&gt; are the trainable matrices, &lt;code&gt;α&lt;/code&gt; is the LoRA alpha scaling factor, and &lt;code&gt;r&lt;/code&gt; is the rank. At initialization, &lt;code&gt;A&lt;/code&gt; is sampled from a small Gaussian and &lt;code&gt;B&lt;/code&gt; is initialized to zero; so the initial &lt;code&gt;BA&lt;/code&gt; product is the zero matrix, meaning the LoRA-augmented model is &lt;em&gt;exactly&lt;/em&gt; the base model at step 0. This is critical: the optimizer starts from the base model, not from a randomly perturbed version, and any drift is purely the result of training.&lt;/p&gt;&#10;&lt;p&gt;&lt;strong&gt;The rank &lt;code&gt;r&lt;/code&gt;.&lt;/strong&gt; This is the bottleneck dimension; the number of &amp;ldquo;directions&amp;rdquo; in weight space the adapter can move along. The cost is linear in &lt;code&gt;r&lt;/code&gt;: doubling the rank doubles the parameter count and roughly doubles the memory and compute cost of the LoRA forward/backward. Higher rank gives the adapter more capacity to fit complex behavior changes; lower rank acts as an implicit regularizer that prevents overfitting on small datasets. The &lt;a href="https://www.databricks.com/blog/efficient-fine-tuning-lora-guide-llms"&gt;production consensus in 2026&lt;/a&gt;: &lt;code&gt;r=16&lt;/code&gt; for stylistic adjustments, &lt;code&gt;r=32&lt;/code&gt; for general SFT, &lt;code&gt;r=64&lt;/code&gt; for complex multi-turn behaviors or coding tasks. Going above 128 is usually a sign you should be running full fine-tuning or switching to a stronger base model.&lt;/p&gt;&#10;&lt;p&gt;&lt;strong&gt;The alpha &lt;code&gt;α&lt;/code&gt;.&lt;/strong&gt; This is the &lt;em&gt;scaling factor&lt;/em&gt; applied to the LoRA delta. The actual update is &lt;code&gt;(α/r) · BA&lt;/code&gt;, not just &lt;code&gt;BA&lt;/code&gt;; so increasing α at fixed r amplifies the adapter&amp;rsquo;s contribution to the forward pass. The convention is to set &lt;code&gt;α = r&lt;/code&gt; (giving a scaling factor of 1.0), or &lt;code&gt;α = 2r&lt;/code&gt; (giving a factor of 2.0) as a stronger default. The &lt;a href="https://docs.unsloth.ai/get-started/fine-tuning-llms-guide/lora-hyperparameters-guide"&gt;Unsloth team&amp;rsquo;s 2026 documentation&lt;/a&gt; recommends starting with &lt;code&gt;α = r = 16&lt;/code&gt; for stability; if the loss curve is sluggish, bump α (not r) to amplify the existing capacity rather than adding new parameters.&lt;/p&gt;&#10;&lt;p&gt;&lt;strong&gt;The target modules.&lt;/strong&gt; The single hyperparameter that most affects final quality. The original LoRA paper applied adapters only to &lt;code&gt;q_proj&lt;/code&gt; and &lt;code&gt;v_proj&lt;/code&gt; (the attention query and value projections), but the &lt;a href="https://arxiv.org/abs/2305.14314"&gt;QLoRA paper (Dettmers et al., 2023)&lt;/a&gt; showed that &lt;strong&gt;applying LoRA to all linear layers; &lt;code&gt;q_proj&lt;/code&gt;, &lt;code&gt;k_proj&lt;/code&gt;, &lt;code&gt;v_proj&lt;/code&gt;, &lt;code&gt;o_proj&lt;/code&gt;, &lt;code&gt;gate_proj&lt;/code&gt;, &lt;code&gt;up_proj&lt;/code&gt;, &lt;code&gt;down_proj&lt;/code&gt;; consistently outperforms attention-only LoRA for the same parameter budget&lt;/strong&gt;. The MLP layers (&lt;code&gt;gate_proj&lt;/code&gt;, &lt;code&gt;up_proj&lt;/code&gt;, &lt;code&gt;down_proj&lt;/code&gt;) are where most factual recall and stylistic behavior is encoded; leaving them frozen means the adapter can&amp;rsquo;t reach the parts of the network that hold the most fine-tunable behavior. The 2026 default in &lt;a href="https://huggingface.co/docs/peft/main/package_reference/lora"&gt;PEFT 0.13+&lt;/a&gt; is &lt;code&gt;target_modules=&amp;quot;all-linear&amp;quot;&lt;/code&gt;, which picks up every linear layer in the model automatically. For most workloads this is the right starting point.&lt;/p&gt;&#10;&lt;p&gt;The math is one paragraph; the rest of LoRA&amp;rsquo;s complexity lives in the hyperparameter interactions and the variants that fix specific failure modes.&lt;/p&gt;&#10;&lt;h2 id="qlora-reduces-base-model-memory"&gt;QLoRA reduces base-model memory&lt;/h2&gt;&#10;&lt;p&gt;&lt;a href="https://arxiv.org/abs/2305.14314"&gt;QLoRA (Dettmers et al., 2023)&lt;/a&gt; is the engineering achievement that put 65B-parameter fine-tuning onto a single 48GB GPU. The pitch: instead of holding the frozen base model in 16-bit precision, &lt;strong&gt;hold it in 4-bit&lt;/strong&gt;, then attach standard 16-bit LoRA adapters on top. The 4-bit base is read-only; gradients flow through it via dequantization on the fly, but the quantized weights themselves never need to be updated, so the imprecision of 4-bit doesn&amp;rsquo;t show up where it would hurt. The trainable LoRA adapters stay in 16-bit, where the optimizer needs the dynamic range to make small updates accurately. Three engineering tricks make this work:&lt;/p&gt;&#10;&lt;p&gt;&lt;strong&gt;NF4 (NormalFloat 4-bit).&lt;/strong&gt; A custom 4-bit data type designed to be information-theoretically near-optimal for normally distributed weights. Standard 4-bit integer quantization spreads its 16 representable values uniformly across the range, but weight distributions are bell-shaped, so most of the precision is wasted on tails that contain few values. NF4 packs the 16 representable values quantile-spaced over a normal distribution, so each value covers roughly the same number of actual weights. Empirically, NF4 matches BF16 performance on downstream tasks; FP4 (the standard 4-bit float) is about 1% behind.&lt;/p&gt;&#10;&lt;p&gt;&lt;strong&gt;Double quantization.&lt;/strong&gt; Standard 4-bit quantization stores the quantization constants (the scale factors that map quantized integers back to floats) in FP32; and at 64 weights per scale factor, those scale factors add up to about 0.5 bits per weight of overhead. Double quantization quantizes the quantization constants themselves down to 8-bit, saving another 0.4 bits per weight at minimal quality cost. The names are confusing but the trick is real: it&amp;rsquo;s worth roughly 1 GB of memory for a 33B model.&lt;/p&gt;&#10;&lt;p&gt;&lt;strong&gt;Paged optimizers.&lt;/strong&gt; The optimizer state for AdamW is 2× the size of the parameters being optimized; for a 7B-parameter LoRA fine-tune, that&amp;rsquo;s manageable, but when the optimizer state is unpredictable (e.g. during a memory spike when a long sequence comes through), the GPU will OOM. QLoRA uses NVIDIA&amp;rsquo;s unified memory to page optimizer state to CPU RAM when GPU memory pressure is high. The performance cost is real (CPU paging is slow) but it&amp;rsquo;s the difference between a workload running and a workload OOMing, which is the right trade-off in practice.&lt;/p&gt;&#10;&lt;p&gt;The end result: a 7B model fine-tunes in about 6 hours on a single A100 40GB at a cost of around $12 for an &lt;a href="https://hjlabs.in/AIML/blog/post/llm-fine-tuning-best-practices.html"&gt;Llama 3 8B equivalent run on a public cloud GPU in 2026&lt;/a&gt;. A 70B model fine-tunes on a single A100 80GB or H100. The cost ratio versus full fine-tuning is roughly 50–100×, with most production benchmarks showing QLoRA within 1–2 percentage points of full fine-tuning on AlpacaEval or domain-specific evals.&lt;/p&gt;&#10;&lt;h2 id="code-a-full-qlora-fine-tune-in-trl--peft--bitsandbytes"&gt;Code: a full QLoRA fine-tune in TRL + PEFT + bitsandbytes&lt;/h2&gt;&#10;&lt;p&gt;The pipeline below trains a 7B model on a single 24GB GPU. Setup: &lt;code&gt;pip install &amp;quot;trl&amp;gt;=0.22&amp;quot; &amp;quot;transformers&amp;gt;=4.46&amp;quot; &amp;quot;datasets&amp;gt;=3&amp;quot; &amp;quot;peft&amp;gt;=0.13&amp;quot; &amp;quot;bitsandbytes&amp;gt;=0.43&amp;quot; accelerate&lt;/code&gt;.&lt;/p&gt;&#10;&lt;figure class="nb-code"&gt;&#10; &lt;header class="nb-code-chrome"&gt;&#10; &lt;span class="nb-code-label"&gt;python&lt;/span&gt;&#10; &#10; &lt;/header&gt;&#10; &lt;div class="nb-code-body"&gt;&lt;div class="highlight"&gt;&lt;div class="chroma"&gt;&#10;&lt;table class="lntable"&gt;&lt;tr&gt;&lt;td class="lntd"&gt;&#10;&lt;pre tabindex="0" class="chroma"&gt;&lt;code&gt;&lt;span class="lnt"&gt; 1&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 2&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 3&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 4&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 5&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 6&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 7&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 8&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 9&#10;&lt;/span&gt;&lt;span class="lnt"&gt;10&#10;&lt;/span&gt;&lt;span class="lnt"&gt;11&#10;&lt;/span&gt;&lt;span class="lnt"&gt;12&#10;&lt;/span&gt;&lt;span class="lnt"&gt;13&#10;&lt;/span&gt;&lt;span class="lnt"&gt;14&#10;&lt;/span&gt;&lt;span class="lnt"&gt;15&#10;&lt;/span&gt;&lt;span class="lnt"&gt;16&#10;&lt;/span&gt;&lt;span class="lnt"&gt;17&#10;&lt;/span&gt;&lt;span class="lnt"&gt;18&#10;&lt;/span&gt;&lt;span class="lnt"&gt;19&#10;&lt;/span&gt;&lt;span class="lnt"&gt;20&#10;&lt;/span&gt;&lt;span class="lnt"&gt;21&#10;&lt;/span&gt;&lt;span class="lnt"&gt;22&#10;&lt;/span&gt;&lt;span class="lnt"&gt;23&#10;&lt;/span&gt;&lt;span class="lnt"&gt;24&#10;&lt;/span&gt;&lt;span class="lnt"&gt;25&#10;&lt;/span&gt;&lt;span class="lnt"&gt;26&#10;&lt;/span&gt;&lt;span class="lnt"&gt;27&#10;&lt;/span&gt;&lt;span class="lnt"&gt;28&#10;&lt;/span&gt;&lt;span class="lnt"&gt;29&#10;&lt;/span&gt;&lt;span class="lnt"&gt;30&#10;&lt;/span&gt;&lt;span class="lnt"&gt;31&#10;&lt;/span&gt;&lt;span class="lnt"&gt;32&#10;&lt;/span&gt;&lt;span class="lnt"&gt;33&#10;&lt;/span&gt;&lt;span class="lnt"&gt;34&#10;&lt;/span&gt;&lt;span class="lnt"&gt;35&#10;&lt;/span&gt;&lt;span class="lnt"&gt;36&#10;&lt;/span&gt;&lt;span class="lnt"&gt;37&#10;&lt;/span&gt;&lt;span class="lnt"&gt;38&#10;&lt;/span&gt;&lt;span class="lnt"&gt;39&#10;&lt;/span&gt;&lt;span class="lnt"&gt;40&#10;&lt;/span&gt;&lt;span class="lnt"&gt;41&#10;&lt;/span&gt;&lt;span class="lnt"&gt;42&#10;&lt;/span&gt;&lt;span class="lnt"&gt;43&#10;&lt;/span&gt;&lt;span class="lnt"&gt;44&#10;&lt;/span&gt;&lt;span class="lnt"&gt;45&#10;&lt;/span&gt;&lt;span class="lnt"&gt;46&#10;&lt;/span&gt;&lt;span class="lnt"&gt;47&#10;&lt;/span&gt;&lt;span class="lnt"&gt;48&#10;&lt;/span&gt;&lt;span class="lnt"&gt;49&#10;&lt;/span&gt;&lt;span class="lnt"&gt;50&#10;&lt;/span&gt;&lt;span class="lnt"&gt;51&#10;&lt;/span&gt;&lt;span class="lnt"&gt;52&#10;&lt;/span&gt;&lt;span class="lnt"&gt;53&#10;&lt;/span&gt;&lt;span class="lnt"&gt;54&#10;&lt;/span&gt;&lt;span class="lnt"&gt;55&#10;&lt;/span&gt;&lt;span class="lnt"&gt;56&#10;&lt;/span&gt;&lt;span class="lnt"&gt;57&#10;&lt;/span&gt;&lt;span class="lnt"&gt;58&#10;&lt;/span&gt;&lt;span class="lnt"&gt;59&#10;&lt;/span&gt;&lt;span class="lnt"&gt;60&#10;&lt;/span&gt;&lt;span class="lnt"&gt;61&#10;&lt;/span&gt;&lt;span class="lnt"&gt;62&#10;&lt;/span&gt;&lt;span class="lnt"&gt;63&#10;&lt;/span&gt;&lt;span class="lnt"&gt;64&#10;&lt;/span&gt;&lt;span class="lnt"&gt;65&#10;&lt;/span&gt;&lt;span class="lnt"&gt;66&#10;&lt;/span&gt;&lt;span class="lnt"&gt;67&#10;&lt;/span&gt;&lt;span class="lnt"&gt;68&#10;&lt;/span&gt;&lt;span class="lnt"&gt;69&#10;&lt;/span&gt;&lt;span class="lnt"&gt;70&#10;&lt;/span&gt;&lt;span class="lnt"&gt;71&#10;&lt;/span&gt;&lt;span class="lnt"&gt;72&#10;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/td&gt;&#10;&lt;td class="lntd"&gt;&#10;&lt;pre tabindex="0" class="chroma"&gt;&lt;code class="language-python" data-lang="python"&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="nn"&gt;torch&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="kn"&gt;from&lt;/span&gt; &lt;span class="nn"&gt;datasets&lt;/span&gt; &lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;load_dataset&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="kn"&gt;from&lt;/span&gt; &lt;span class="nn"&gt;peft&lt;/span&gt; &lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;LoraConfig&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;get_peft_model&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="kn"&gt;from&lt;/span&gt; &lt;span class="nn"&gt;transformers&lt;/span&gt; &lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;AutoModelForCausalLM&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;AutoTokenizer&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;BitsAndBytesConfig&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="p"&gt;)&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="kn"&gt;from&lt;/span&gt; &lt;span class="nn"&gt;trl&lt;/span&gt; &lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;SFTConfig&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;SFTTrainer&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="n"&gt;BASE_MODEL&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s2"&gt;&amp;#34;Qwen/Qwen2.5-7B-Instruct&amp;#34;&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="n"&gt;OUT_DIR&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s2"&gt;&amp;#34;./qlora-qwen-7b&amp;#34;&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="c1"&gt;# 4-bit NF4 quantization: the frozen base sits in 4-bit; LoRA adapters stay in bf16.&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="n"&gt;bnb_config&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;BitsAndBytesConfig&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;load_in_4bit&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="kc"&gt;True&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;bnb_4bit_quant_type&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;#34;nf4&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="c1"&gt;# information-theoretically near-optimal&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;bnb_4bit_use_double_quant&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="kc"&gt;True&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="c1"&gt;# ~0.4 bits/weight saved&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;bnb_4bit_compute_dtype&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;torch&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;bfloat16&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="p"&gt;)&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="n"&gt;tokenizer&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;AutoTokenizer&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;from_pretrained&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;BASE_MODEL&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="n"&gt;model&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;AutoModelForCausalLM&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;from_pretrained&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;BASE_MODEL&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;quantization_config&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;bnb_config&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;device_map&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;#34;auto&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="p"&gt;)&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="c1"&gt;# LoRA config: the hyperparameter triangle.&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="n"&gt;peft_config&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;LoraConfig&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;32&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="c1"&gt;# rank: 32 is the general-SFT default&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;lora_alpha&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;64&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="c1"&gt;# alpha = 2r is a common stronger default&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;target_modules&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;#34;all-linear&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="c1"&gt;# 2026 default; picks every linear layer&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;lora_dropout&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mf"&gt;0.05&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;bias&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;#34;none&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;task_type&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;#34;CAUSAL_LM&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;use_dora&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="kc"&gt;True&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="c1"&gt;# DoRA: cheap quality upgrade&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="c1"&gt;# init_lora_weights=&amp;#34;pissa&amp;#34;, # PiSSA: faster convergence; expensive init&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="c1"&gt;# use_rslora=True, # rsLoRA: enable if going to r&amp;gt;64&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="p"&gt;)&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="c1"&gt;# Wrap the base model with the LoRA adapter.&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="n"&gt;model&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;get_peft_model&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;model&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;peft_config&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="n"&gt;model&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;print_trainable_parameters&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="c1"&gt;# &amp;gt;&amp;gt;&amp;gt; trainable params: 21,143,552 || all params: 7,636,361,856 || trainable%: 0.27&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="c1"&gt;# Public instruction-tuning dataset; small subset for demonstration.&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="n"&gt;ds&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;load_dataset&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;&amp;#34;HuggingFaceH4/ultrachat_200k&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;split&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;#34;train_sft[:5000]&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="n"&gt;config&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;SFTConfig&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;output_dir&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;OUT_DIR&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;num_train_epochs&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;per_device_train_batch_size&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;4&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;gradient_accumulation_steps&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;4&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="c1"&gt;# effective batch size 16&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;learning_rate&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mf"&gt;2e-4&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="c1"&gt;# LoRA LR is ~10x higher than full FT&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;lr_scheduler_type&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;#34;cosine&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;warmup_ratio&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mf"&gt;0.1&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;max_seq_length&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;2048&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;bf16&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="kc"&gt;True&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;logging_steps&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;20&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;save_strategy&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;#34;epoch&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;optim&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;#34;paged_adamw_8bit&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="c1"&gt;# QLoRA&amp;#39;s paged optimizer&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="p"&gt;)&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="n"&gt;trainer&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;SFTTrainer&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;model&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;model&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;args&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;config&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;train_dataset&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;ds&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;processing_class&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;tokenizer&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="p"&gt;)&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="n"&gt;trainer&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;train&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="n"&gt;trainer&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;save_model&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;OUT_DIR&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="c1"&gt;# saves only the ~84 MB adapter, not 14 GB base&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/td&gt;&lt;/tr&gt;&lt;/table&gt;&#10;&lt;/div&gt;&#10;&lt;/div&gt;&lt;/div&gt;&#10;&lt;/figure&gt;&#10;&lt;p&gt;The scaffold makes several training choices explicit.&lt;/p&gt;&#10;&lt;p&gt;&lt;code&gt;optim=&amp;quot;paged_adamw_8bit&amp;quot;&lt;/code&gt; is the paged optimizer from QLoRA. It pages optimizer state to CPU RAM under memory pressure; without it, long sequences in your batch can OOM the GPU during the AdamW second-moment update. Use it whenever you&amp;rsquo;re at the memory ceiling on a single GPU.&lt;/p&gt;&#10;&lt;p&gt;Second, the saved checkpoint at &lt;code&gt;OUT_DIR&lt;/code&gt; is the &lt;em&gt;adapter only&lt;/em&gt;; typically 50–200 MB depending on rank and model size. The base model is not duplicated; you reload it at inference time from the original checkpoint and attach the adapter. This is what makes multi-tenant serving viable: 100 adapters at 100 MB each is 10 GB of disk, versus 100× the base model size if you&amp;rsquo;d done full fine-tuning.&lt;/p&gt;&#10;&lt;p&gt;Third, the learning rate (&lt;code&gt;2e-4&lt;/code&gt;) is roughly 10× higher than full fine-tuning&amp;rsquo;s &lt;code&gt;2e-5&lt;/code&gt;. The LoRA adapter starts from zero contribution, so the optimizer needs aggressive updates to install the behavior; a too-low LR will just stall.&lt;/p&gt;&#10;&lt;h2 id="merge-or-hot-swap-adapters"&gt;Merge or hot-swap adapters&lt;/h2&gt;&#10;&lt;p&gt;After training, the adapter and base can either be merged into a single checkpoint or served as two artifacts composed at runtime. The decision is a single-axis trade-off and the right answer depends entirely on whether you&amp;rsquo;re serving one fine-tuned model or many.&lt;/p&gt;&#10;&lt;p&gt;&lt;strong&gt;Single-tenant: merge.&lt;/strong&gt; If exactly one adapter will ever be served against this base model, &lt;a href="https://huggingface.co/docs/peft/main/en/developer_guides/lora"&gt;&lt;code&gt;merge_and_unload()&lt;/code&gt;&lt;/a&gt; folds the adapter into the base weights and returns a standard transformers checkpoint. Inference latency drops by roughly 5–15% (no per-layer LoRA matmul) and deployment simplifies (no PEFT dependency at inference time). The cost is permanent: once merged, you can&amp;rsquo;t swap adapters or unmerge cheaply.&lt;/p&gt;&#10;&lt;figure class="nb-code"&gt;&#10; &lt;header class="nb-code-chrome"&gt;&#10; &lt;span class="nb-code-label"&gt;python&lt;/span&gt;&#10; &#10; &lt;/header&gt;&#10; &lt;div class="nb-code-body"&gt;&lt;div class="highlight"&gt;&lt;div class="chroma"&gt;&#10;&lt;table class="lntable"&gt;&lt;tr&gt;&lt;td class="lntd"&gt;&#10;&lt;pre tabindex="0" class="chroma"&gt;&lt;code&gt;&lt;span class="lnt"&gt;1&#10;&lt;/span&gt;&lt;span class="lnt"&gt;2&#10;&lt;/span&gt;&lt;span class="lnt"&gt;3&#10;&lt;/span&gt;&lt;span class="lnt"&gt;4&#10;&lt;/span&gt;&lt;span class="lnt"&gt;5&#10;&lt;/span&gt;&lt;span class="lnt"&gt;6&#10;&lt;/span&gt;&lt;span class="lnt"&gt;7&#10;&lt;/span&gt;&lt;span class="lnt"&gt;8&#10;&lt;/span&gt;&lt;span class="lnt"&gt;9&#10;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/td&gt;&#10;&lt;td class="lntd"&gt;&#10;&lt;pre tabindex="0" class="chroma"&gt;&lt;code class="language-python" data-lang="python"&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="kn"&gt;from&lt;/span&gt; &lt;span class="nn"&gt;peft&lt;/span&gt; &lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;PeftModel&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="kn"&gt;from&lt;/span&gt; &lt;span class="nn"&gt;transformers&lt;/span&gt; &lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;AutoModelForCausalLM&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="n"&gt;base&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;AutoModelForCausalLM&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;from_pretrained&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;BASE_MODEL&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;torch_dtype&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;#34;bfloat16&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="n"&gt;peft_model&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;PeftModel&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;from_pretrained&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;base&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;&amp;#34;./qlora-qwen-7b&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="c1"&gt;# Merge LoRA into the base; returns a standard transformers model.&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="n"&gt;merged&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;peft_model&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;merge_and_unload&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="n"&gt;merged&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;save_pretrained&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;&amp;#34;./qwen-7b-merged&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/td&gt;&lt;/tr&gt;&lt;/table&gt;&#10;&lt;/div&gt;&#10;&lt;/div&gt;&lt;/div&gt;&#10;&lt;/figure&gt;&#10;&lt;p&gt;&lt;strong&gt;Multi-tenant: serve unmerged with hot-swap.&lt;/strong&gt; If you&amp;rsquo;re serving N adapters against the same base model; one per customer, one per use case, one per A/B test variant; keep them unmerged. &lt;a href="https://blog.vllm.ai/2026/02/26/multi-lora.html"&gt;vLLM 0.7+ supports native dynamic LoRA loading&lt;/a&gt;: the base model lives permanently in GPU memory, adapters sit in CPU RAM (or NVMe), and the runtime pages an adapter onto the GPU per request; a transfer that takes single-digit milliseconds because the payload is tiny. The &lt;code&gt;--enable-lora&lt;/code&gt; and &lt;code&gt;--max-loras&lt;/code&gt; flags configure how many adapters can be resident on the GPU at once; PCIe transfer overhead is amortized across the request&amp;rsquo;s prefill and decode passes.&lt;/p&gt;&#10;&lt;figure class="nb-code"&gt;&#10; &lt;header class="nb-code-chrome"&gt;&#10; &lt;span class="nb-code-label"&gt;bash&lt;/span&gt;&#10; &#10; &lt;/header&gt;&#10; &lt;div class="nb-code-body"&gt;&lt;div class="highlight"&gt;&lt;div class="chroma"&gt;&#10;&lt;table class="lntable"&gt;&lt;tr&gt;&lt;td class="lntd"&gt;&#10;&lt;pre tabindex="0" class="chroma"&gt;&lt;code&gt;&lt;span class="lnt"&gt;1&#10;&lt;/span&gt;&lt;span class="lnt"&gt;2&#10;&lt;/span&gt;&lt;span class="lnt"&gt;3&#10;&lt;/span&gt;&lt;span class="lnt"&gt;4&#10;&lt;/span&gt;&lt;span class="lnt"&gt;5&#10;&lt;/span&gt;&lt;span class="lnt"&gt;6&#10;&lt;/span&gt;&lt;span class="lnt"&gt;7&#10;&lt;/span&gt;&lt;span class="lnt"&gt;8&#10;&lt;/span&gt;&lt;span class="lnt"&gt;9&#10;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/td&gt;&#10;&lt;td class="lntd"&gt;&#10;&lt;pre tabindex="0" class="chroma"&gt;&lt;code class="language-bash" data-lang="bash"&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="c1"&gt;# Launch vLLM with multi-LoRA serving&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;vllm serve meta-llama/Llama-3.1-8B-Instruct &lt;span class="se"&gt;\&#10;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; --enable-lora &lt;span class="se"&gt;\&#10;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; --max-loras &lt;span class="m"&gt;8&lt;/span&gt; &lt;span class="se"&gt;\&#10;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; --max-lora-rank &lt;span class="m"&gt;64&lt;/span&gt; &lt;span class="se"&gt;\&#10;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; --lora-modules &lt;span class="se"&gt;\&#10;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; tenant-a&lt;span class="o"&gt;=&lt;/span&gt;./adapters/tenant-a &lt;span class="se"&gt;\&#10;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; tenant-b&lt;span class="o"&gt;=&lt;/span&gt;./adapters/tenant-b &lt;span class="se"&gt;\&#10;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; tenant-c&lt;span class="o"&gt;=&lt;/span&gt;./adapters/tenant-c&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/td&gt;&lt;/tr&gt;&lt;/table&gt;&#10;&lt;/div&gt;&#10;&lt;/div&gt;&lt;/div&gt;&#10;&lt;/figure&gt;&#10;&lt;p&gt;At inference time, the client requests the adapter by name in the &lt;code&gt;model&lt;/code&gt; field of the API call, and vLLM routes the request through the correct LoRA. The cost economics flips here: instead of provisioning one GPU per fine-tuned model, you provision one GPU and host dozens to thousands of tenants on it. The break-even point is around 3–4 adapters; below that, single-adapter merged serving is faster, but above it, multi-LoRA dominates by an order of magnitude on dollar cost per request.&lt;/p&gt;&#10;&lt;p&gt;The &lt;a href="https://arxiv.org/abs/2311.03285"&gt;S-LoRA paper (Sheng et al., 2023)&lt;/a&gt; is the foundational work on this serving pattern; they showed that with batched LoRA computation and adapter caching policies, throughput stays close to base-model throughput even with thousands of adapters resident on a single GPU. vLLM&amp;rsquo;s implementation is the production-grade descendant of that work.&lt;/p&gt;&#10;&lt;h2 id="further-reading-from-the-field"&gt;Further reading from the field&lt;/h2&gt;&#10;&lt;ul&gt;&#10;&lt;li&gt;&lt;a href="https://arxiv.org/abs/2106.09685"&gt;Hu et al., 2021; &amp;ldquo;LoRA: Low-Rank Adaptation of Large Language Models&amp;rdquo;&lt;/a&gt;; the original paper. Section 4 is the empirical evidence that fine-tuning updates are low-rank; Section 7 is the comparison against other PEFT methods that LoRA superseded. Read it once for the intuition, even if you&amp;rsquo;ll never train without QLoRA on top.&lt;/li&gt;&#10;&lt;li&gt;&lt;a href="https://arxiv.org/abs/2305.14314"&gt;Dettmers et al., 2023; &amp;ldquo;QLoRA: Efficient Finetuning of Quantized LLMs&amp;rdquo;&lt;/a&gt;; the engineering work that made LoRA truly cheap. The NF4 derivation, double quantization, and paged optimizers are all here, along with the Guanaco results that anchored the open-source fine-tuning community for two years.&lt;/li&gt;&#10;&lt;li&gt;&lt;a href="https://magazine.sebastianraschka.com/p/practical-tips-for-finetuning-llms"&gt;Sebastian Raschka; &amp;ldquo;Practical Tips for Finetuning LLMs Using LoRA&amp;rdquo;&lt;/a&gt;; the cleanest single-source treatment of LoRA hyperparameter selection, with the empirical sweeps over rank, alpha, and target modules. If you&amp;rsquo;re tuning a LoRA pipeline, this is the article that decides your defaults.&lt;/li&gt;&#10;&lt;li&gt;&lt;a href="https://blog.vllm.ai/2026/02/26/multi-lora.html"&gt;vLLM blog; &amp;ldquo;Efficiently serve dozens of fine-tuned models with vLLM&amp;rdquo;&lt;/a&gt;; the production multi-LoRA pattern, with the configuration, the cost economics, and the throughput numbers. Pair with the &lt;a href="https://huggingface.co/docs/peft/main/en/developer_guides/lora"&gt;Hugging Face PEFT LoRA docs&lt;/a&gt; for the training-side API surface.&lt;/li&gt;&#10;&lt;/ul&gt;&#10;</content:encoded></item><item><title>DPO and Modern Alignment</title><link>https://jatinbansal.com/ai-engineering/dpo-and-alignment/</link><guid isPermaLink="true">https://jatinbansal.com/ai-engineering/dpo-and-alignment/</guid><pubDate>Tue, 26 May 2026 18:55:00 +0000</pubDate><dc:creator>Jatin Bansal</dc:creator><category>AI Engineering</category><description>Direct Preference Optimization, its common variants, production workflow, and failure modes.</description><content:encoded>&lt;p&gt;Direct Preference Optimization trains on &lt;code&gt;(prompt, chosen, rejected)&lt;/code&gt; triples without a separate reward model or online PPO loop. Operationally it resembles supervised fine-tuning with a custom loss and a frozen reference model, making preference optimization much easier to run and debug than a multi-component RLHF pipeline.&lt;/p&gt;&#10;&lt;h2 id="the-dpo-training-contract"&gt;The DPO training contract&lt;/h2&gt;&#10;&lt;p&gt;Direct Preference Optimization (DPO) is a supervised-learning algorithm that fits a language model to a preference dataset by directly optimizing a closed-form expression of the optimal RLHF policy, without training a separate reward model and without running a reinforcement-learning loop. The input is the same &lt;code&gt;(prompt, chosen, rejected)&lt;/code&gt; triples that an RLHF reward model would train on; the output is a model whose likelihoods on chosen responses go up relative to rejected ones, regularized by KL divergence against a frozen reference model. Mechanically, DPO is one custom loss function plugged into a standard &lt;code&gt;Trainer&lt;/code&gt;, with the only oddity being that each batch requires two forward passes; one through the trainable policy and one through the frozen reference; so the chosen/rejected log-probability ratios can be compared.&lt;/p&gt;&#10;&lt;p&gt;The reason DPO matters operationally is the absence of the reward model. The original &lt;a href="https://arxiv.org/abs/2203.02155"&gt;InstructGPT-style RLHF pipeline (Ouyang et al., 2022)&lt;/a&gt; required &lt;em&gt;training&lt;/em&gt; a reward model on the preferences, then running PPO with that reward model in the loop; six components, each a separate forward pass at training time, each a tuning surface, each a place for the pipeline to silently misbehave. The reward model in particular is the brittle joint: once the policy learns to exploit a gap in the reward model&amp;rsquo;s coverage, the entire pipeline starts producing reward-hacked outputs that score well on the reward model and poorly on humans. DPO sidesteps this entirely. The reward model is implicit; it falls out of the closed-form derivation; and there&amp;rsquo;s no separate component to hack.&lt;/p&gt;&#10;&lt;h2 id="understand-the-dpo-loss"&gt;Understand the DPO loss&lt;/h2&gt;&#10;&lt;p&gt;Two pieces are worth pulling apart because they govern the practical behavior of the trainer.&lt;/p&gt;&#10;&lt;p&gt;The implicit reward margin. Define &lt;code&gt;r̂_θ(x, y) = β log(π_θ(y|x) / π_ref(y|x))&lt;/code&gt;. This is the &amp;ldquo;implicit reward&amp;rdquo; the trained model assigns to a response, expressed as a log-likelihood ratio scaled by &lt;code&gt;β&lt;/code&gt;. The DPO loss pushes &lt;code&gt;r̂_θ(x, y_w) - r̂_θ(x, y_l)&lt;/code&gt; up; the implicit reward gap between chosen and rejected. &lt;em&gt;β controls the conversion factor between log-likelihood ratios and reward.&lt;/em&gt; High β (1.0+) means a small log-likelihood gap counts as a big reward gap; the loss is satisfied with small moves away from the reference and the policy stays close to the SFT model. Low β (0.01–0.05) means the log-likelihood gap has to be large to register as reward; the policy drifts further from the reference and learns more aggressively. The canonical default is 0.1, and most production runs sit between 0.05 and 0.5; &lt;a href="https://huggingface.co/docs/trl/en/dpo_trainer"&gt;Llama 3&amp;rsquo;s post-training&lt;/a&gt; used β = 0.1.&lt;/p&gt;&#10;&lt;p&gt;The two forward passes. Each training step requires four log-probabilities: &lt;code&gt;log π_θ(y_w|x)&lt;/code&gt;, &lt;code&gt;log π_θ(y_l|x)&lt;/code&gt;, &lt;code&gt;log π_ref(y_w|x)&lt;/code&gt;, &lt;code&gt;log π_ref(y_l|x)&lt;/code&gt;. The first two come from the trainable model; the second two come from the frozen reference. In TRL the reference model is loaded automatically as a copy of the policy at the start of training, kept on the same device, and never updated. This doubles the GPU memory footprint of the training run versus SFT; both models need to fit on the device, plus optimizer state for one of them. For large models, &lt;a href="https://jatinbansal.com/ai-engineering/lora-peft/"&gt;PEFT/LoRA&lt;/a&gt; sidesteps this by loading the reference model in 4-bit and applying LoRA adapters to the trainable copy, dropping the memory cost back to roughly SFT levels.&lt;/p&gt;&#10;&lt;p&gt;A subtler point about the loss: DPO does not push up the absolute likelihood of the chosen response. It pushes up the &lt;em&gt;relative&lt;/em&gt; likelihood of chosen versus rejected. In practice, &lt;a href="https://arxiv.org/abs/2403.07691"&gt;DPO often decreases the absolute likelihood of both responses&lt;/a&gt;; chosen drops a little, rejected drops a lot, the margin goes up, the model &amp;ldquo;wins&amp;rdquo; on win-rate metrics. This is the chosen-probability-drop phenomenon that DPO papers have been documenting since 2024, and it&amp;rsquo;s the single most surprising fact about DPO training to engineers who came from supervised learning. The model is getting better at the &lt;em&gt;comparative&lt;/em&gt; objective; whether it&amp;rsquo;s also getting better at the &lt;em&gt;generative&lt;/em&gt; objective is a separate question.&lt;/p&gt;&#10;&lt;h2 id="choose-a-loss-variant-deliberately"&gt;Choose a loss variant deliberately&lt;/h2&gt;&#10;&lt;p&gt;DPO spawned a family of follow-ups, each fixing one of DPO&amp;rsquo;s documented limitations. The four worth knowing in 2026:&lt;/p&gt;&#10;&lt;p&gt;&lt;a href="https://arxiv.org/abs/2310.12036"&gt;IPO (Identity Preference Optimization, Azar et al., 2023)&lt;/a&gt;. Fixes the overfitting failure mode where DPO&amp;rsquo;s sigmoid loss can drive the model to extreme certainty on a few high-confidence preferences while ignoring the rest. The fix is to replace the sigmoid with a squared loss on the &lt;em&gt;raw&lt;/em&gt; implicit reward margin; the loss is bounded, gradient saturation can&amp;rsquo;t happen, and the model can&amp;rsquo;t trade off everything else for hyper-certain agreement on a handful of pairs. In TRL: &lt;code&gt;loss_type=&amp;quot;ipo&amp;quot;&lt;/code&gt;. Use it when your preference data has a long tail of subtle preferences that DPO would otherwise drown out by over-fitting on the loud pairs.&lt;/p&gt;&#10;&lt;p&gt;&lt;a href="https://arxiv.org/abs/2402.01306"&gt;KTO (Kahneman-Tversky Optimization, Ethayarajh et al., 2024)&lt;/a&gt;. Drops the paired-comparison data shape entirely. Instead of &lt;code&gt;(prompt, chosen, rejected)&lt;/code&gt; triples, KTO trains on &lt;code&gt;(prompt, response, label)&lt;/code&gt; rows where &lt;code&gt;label ∈ {desirable, undesirable}&lt;/code&gt;; exactly the shape of &lt;a href="https://jatinbansal.com/ai-engineering/human-feedback-loops/"&gt;production thumbs-up/down feedback&lt;/a&gt;. The loss is grounded in &lt;a href="https://en.wikipedia.org/wiki/Prospect_theory"&gt;prospect theory&amp;rsquo;s value function&lt;/a&gt;; the asymmetry between gain and loss that Kahneman &amp;amp; Tversky identified; and applies different loss weights to desirable and undesirable signals, mirroring loss aversion. The practical implication is enormous: instead of asking annotators to compare pairs (slow, expensive, hard to get right when both responses are mediocre), you can train directly on the thumbs your users are already leaving. In TRL: &lt;code&gt;KTOTrainer&lt;/code&gt;. Use it when your data is unpaired binary feedback and you have substantially more &amp;ldquo;good&amp;rdquo; than &amp;ldquo;bad&amp;rdquo; rows (or vice versa); KTO handles imbalanced classes naturally where DPO needs careful pairing.&lt;/p&gt;&#10;&lt;p&gt;&lt;a href="https://arxiv.org/abs/2403.07691"&gt;ORPO (Odds Ratio Preference Optimization, Hong et al., 2024)&lt;/a&gt;. Fuses the SFT and DPO stages into a single objective that does both jobs in one pass. The loss is &lt;code&gt;L_ORPO = L_SFT(y_w | x) + λ · L_OR(y_w, y_l | x)&lt;/code&gt;, where the SFT term is standard cross-entropy on the chosen response and the OR term is an odds-ratio loss on the preference. &lt;em&gt;No reference model needed&lt;/em&gt;; ORPO is reference-free, because the SFT term anchors the model to the chosen response directly, and the OR term sharpens the margin against the rejected response. Memory cost is roughly half of DPO (no reference model on the device), training time is roughly half (no second forward pass), and the &lt;a href="https://aclanthology.org/2024.emnlp-main.626/"&gt;original paper showed&lt;/a&gt; ORPO-fine-tuned Phi-2/Llama-2/Mistral outperforming SFT+DPO at smaller scales on AlpacaEval and MT-Bench. Use it when you don&amp;rsquo;t already have an SFT checkpoint and want to do SFT + preference optimization in one pass, particularly when memory is tight. The trade-off: ORPO needs more diverse preference data than DPO to converge well, because it can&amp;rsquo;t lean on a pre-aligned SFT base.&lt;/p&gt;&#10;&lt;p&gt;&lt;a href="https://arxiv.org/abs/2405.14734"&gt;SimPO (Simple Preference Optimization, Meng et al., 2024)&lt;/a&gt;. The newer reference-free alternative. SimPO replaces the log-likelihood ratio against the reference with the &lt;em&gt;average&lt;/em&gt; (length-normalized) log-likelihood of the response itself, and adds a target margin term to widen the gap. The loss is:&lt;/p&gt;&#10;&lt;figure class="nb-code"&gt;&#10; &lt;header class="nb-code-chrome"&gt;&#10; &lt;span class="nb-code-label"&gt;text&lt;/span&gt;&#10; &#10; &lt;/header&gt;&#10; &lt;div class="nb-code-body"&gt;&lt;div class="highlight"&gt;&lt;pre tabindex="0" class="chroma"&gt;&lt;code class="language-text" data-lang="text"&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;L_SimPO = -log σ(β · (avg_log_π(y_w|x) - avg_log_π(y_l|x)) - γ)&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;/div&gt;&#10;&lt;/figure&gt;&#10;&lt;p&gt;where γ is the target margin. Two things to notice. The &lt;em&gt;average&lt;/em&gt; log-likelihood; divided by token count; directly attacks DPO&amp;rsquo;s length-bias problem, because longer responses can&amp;rsquo;t accumulate higher rewards just by being longer. The absence of &lt;code&gt;π_ref&lt;/code&gt; means SimPO&amp;rsquo;s training cost is half of DPO&amp;rsquo;s (one forward pass per batch). &lt;a href="https://proceedings.neurips.cc/paper_files/paper/2024/hash/e099c1c9699814af0be873a175361713-Abstract-Conference.html"&gt;SimPO&amp;rsquo;s NeurIPS 2024 paper&lt;/a&gt; reported a 53.7 length-controlled win rate on AlpacaEval 2 against Llama-3-8B-Instruct; a benchmark that explicitly penalizes the verbosity DPO is biased toward. Use SimPO when length bias is the failure mode you&amp;rsquo;re seeing, or when memory pressure makes the reference model expensive.&lt;/p&gt;&#10;&lt;p&gt;The main variants differ as follows:&lt;/p&gt;&#10;&lt;table&gt;&#10;&#9;&lt;thead&gt;&#10;&#9;&#9;&#9;&lt;tr&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;th&gt;Method&lt;/th&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;th&gt;Reference model?&lt;/th&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;th&gt;Data shape&lt;/th&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;th&gt;Primary fix vs DPO&lt;/th&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;th&gt;TRL trainer&lt;/th&gt;&#10;&#9;&#9;&#9;&lt;/tr&gt;&#10;&#9;&lt;/thead&gt;&#10;&#9;&lt;tbody&gt;&#10;&#9;&#9;&#9;&lt;tr&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;td&gt;DPO&lt;/td&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;td&gt;Required&lt;/td&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;td&gt;Paired &lt;code&gt;(p, w, l)&lt;/code&gt;&lt;/td&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;td&gt;; (baseline)&lt;/td&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;td&gt;&lt;code&gt;DPOTrainer&lt;/code&gt;&lt;/td&gt;&#10;&#9;&#9;&#9;&lt;/tr&gt;&#10;&#9;&#9;&#9;&lt;tr&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;td&gt;IPO&lt;/td&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;td&gt;Required&lt;/td&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;td&gt;Paired&lt;/td&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;td&gt;Bounds the loss, prevents overconfidence&lt;/td&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;td&gt;&lt;code&gt;DPOTrainer(loss_type=&amp;quot;ipo&amp;quot;)&lt;/code&gt;&lt;/td&gt;&#10;&#9;&#9;&#9;&lt;/tr&gt;&#10;&#9;&#9;&#9;&lt;tr&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;td&gt;KTO&lt;/td&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;td&gt;Required&lt;/td&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;td&gt;Unpaired &lt;code&gt;(p, y, label)&lt;/code&gt;&lt;/td&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;td&gt;Removes pairing requirement&lt;/td&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;td&gt;&lt;code&gt;KTOTrainer&lt;/code&gt;&lt;/td&gt;&#10;&#9;&#9;&#9;&lt;/tr&gt;&#10;&#9;&#9;&#9;&lt;tr&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;td&gt;ORPO&lt;/td&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;td&gt;Not needed&lt;/td&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;td&gt;Paired&lt;/td&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;td&gt;Fuses SFT+DPO, reference-free&lt;/td&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;td&gt;&lt;code&gt;ORPOTrainer&lt;/code&gt;&lt;/td&gt;&#10;&#9;&#9;&#9;&lt;/tr&gt;&#10;&#9;&#9;&#9;&lt;tr&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;td&gt;SimPO&lt;/td&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;td&gt;Not needed&lt;/td&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;td&gt;Paired&lt;/td&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;td&gt;Length-normalized, reference-free&lt;/td&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;td&gt;&lt;code&gt;DPOTrainer(loss_type=&amp;quot;simpo&amp;quot;)&lt;/code&gt;&lt;/td&gt;&#10;&#9;&#9;&#9;&lt;/tr&gt;&#10;&#9;&lt;/tbody&gt;&#10;&lt;/table&gt;&#10;&lt;p&gt;The honest 2026 pattern: start with DPO at β=0.1, watch the length and chosen-probability metrics, and switch to SimPO if length bias dominates or to KTO if your data is unpaired. The other variants are situational.&lt;/p&gt;&#10;&lt;h2 id="code-full-dpo-pipeline-in-trl"&gt;Code: full DPO pipeline in TRL&lt;/h2&gt;&#10;&lt;p&gt;The pipeline below assumes you already have an SFT checkpoint (per &lt;a href="https://jatinbansal.com/ai-engineering/pretraining-to-rlhf/"&gt;the training-stage overview&lt;/a&gt;). The setup is &lt;code&gt;pip install &amp;quot;trl&amp;gt;=0.22&amp;quot; &amp;quot;transformers&amp;gt;=4.46&amp;quot; &amp;quot;datasets&amp;gt;=3&amp;quot; peft accelerate bitsandbytes&lt;/code&gt;.&lt;/p&gt;&#10;&lt;figure class="nb-code"&gt;&#10; &lt;header class="nb-code-chrome"&gt;&#10; &lt;span class="nb-code-label"&gt;python&lt;/span&gt;&#10; &#10; &lt;/header&gt;&#10; &lt;div class="nb-code-body"&gt;&lt;div class="highlight"&gt;&lt;div class="chroma"&gt;&#10;&lt;table class="lntable"&gt;&lt;tr&gt;&lt;td class="lntd"&gt;&#10;&lt;pre tabindex="0" class="chroma"&gt;&lt;code&gt;&lt;span class="lnt"&gt; 1&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 2&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 3&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 4&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 5&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 6&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 7&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 8&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 9&#10;&lt;/span&gt;&lt;span class="lnt"&gt;10&#10;&lt;/span&gt;&lt;span class="lnt"&gt;11&#10;&lt;/span&gt;&lt;span class="lnt"&gt;12&#10;&lt;/span&gt;&lt;span class="lnt"&gt;13&#10;&lt;/span&gt;&lt;span class="lnt"&gt;14&#10;&lt;/span&gt;&lt;span class="lnt"&gt;15&#10;&lt;/span&gt;&lt;span class="lnt"&gt;16&#10;&lt;/span&gt;&lt;span class="lnt"&gt;17&#10;&lt;/span&gt;&lt;span class="lnt"&gt;18&#10;&lt;/span&gt;&lt;span class="lnt"&gt;19&#10;&lt;/span&gt;&lt;span class="lnt"&gt;20&#10;&lt;/span&gt;&lt;span class="lnt"&gt;21&#10;&lt;/span&gt;&lt;span class="lnt"&gt;22&#10;&lt;/span&gt;&lt;span class="lnt"&gt;23&#10;&lt;/span&gt;&lt;span class="lnt"&gt;24&#10;&lt;/span&gt;&lt;span class="lnt"&gt;25&#10;&lt;/span&gt;&lt;span class="lnt"&gt;26&#10;&lt;/span&gt;&lt;span class="lnt"&gt;27&#10;&lt;/span&gt;&lt;span class="lnt"&gt;28&#10;&lt;/span&gt;&lt;span class="lnt"&gt;29&#10;&lt;/span&gt;&lt;span class="lnt"&gt;30&#10;&lt;/span&gt;&lt;span class="lnt"&gt;31&#10;&lt;/span&gt;&lt;span class="lnt"&gt;32&#10;&lt;/span&gt;&lt;span class="lnt"&gt;33&#10;&lt;/span&gt;&lt;span class="lnt"&gt;34&#10;&lt;/span&gt;&lt;span class="lnt"&gt;35&#10;&lt;/span&gt;&lt;span class="lnt"&gt;36&#10;&lt;/span&gt;&lt;span class="lnt"&gt;37&#10;&lt;/span&gt;&lt;span class="lnt"&gt;38&#10;&lt;/span&gt;&lt;span class="lnt"&gt;39&#10;&lt;/span&gt;&lt;span class="lnt"&gt;40&#10;&lt;/span&gt;&lt;span class="lnt"&gt;41&#10;&lt;/span&gt;&lt;span class="lnt"&gt;42&#10;&lt;/span&gt;&lt;span class="lnt"&gt;43&#10;&lt;/span&gt;&lt;span class="lnt"&gt;44&#10;&lt;/span&gt;&lt;span class="lnt"&gt;45&#10;&lt;/span&gt;&lt;span class="lnt"&gt;46&#10;&lt;/span&gt;&lt;span class="lnt"&gt;47&#10;&lt;/span&gt;&lt;span class="lnt"&gt;48&#10;&lt;/span&gt;&lt;span class="lnt"&gt;49&#10;&lt;/span&gt;&lt;span class="lnt"&gt;50&#10;&lt;/span&gt;&lt;span class="lnt"&gt;51&#10;&lt;/span&gt;&lt;span class="lnt"&gt;52&#10;&lt;/span&gt;&lt;span class="lnt"&gt;53&#10;&lt;/span&gt;&lt;span class="lnt"&gt;54&#10;&lt;/span&gt;&lt;span class="lnt"&gt;55&#10;&lt;/span&gt;&lt;span class="lnt"&gt;56&#10;&lt;/span&gt;&lt;span class="lnt"&gt;57&#10;&lt;/span&gt;&lt;span class="lnt"&gt;58&#10;&lt;/span&gt;&lt;span class="lnt"&gt;59&#10;&lt;/span&gt;&lt;span class="lnt"&gt;60&#10;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/td&gt;&#10;&lt;td class="lntd"&gt;&#10;&lt;pre tabindex="0" class="chroma"&gt;&lt;code class="language-python" data-lang="python"&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="nn"&gt;torch&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="kn"&gt;from&lt;/span&gt; &lt;span class="nn"&gt;datasets&lt;/span&gt; &lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;load_dataset&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="kn"&gt;from&lt;/span&gt; &lt;span class="nn"&gt;peft&lt;/span&gt; &lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;LoraConfig&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="kn"&gt;from&lt;/span&gt; &lt;span class="nn"&gt;transformers&lt;/span&gt; &lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;AutoModelForCausalLM&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;AutoTokenizer&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="kn"&gt;from&lt;/span&gt; &lt;span class="nn"&gt;trl&lt;/span&gt; &lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;DPOConfig&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;DPOTrainer&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="n"&gt;SFT_MODEL&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s2"&gt;&amp;#34;./sft-qwen&amp;#34;&lt;/span&gt; &lt;span class="c1"&gt;# output of the SFT stage&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="n"&gt;OUT_DIR&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s2"&gt;&amp;#34;./dpo-qwen&amp;#34;&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="n"&gt;tokenizer&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;AutoTokenizer&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;from_pretrained&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;SFT_MODEL&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="n"&gt;model&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;AutoModelForCausalLM&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;from_pretrained&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;SFT_MODEL&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;torch_dtype&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;torch&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;bfloat16&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;device_map&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;#34;auto&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="p"&gt;)&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="c1"&gt;# UltraFeedback is the canonical public DPO dataset: ~60k (prompt, chosen, rejected)&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="c1"&gt;# triples derived from GPT-4 ratings of multi-model responses to a diverse prompt set.&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="n"&gt;ds&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;load_dataset&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;&amp;#34;trl-lib/ultrafeedback_binarized&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;split&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;#34;train&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="c1"&gt;# LoRA cuts memory by ~4x: trainable copy gets adapters, reference stays full-precision.&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="n"&gt;peft_config&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;LoraConfig&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;16&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;lora_alpha&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;32&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;target_modules&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;&amp;#34;q_proj&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;&amp;#34;k_proj&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;&amp;#34;v_proj&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;&amp;#34;o_proj&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;lora_dropout&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mf"&gt;0.05&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;bias&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;#34;none&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;task_type&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;#34;CAUSAL_LM&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="p"&gt;)&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="n"&gt;config&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;DPOConfig&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;output_dir&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;OUT_DIR&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;num_train_epochs&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;per_device_train_batch_size&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;2&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;gradient_accumulation_steps&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;8&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="c1"&gt;# effective batch size 16&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;learning_rate&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mf"&gt;5e-7&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="c1"&gt;# DPO LR is ~50x lower than SFT&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;lr_scheduler_type&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;#34;cosine&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;warmup_ratio&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mf"&gt;0.1&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;beta&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mf"&gt;0.1&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="c1"&gt;# KL strength; canonical default&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;loss_type&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;#34;sigmoid&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="c1"&gt;# vanilla DPO; switch to &amp;#34;ipo&amp;#34; or &amp;#34;simpo&amp;#34; as needed&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;max_length&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;2048&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;max_prompt_length&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;1024&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;logging_steps&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;20&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;save_strategy&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;#34;epoch&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;bf16&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="kc"&gt;True&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="c1"&gt;# length-normalization mitigation if you stay on vanilla DPO:&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="c1"&gt;# rpo_alpha=1.0, # adds SFT-style NLL on chosen to combat chosen-prob drop&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="p"&gt;)&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="n"&gt;trainer&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;DPOTrainer&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;model&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;model&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;args&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;config&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;train_dataset&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;ds&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;processing_class&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;tokenizer&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;peft_config&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;peft_config&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="c1"&gt;# ref_model is auto-loaded from `model` if not specified; with LoRA, the base&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="c1"&gt;# model with adapters disabled serves as the reference, halving the memory cost.&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="p"&gt;)&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="n"&gt;trainer&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;train&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="n"&gt;trainer&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;save_model&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;OUT_DIR&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/td&gt;&lt;/tr&gt;&lt;/table&gt;&#10;&lt;/div&gt;&#10;&lt;/div&gt;&lt;/div&gt;&#10;&lt;/figure&gt;&#10;&lt;p&gt;Operational notes for this scaffold:&lt;/p&gt;&#10;&lt;p&gt;the &lt;code&gt;rpo_alpha&lt;/code&gt; parameter (commented out above) implements &lt;a href="https://arxiv.org/abs/2404.19733"&gt;Pang et al.&amp;rsquo;s iterative reasoning preference optimization (RPO)&lt;/a&gt; mitigation for the chosen-probability-drop problem. It adds a small SFT-style cross-entropy term on the chosen response to the DPO loss, keeping &lt;code&gt;log π(y_w|x)&lt;/code&gt; from collapsing. Set &lt;code&gt;rpo_alpha=1.0&lt;/code&gt; if you&amp;rsquo;re seeing the chosen-probability metric in TRL&amp;rsquo;s training logs dropping below baseline; it&amp;rsquo;s a cheap fix.&lt;/p&gt;&#10;&lt;p&gt;Second, the learning rate (&lt;code&gt;5e-7&lt;/code&gt;) is &lt;em&gt;much&lt;/em&gt; lower than SFT (&lt;code&gt;2e-5&lt;/code&gt;). DPO is a refinement pass; aggressive LRs blow out the alignment. If your loss curve is noisy or diverges, drop LR before changing anything else.&lt;/p&gt;&#10;&lt;p&gt;Third, the &lt;code&gt;processing_class&lt;/code&gt; parameter replaced &lt;code&gt;tokenizer&lt;/code&gt; in TRL&amp;rsquo;s API in &lt;a href="https://github.com/huggingface/trl/releases"&gt;version 0.22&lt;/a&gt;. This is the kind of API-drift breakage that bites pipelines pinned to older docs; if you&amp;rsquo;re following an older tutorial and hit a &lt;code&gt;TypeError&lt;/code&gt;, this is usually why.&lt;/p&gt;&#10;&lt;h2 id="iterate-from-observed-model-failures"&gt;Iterate from observed model failures&lt;/h2&gt;&#10;&lt;p&gt;A single DPO pass on a static preference dataset is the toy version. The production pattern, &lt;a href="https://ai.meta.com/research/publications/the-llama-3-herd-of-models/"&gt;used by Llama 3&amp;rsquo;s post-training&lt;/a&gt; and most open-weight model releases since, is iterative DPO: alternate between sampling responses from the current model, scoring them with a reward model or LLM judge, and running a DPO pass on the resulting fresh preference pairs. Each iteration uses the previous iteration&amp;rsquo;s checkpoint as the reference model. The reason this matters: the preference data in a static dataset was generated against responses from &lt;em&gt;some other model&lt;/em&gt;; not the model you&amp;rsquo;re training. As your model improves, its response distribution drifts away from the responses in the dataset, and the dataset becomes off-policy. Iterative DPO keeps the data on-policy.&lt;/p&gt;&#10;&lt;p&gt;The Llama 3 post-training pipeline used six iterations of this loop. The &lt;a href="https://arxiv.org/abs/2411.15124"&gt;Tülu 3 release from AI2&lt;/a&gt; used a similar shape with public datasets, making it the cleanest open reproduction of the pattern to study. The cost is roughly N× a single DPO run (where N is the number of iterations), plus the inference cost of sampling on-policy responses at each step, plus the labeling cost (whether human or LLM-judge). The benefit is that the final model is markedly better than what a single static pass would have produced; the consistent finding across published iterative-DPO runs is a 5–15% lift on AlpacaEval / Arena-Hard between iteration 1 and iteration 4-6.&lt;/p&gt;&#10;&lt;p&gt;The architecture sketch for iterative DPO is something like:&lt;/p&gt;&#10;&lt;figure class="nb-code"&gt;&#10; &lt;header class="nb-code-chrome"&gt;&#10; &lt;span class="nb-code-label"&gt;text&lt;/span&gt;&#10; &#10; &lt;/header&gt;&#10; &lt;div class="nb-code-body"&gt;&lt;div class="highlight"&gt;&lt;pre tabindex="0" class="chroma"&gt;&lt;code class="language-text" data-lang="text"&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;loop for N iterations:&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; 1. Sample K responses per prompt from the current policy&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; 2. Score them (reward model OR LLM judge OR rule-based verifier)&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; 3. Form (prompt, best, worst) triples — typically top vs bottom of the K samples&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; 4. Run a DPO pass with the current policy as both the trainable model and the reference&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; 5. Replace the current policy with the new checkpoint; goto 1&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;/div&gt;&#10;&lt;/figure&gt;&#10;&lt;p&gt;This is also the place where the &lt;a href="https://jatinbansal.com/ai-engineering/human-feedback-loops/"&gt;feedback queues you built in the Evaluation subtree&lt;/a&gt; start paying off. Production thumbs and edits are &lt;em&gt;on-policy&lt;/em&gt; preference data by construction; they were generated against the model you&amp;rsquo;re actually shipping. A production team running iterative DPO can use their accumulated user feedback as one of the data streams feeding step 2, alongside synthetic on-policy preferences scored by an LLM judge. The data flywheel and the post-training pipeline are the same loop seen from different ends.&lt;/p&gt;&#10;&lt;h2 id="further-reading-from-the-field"&gt;Further reading from the field&lt;/h2&gt;&#10;&lt;ul&gt;&#10;&lt;li&gt;&lt;a href="https://arxiv.org/abs/2305.18290"&gt;Rafailov et al., 2023; &amp;ldquo;Direct Preference Optimization: Your Language Model is Secretly a Reward Model&amp;rdquo;&lt;/a&gt;; the original paper. Section 4 is the derivation, Section 5 is the empirical comparison vs PPO-RLHF. Read this once before you read any of the variants; the variants only make sense as edits to its derivation.&lt;/li&gt;&#10;&lt;li&gt;&lt;a href="https://rlhfbook.com/c/12-direct-alignment.html"&gt;Nathan Lambert; RLHF Book, &amp;ldquo;Direct Alignment Algorithms&amp;rdquo; chapter&lt;/a&gt;; the most comprehensive single-author treatment of DPO and its descendants, with the operational detail (what β actually does, why iterative DPO matters, the chosen-probability-drop phenomenon) you can&amp;rsquo;t get from individual papers.&lt;/li&gt;&#10;&lt;li&gt;&lt;a href="https://www.philschmid.de/rl-with-llms-in-2025-dpo"&gt;Phil Schmid; &amp;ldquo;How to align open LLMs in 2025 with DPO &amp;amp; synthetic data&amp;rdquo;&lt;/a&gt;; the cleanest end-to-end walkthrough of an iterative-DPO pipeline against a real open-weight base, with the data generation, scoring, and training all in one place. Pair with the TRL docs for the API specifics.&lt;/li&gt;&#10;&lt;li&gt;&lt;a href="https://huggingface.co/docs/trl/main/en/dpo_trainer"&gt;Hugging Face; TRL DPO Trainer documentation&lt;/a&gt;; the source of truth for the API surface. Loss types, hyperparameters, the &lt;code&gt;rpo_alpha&lt;/code&gt; and &lt;code&gt;sft_weight&lt;/code&gt; modifiers, the various reference-model strategies. If you&amp;rsquo;re operating a DPO pipeline, this is the page that decides what your loss function actually looks like.&lt;/li&gt;&#10;&lt;/ul&gt;&#10;</content:encoded></item><item><title>From Pre-Training to RLHF</title><link>https://jatinbansal.com/ai-engineering/pretraining-to-rlhf/</link><guid isPermaLink="true">https://jatinbansal.com/ai-engineering/pretraining-to-rlhf/</guid><pubDate>Tue, 26 May 2026 18:45:00 +0000</pubDate><dc:creator>Jatin Bansal</dc:creator><category>AI Engineering</category><description>LLM pretraining, supervised fine-tuning, preference optimization, and verifiable rewards.</description><content:encoded>&lt;p&gt;An API model is the result of several training stages. Pretraining learns statistical structure from a large corpus. Supervised fine-tuning (SFT) teaches the model to produce assistant responses in desired formats. Preference optimization changes the relative likelihood of plausible responses. Some reasoning models add reinforcement learning against verifiable outcomes. Each stage uses different data and a different objective, so a failure introduced in one stage cannot always be repaired by another.&lt;/p&gt;&#10;&lt;h2 id="definition"&gt;Definition&lt;/h2&gt;&#10;&lt;p&gt;The &lt;a href="https://arxiv.org/abs/2203.02155"&gt;InstructGPT pipeline&lt;/a&gt; established a widely used sequence: pretraining, SFT, reward-model training, and reinforcement learning. “Post-training” now covers SFT and the preference or reinforcement stages applied after the base model is trained. Direct preference methods can replace the separate reward model and PPO loop, while verifiable tasks can use deterministic rewards.&lt;/p&gt;&#10;&lt;h2 id="stage-1-pretraining"&gt;Stage 1: Pretraining&lt;/h2&gt;&#10;&lt;p&gt;Pretraining minimizes next-token cross-entropy over a large, cleaned corpus containing text, code, mathematics, dialogue, and other sources. Every token position supplies a training target. The output is a base model that can continue many kinds of text but has not necessarily learned the turn structure or response policy expected from a chat assistant.&lt;/p&gt;&#10;&lt;p&gt;The corpus determines much of the model&amp;rsquo;s factual coverage and linguistic capability. Later training can make existing capability easier to elicit and can teach limited new behavior, but it is an inefficient way to maintain changing factual knowledge. &lt;a href="https://jatinbansal.com/ai-engineering/finetuning-vs-rag/"&gt;Retrieval is usually better suited to knowledge injection&lt;/a&gt;.&lt;/p&gt;&#10;&lt;p&gt;Base-model behavior also follows the distribution of its corpus rather than a product policy. Chat templates, refusal patterns, tool-call formats, and other interface conventions are introduced during post-training. This is why evaluating a base checkpoint with an instruction-tuned prompt can produce misleading results.&lt;/p&gt;&#10;&lt;h2 id="stage-2-supervised-fine-tuning-sft"&gt;Stage 2: Supervised Fine-Tuning (SFT)&lt;/h2&gt;&#10;&lt;p&gt;SFT trains on prompt-and-demonstration examples. The loss is usually masked so only assistant tokens contribute. The model learns the chat template, instruction-response relationship, output formats, and behaviors represented in the demonstrations. Examples may be written by people or &lt;a href="https://jatinbansal.com/ai-engineering/quantization-distillation/"&gt;distilled from a stronger model&lt;/a&gt;.&lt;/p&gt;&#10;&lt;p&gt;SFT imitates demonstrations; it does not directly learn a ranking over several acceptable answers. Its behavior reflects the coverage and consistency of the dataset. Conflicting styles average together, rare cases remain weak, and memorized refusal wording may not generalize to adversarial paraphrases. Preference data addresses ranking more directly, although it introduces its own label-quality problems.&lt;/p&gt;&#10;&lt;p&gt;The minimal SFT loop in &lt;a href="https://huggingface.co/docs/trl"&gt;TRL&lt;/a&gt; looks like this:&lt;/p&gt;&#10;&lt;figure class="nb-code"&gt;&#10; &lt;header class="nb-code-chrome"&gt;&#10; &lt;span class="nb-code-label"&gt;python&lt;/span&gt;&#10; &#10; &lt;/header&gt;&#10; &lt;div class="nb-code-body"&gt;&lt;div class="highlight"&gt;&lt;div class="chroma"&gt;&#10;&lt;table class="lntable"&gt;&lt;tr&gt;&lt;td class="lntd"&gt;&#10;&lt;pre tabindex="0" class="chroma"&gt;&lt;code&gt;&lt;span class="lnt"&gt; 1&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 2&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 3&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 4&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 5&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 6&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 7&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 8&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 9&#10;&lt;/span&gt;&lt;span class="lnt"&gt;10&#10;&lt;/span&gt;&lt;span class="lnt"&gt;11&#10;&lt;/span&gt;&lt;span class="lnt"&gt;12&#10;&lt;/span&gt;&lt;span class="lnt"&gt;13&#10;&lt;/span&gt;&lt;span class="lnt"&gt;14&#10;&lt;/span&gt;&lt;span class="lnt"&gt;15&#10;&lt;/span&gt;&lt;span class="lnt"&gt;16&#10;&lt;/span&gt;&lt;span class="lnt"&gt;17&#10;&lt;/span&gt;&lt;span class="lnt"&gt;18&#10;&lt;/span&gt;&lt;span class="lnt"&gt;19&#10;&lt;/span&gt;&lt;span class="lnt"&gt;20&#10;&lt;/span&gt;&lt;span class="lnt"&gt;21&#10;&lt;/span&gt;&lt;span class="lnt"&gt;22&#10;&lt;/span&gt;&lt;span class="lnt"&gt;23&#10;&lt;/span&gt;&lt;span class="lnt"&gt;24&#10;&lt;/span&gt;&lt;span class="lnt"&gt;25&#10;&lt;/span&gt;&lt;span class="lnt"&gt;26&#10;&lt;/span&gt;&lt;span class="lnt"&gt;27&#10;&lt;/span&gt;&lt;span class="lnt"&gt;28&#10;&lt;/span&gt;&lt;span class="lnt"&gt;29&#10;&lt;/span&gt;&lt;span class="lnt"&gt;30&#10;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/td&gt;&#10;&lt;td class="lntd"&gt;&#10;&lt;pre tabindex="0" class="chroma"&gt;&lt;code class="language-python" data-lang="python"&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="c1"&gt;# pip install &amp;#34;trl&amp;gt;=0.22&amp;#34; &amp;#34;transformers&amp;gt;=4.46&amp;#34; &amp;#34;datasets&amp;gt;=3&amp;#34; accelerate peft&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="kn"&gt;from&lt;/span&gt; &lt;span class="nn"&gt;datasets&lt;/span&gt; &lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;load_dataset&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="kn"&gt;from&lt;/span&gt; &lt;span class="nn"&gt;transformers&lt;/span&gt; &lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;AutoModelForCausalLM&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;AutoTokenizer&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="kn"&gt;from&lt;/span&gt; &lt;span class="nn"&gt;trl&lt;/span&gt; &lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;SFTConfig&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;SFTTrainer&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="n"&gt;MODEL&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s2"&gt;&amp;#34;Qwen/Qwen2.5-0.5B&amp;#34;&lt;/span&gt; &lt;span class="c1"&gt;# small enough to fine-tune on a single GPU&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="n"&gt;model&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;AutoModelForCausalLM&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;from_pretrained&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;MODEL&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="n"&gt;tokenizer&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;AutoTokenizer&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;from_pretrained&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;MODEL&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="c1"&gt;# UltraChat is a public SFT dataset of multi-turn assistant dialogues&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="n"&gt;ds&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;load_dataset&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;&amp;#34;HuggingFaceH4/ultrachat_200k&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;split&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;#34;train_sft[:5000]&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="n"&gt;config&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;SFTConfig&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;output_dir&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;#34;./sft-qwen&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;num_train_epochs&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;per_device_train_batch_size&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;4&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;learning_rate&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mf"&gt;2e-5&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="c1"&gt;# ~100× lower than pretraining&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;logging_steps&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;20&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;max_length&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;1024&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;completion_only_loss&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="kc"&gt;True&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="c1"&gt;# mask out user-turn tokens from the loss&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="p"&gt;)&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="n"&gt;trainer&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;SFTTrainer&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;model&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;model&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;args&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;config&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;train_dataset&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;ds&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;processing_class&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;tokenizer&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="p"&gt;)&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="n"&gt;trainer&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;train&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="n"&gt;trainer&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;save_model&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;&amp;#34;./sft-qwen&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/td&gt;&lt;/tr&gt;&lt;/table&gt;&#10;&lt;/div&gt;&#10;&lt;/div&gt;&lt;/div&gt;&#10;&lt;/figure&gt;&#10;&lt;p&gt;&lt;code&gt;completion_only_loss=True&lt;/code&gt; excludes user tokens from the training objective. Confirm that the selected chat template and dataset expose the assistant-token mask correctly; otherwise a syntactically valid run may optimize the wrong positions. &lt;a href="https://jatinbansal.com/ai-engineering/lora-peft/"&gt;LoRA&lt;/a&gt; can reduce the trainable parameter and optimizer footprint when full-parameter SFT is unnecessary.&lt;/p&gt;&#10;&lt;h2 id="stage-3-preference-optimization"&gt;Stage 3: Preference Optimization&lt;/h2&gt;&#10;&lt;p&gt;Preference optimization trains on judgments between plausible responses. A common record contains &lt;code&gt;(prompt, chosen, rejected)&lt;/code&gt;. The objective increases the relative probability or reward of the chosen response while constraining movement away from the SFT model.&lt;/p&gt;&#10;&lt;h3 id="era-1-20222023-ppo-style-rlhf"&gt;Era 1 (2022–2023): PPO-style RLHF&lt;/h3&gt;&#10;&lt;p&gt;The original &lt;a href="https://arxiv.org/abs/2203.02155"&gt;InstructGPT pipeline&lt;/a&gt; split preference optimization into two sub-stages. train a reward model, a separate transformer head, to score a response as a scalar by minimizing a Bradley-Terry-style loss on preference pairs. treat the language model as a policy and the reward model as the environment, then run proximal policy optimization (PPO) to push the policy toward higher-reward outputs. A KL-divergence penalty against the SFT model limits policy drift. This reference-model anchor reduces reward hacking and mode collapse.&lt;/p&gt;&#10;&lt;p&gt;PPO-style RLHF requires a policy, reference policy, reward model, value model, rollout generation, and an optimizer. The learned reward is an imperfect proxy, so the policy can exploit reward-model errors. KL regularization, held-out human evaluation, and inspection of high-reward samples are part of the training loop rather than optional checks.&lt;/p&gt;&#10;&lt;h3 id="era-2-20242025-dpo-and-direct-preference-methods"&gt;Era 2 (2024–2025): DPO and direct preference methods&lt;/h3&gt;&#10;&lt;p&gt;&lt;a href="https://arxiv.org/abs/2305.18290"&gt;Direct Preference Optimization (Rafailov et al., 2023)&lt;/a&gt; expresses the preference objective as a supervised loss under the Bradley-Terry model. It avoids a separately trained reward model, PPO sampling loop, and value model. Training still compares the policy with a frozen reference model for chosen and rejected responses.&lt;/p&gt;&#10;&lt;p&gt;&lt;a href="https://jatinbansal.com/ai-engineering/dpo-and-alignment/"&gt;Direct Preference Optimization (DPO)&lt;/a&gt; avoids explicit reward-model training and online PPO rollouts. It is often easier to operate, but still depends on a reference model, preference quality, and the strength of the regularization. &lt;a href="https://arxiv.org/abs/2310.12036"&gt;IPO&lt;/a&gt; modifies the objective to limit overfitting to separable preferences. &lt;a href="https://arxiv.org/abs/2402.01306"&gt;KTO&lt;/a&gt; can use unpaired desirable and undesirable examples, which fits some &lt;a href="https://jatinbansal.com/ai-engineering/human-feedback-loops/"&gt;production-feedback&lt;/a&gt; datasets better than paired comparisons.&lt;/p&gt;&#10;&lt;p&gt;The minimal DPO loop in TRL:&lt;/p&gt;&#10;&lt;figure class="nb-code"&gt;&#10; &lt;header class="nb-code-chrome"&gt;&#10; &lt;span class="nb-code-label"&gt;python&lt;/span&gt;&#10; &#10; &lt;/header&gt;&#10; &lt;div class="nb-code-body"&gt;&lt;div class="highlight"&gt;&lt;div class="chroma"&gt;&#10;&lt;table class="lntable"&gt;&lt;tr&gt;&lt;td class="lntd"&gt;&#10;&lt;pre tabindex="0" class="chroma"&gt;&lt;code&gt;&lt;span class="lnt"&gt; 1&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 2&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 3&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 4&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 5&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 6&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 7&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 8&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 9&#10;&lt;/span&gt;&lt;span class="lnt"&gt;10&#10;&lt;/span&gt;&lt;span class="lnt"&gt;11&#10;&lt;/span&gt;&lt;span class="lnt"&gt;12&#10;&lt;/span&gt;&lt;span class="lnt"&gt;13&#10;&lt;/span&gt;&lt;span class="lnt"&gt;14&#10;&lt;/span&gt;&lt;span class="lnt"&gt;15&#10;&lt;/span&gt;&lt;span class="lnt"&gt;16&#10;&lt;/span&gt;&lt;span class="lnt"&gt;17&#10;&lt;/span&gt;&lt;span class="lnt"&gt;18&#10;&lt;/span&gt;&lt;span class="lnt"&gt;19&#10;&lt;/span&gt;&lt;span class="lnt"&gt;20&#10;&lt;/span&gt;&lt;span class="lnt"&gt;21&#10;&lt;/span&gt;&lt;span class="lnt"&gt;22&#10;&lt;/span&gt;&lt;span class="lnt"&gt;23&#10;&lt;/span&gt;&lt;span class="lnt"&gt;24&#10;&lt;/span&gt;&lt;span class="lnt"&gt;25&#10;&lt;/span&gt;&lt;span class="lnt"&gt;26&#10;&lt;/span&gt;&lt;span class="lnt"&gt;27&#10;&lt;/span&gt;&lt;span class="lnt"&gt;28&#10;&lt;/span&gt;&lt;span class="lnt"&gt;29&#10;&lt;/span&gt;&lt;span class="lnt"&gt;30&#10;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/td&gt;&#10;&lt;td class="lntd"&gt;&#10;&lt;pre tabindex="0" class="chroma"&gt;&lt;code class="language-python" data-lang="python"&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="c1"&gt;# pip install &amp;#34;trl&amp;gt;=0.22&amp;#34;&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="kn"&gt;from&lt;/span&gt; &lt;span class="nn"&gt;datasets&lt;/span&gt; &lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;load_dataset&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="kn"&gt;from&lt;/span&gt; &lt;span class="nn"&gt;transformers&lt;/span&gt; &lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;AutoModelForCausalLM&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;AutoTokenizer&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="kn"&gt;from&lt;/span&gt; &lt;span class="nn"&gt;trl&lt;/span&gt; &lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;DPOConfig&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;DPOTrainer&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="n"&gt;MODEL&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s2"&gt;&amp;#34;./sft-qwen&amp;#34;&lt;/span&gt; &lt;span class="c1"&gt;# the output of the SFT stage above&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="n"&gt;model&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;AutoModelForCausalLM&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;from_pretrained&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;MODEL&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="n"&gt;tokenizer&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;AutoTokenizer&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;from_pretrained&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;MODEL&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="c1"&gt;# UltraFeedback is a public preference dataset of (prompt, chosen, rejected) triples&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="n"&gt;ds&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;load_dataset&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;&amp;#34;trl-lib/ultrafeedback_binarized&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;split&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;#34;train[:5000]&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="n"&gt;config&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;DPOConfig&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;output_dir&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;#34;./dpo-qwen&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;num_train_epochs&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;per_device_train_batch_size&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;2&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;learning_rate&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mf"&gt;5e-7&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="c1"&gt;# DPO uses lower LRs than SFT&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;beta&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mf"&gt;0.1&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="c1"&gt;# KL strength; 0.1 is the canonical default&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;logging_steps&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;20&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="p"&gt;)&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="n"&gt;trainer&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;DPOTrainer&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;model&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;model&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;args&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;config&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;train_dataset&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;ds&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;processing_class&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;tokenizer&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="c1"&gt;# ref_model is auto-loaded from MODEL if not specified&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="p"&gt;)&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="n"&gt;trainer&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;train&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="n"&gt;trainer&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;save_model&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;&amp;#34;./dpo-qwen&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/td&gt;&lt;/tr&gt;&lt;/table&gt;&#10;&lt;/div&gt;&#10;&lt;/div&gt;&lt;/div&gt;&#10;&lt;/figure&gt;&#10;&lt;p&gt;&lt;code&gt;beta&lt;/code&gt; controls the preference-versus-reference trade-off under TRL&amp;rsquo;s DPO convention. Treat &lt;code&gt;0.1&lt;/code&gt; as an experiment starting point, then tune against held-out preference accuracy and general-capability evaluations. The meaning of the direction can differ across formulations and libraries, so verify the implementation rather than relying on a remembered rule.&lt;/p&gt;&#10;&lt;h3 id="era-3-20252026-rlaif-and-rl-with-verifiable-rewards"&gt;Era 3 (2025–2026): RLAIF and RL with verifiable rewards&lt;/h3&gt;&#10;&lt;p&gt;&lt;a href="https://arxiv.org/abs/2212.08073"&gt;Reinforcement Learning from AI Feedback (RLAIF)&lt;/a&gt; uses model-generated preference labels, sometimes against a written constitution. It can expand a dataset cheaply, but the judge&amp;rsquo;s systematic errors become training signal. Human calibration sets and disagreement sampling help detect this transfer.&lt;/p&gt;&#10;&lt;p&gt;Reinforcement Learning with Verifiable Rewards (RLVR) uses outcomes such as unit tests, exact mathematical answers, or schema validation. &lt;a href="https://arxiv.org/abs/2501.12948"&gt;DeepSeek-R1&lt;/a&gt; used &lt;a href="https://arxiv.org/abs/2402.03300"&gt;GRPO&lt;/a&gt;, which estimates advantage from groups of sampled responses without a separate critic. Verifiable rewards are strongest where the checker measures the full task; a weak checker invites reward hacking just as a learned reward model does.&lt;/p&gt;&#10;&lt;p&gt;These methods can be composed, but the order is a design choice. Preserve a checkpoint and evaluation suite at each boundary. A later stage can improve its target metric while reducing instruction following, calibration, safety behavior, or domain performance learned earlier.&lt;/p&gt;&#10;&lt;h2 id="data-and-evaluation-boundaries"&gt;Data and evaluation boundaries&lt;/h2&gt;&#10;&lt;p&gt;Each stage needs a separate data contract. Pretraining examples are token sequences with document-level provenance and filtering metadata. SFT examples need a chat template, an explicit assistant mask, and a policy for multi-turn conversations. Preference records need the prompt, both responses, the judgment, the label source, and enough metadata to detect repeated annotators or model-generated duplicates. Verifiable-reward records also need a checker version because changing a unit test changes the learning objective.&lt;/p&gt;&#10;&lt;p&gt;Split data by source or task family before generating near-duplicates. A random row split can put paraphrases, adjacent document chunks, or several sampled responses to the same prompt on both sides of the boundary. The resulting validation loss looks better without measuring generalization. Deduplicate across train and evaluation sets, and keep a small contamination audit for public benchmarks.&lt;/p&gt;&#10;&lt;p&gt;Evaluate the checkpoint produced by every stage. A base model needs language-model loss and capability benchmarks. SFT adds instruction following, format validity, and turn-taking. Preference optimization adds pairwise preference win rate, but that metric must be accompanied by general-capability, calibration, safety, and style checks. RLVR needs held-out problems and adversarial tests of the verifier. The training reward is unsuitable as the only evaluation because the optimizer was trained to increase it.&lt;/p&gt;&#10;&lt;p&gt;Compare against the previous checkpoint with paired prompts and fixed decoding settings. Where sampling is part of the product, repeat generations under several seeds and report uncertainty. Keep examples of regressions, not only aggregate scores; a small average gain can conceal a severe loss in one domain.&lt;/p&gt;&#10;&lt;h2 id="common-post-training-failures"&gt;Common post-training failures&lt;/h2&gt;&#10;&lt;p&gt;Reward hacking occurs when the model discovers an output that scores well without satisfying the intended task. It can exploit a learned judge, a parser, or an incomplete unit test. Inspect the highest-reward samples and strengthen the evaluator before increasing optimization pressure.&lt;/p&gt;&#10;&lt;p&gt;Mode narrowing appears when preference training repeatedly rewards one style or response pattern. Diversity, calibration, and performance on neutral prompts may decline even while preference accuracy rises. A reference-model constraint limits movement but does not replace broad evaluation.&lt;/p&gt;&#10;&lt;p&gt;Catastrophic forgetting is most likely when the new dataset is narrow or the learning rate is too high. Mix replay examples from general capabilities, use conservative updates, and stop on held-out regressions. If only a small behavior change is required, &lt;a href="https://jatinbansal.com/ai-engineering/lora-peft/"&gt;parameter-efficient fine-tuning&lt;/a&gt; can simplify rollback and comparison, although it does not eliminate forgetting inside the adapter&amp;rsquo;s effect.&lt;/p&gt;&#10;&lt;p&gt;Label bias is a data problem rather than an optimizer problem. Annotator instructions, judge prompts, sampling temperature, and response ordering can all alter preferences. Measure agreement, randomize presentation order, retain disagreement, and avoid converting uncertain judgments into confident binary labels.&lt;/p&gt;&#10;&lt;h2 id="further-reading-from-the-field"&gt;Further reading from the field&lt;/h2&gt;&#10;&lt;ul&gt;&#10;&lt;li&gt;&lt;a href="https://arxiv.org/abs/2203.02155"&gt;Training language models to follow instructions with human feedback&lt;/a&gt;&lt;/li&gt;&#10;&lt;li&gt;&lt;a href="https://rlhfbook.com/"&gt;RLHF Book&lt;/a&gt;&lt;/li&gt;&#10;&lt;li&gt;&lt;a href="https://arxiv.org/abs/2305.18290"&gt;Direct Preference Optimization&lt;/a&gt;&lt;/li&gt;&#10;&lt;li&gt;&lt;a href="https://magazine.sebastianraschka.com/p/llm-training-rlhf-and-its-alternatives"&gt;LLM Training: RLHF and Its Alternatives&lt;/a&gt;&lt;/li&gt;&#10;&lt;li&gt;&lt;a href="https://magazine.sebastianraschka.com/p/the-state-of-llm-reasoning-model-training"&gt;The State of Reinforcement Learning for LLM Reasoning&lt;/a&gt;&lt;/li&gt;&#10;&lt;/ul&gt;&#10;</content:encoded></item><item><title>Agent Budgets and Runaway Prevention</title><link>https://jatinbansal.com/ai-engineering/agent-budgets-and-runaway-prevention/</link><guid isPermaLink="true">https://jatinbansal.com/ai-engineering/agent-budgets-and-runaway-prevention/</guid><pubDate>Tue, 26 May 2026 18:34:20 +0000</pubDate><dc:creator>Jatin Bansal</dc:creator><category>AI Engineering</category><description>How agent harnesses enforce step, time, token, cost, and tool-call limits.</description><content:encoded>&lt;p&gt;In November 2025 a team running &lt;a href="https://github.com/vectara/awesome-agent-failures/blob/main/docs/case-studies/langchain-a2a-47k-infinite-loop.md"&gt;four LangChain agents&lt;/a&gt; watched an Analyzer and a Verifier fall into mutual recursion. The Analyzer produced an analysis; the Verifier asked for further analysis; the Analyzer obliged; eleven days later the bill was $47,000. The team had observability. The dashboard &lt;em&gt;showed&lt;/em&gt; the spend climbing, but no enforcement. Alerts are asynchronous: by the time the page fires, every API call between the alert and the human reading it has already happened, and on a long weekend that gap is the entire damage window. A second account from the same year: &lt;a href="https://leanopstech.com/blog/agentic-ai-cost-runaway-token-budget-2026/"&gt;a 35-engineer SaaS shop with an $87,000 April 2026 bill&lt;/a&gt;, one developer&amp;rsquo;s autonomous refactoring weekend burning $4,200. None of these incidents involved a malicious agent. They were healthy agents running healthy loops, until they weren&amp;rsquo;t, and the only thing standing between healthy and catastrophic was a budget the harness either had or didn&amp;rsquo;t.&lt;/p&gt;&#10;&lt;h2 id="budgets-must-stop-the-next-action"&gt;Budgets must stop the next action&lt;/h2&gt;&#10;&lt;p&gt;An agent budget is the harness-enforced set of preconditions checked before every step that, when any one fails, terminates the run with persisted partial state; not after, not during, &lt;em&gt;before&lt;/em&gt; the next side effect. Three properties distinguish a &lt;em&gt;budget&lt;/em&gt; from observability or rate-limiting. It is &lt;em&gt;enforced inside the request path&lt;/em&gt;, not by a sidecar or a cron job. The check happens synchronously between the loop&amp;rsquo;s iteration and the next provider call. It is &lt;em&gt;the disjunction of multiple independent predicates&lt;/em&gt;, not a single number; step caps, wall-clock deadlines, token ceilings, dollar caps, tool quotas, oscillation detectors, and external abort signals all fire on the same conditional. And it &lt;em&gt;terminates with persisted partial state&lt;/em&gt;, not silently. The aborted-with-partial-results outcome is a first-class success state the way &lt;a href="https://jatinbansal.com/ai-engineering/long-horizon-reliability/"&gt;the long-horizon-reliability article&lt;/a&gt; made explicit, not a failure to be retried.&lt;/p&gt;&#10;&lt;p&gt;Alerts are not enforcement. The $47K incident grew during the gap between the alert and the session shutdown. Observability platforms such as &lt;a href="https://langfuse.com/"&gt;Langfuse&lt;/a&gt;, &lt;a href="https://www.langchain.com/langsmith"&gt;LangSmith&lt;/a&gt;, and &lt;a href="https://github.com/Arize-ai/phoenix"&gt;Phoenix&lt;/a&gt; report token counts and costs, but do not stop the next call by default. Tracing records the failure; a budget gate prevents it.&lt;/p&gt;&#10;&lt;h2 id="enforcement-primitives"&gt;Enforcement primitives&lt;/h2&gt;&#10;&lt;p&gt;Every defensible agent budget is the disjunction of &lt;em&gt;at least&lt;/em&gt; the following predicates. A harness that ships only a step cap has shipped 14% of the surface; the rest of this section walks the remaining 86%.&lt;/p&gt;&#10;&lt;h3 id="1-step-cap"&gt;1. Step cap&lt;/h3&gt;&#10;&lt;p&gt;The simplest predicate: iteration count exceeds N. Every framework ships this with a default; Vercel AI SDK&amp;rsquo;s &lt;code&gt;stepCountIs(20)&lt;/code&gt; &lt;a href="https://ai-sdk.dev/docs/agents/loop-control"&gt;is the documented default&lt;/a&gt;, the &lt;a href="https://openai.github.io/openai-agents-python/ref/run/"&gt;OpenAI Agents SDK&amp;rsquo;s &lt;code&gt;maxTurns&lt;/code&gt; defaults to 10 and raises &lt;code&gt;MaxTurnsExceededError&lt;/code&gt;&lt;/a&gt; when exceeded, &lt;a href="https://docs.langchain.com/oss/python/langgraph/errors/GRAPH_RECURSION_LIMIT"&gt;LangGraph&amp;rsquo;s &lt;code&gt;recursion_limit&lt;/code&gt; defaults to 25&lt;/a&gt; and raises &lt;code&gt;GraphRecursionError&lt;/code&gt; on breach. Each is a wrapper over the same &lt;code&gt;if step &amp;gt;= cap: abort&lt;/code&gt; check.&lt;/p&gt;&#10;&lt;p&gt;The trap is treating the default as critical. The Vercel default of 20, OpenAI&amp;rsquo;s 10, LangGraph&amp;rsquo;s 25; these are calibration choices the framework author made for their median user, not for your workload. A research agent that needs 80 turns will hit &lt;code&gt;MaxTurnsExceededError&lt;/code&gt; long before completing a real task; a customer-service agent with a step cap of 80 will burn a quarter of an hour on the wrong path before the cap fires. Tune the step cap to your workload&amp;rsquo;s 95th-percentile completion turn count plus a safety margin. Run a sample of real tasks, observe the distribution, set the cap at p95 + 20%. The cap is the fork-bomb backstop, not the typical termination point. The typical termination is &lt;code&gt;stop_reason != &amp;quot;tool_use&amp;quot;&lt;/code&gt; (the model itself stopping). If the cap is firing often, the cap is wrong or the model is stuck and you need finer-grained detection (sections 6 and 7 below).&lt;/p&gt;&#10;&lt;h3 id="2-wall-clock-deadline"&gt;2. Wall-clock deadline&lt;/h3&gt;&#10;&lt;p&gt;The next predicate: &lt;code&gt;time.monotonic() - started &amp;gt; deadline_s&lt;/code&gt;. Step cap is the count limit; deadline is the time limit, and they fire under different failure modes. A model that makes one slow tool call per step can reach step 5 and have burned 60 seconds; a model that makes a hundred fast calls can stay under a 60-second wall-clock but exceed a 50-step cap. Both deserve to terminate.&lt;/p&gt;&#10;&lt;p&gt;The discipline that matters: the deadline is per-run, not per-call. A 60-second per-call deadline on a 50-step run gives the agent 50 × 60 = 3000 seconds; fifty minutes; before either limit fires. The right shape is &lt;code&gt;total_deadline = 60s; per_call_deadline = min(remaining_budget, max_per_call)&lt;/code&gt;. The remaining budget shrinks monotonically; the per-call deadline tightens as the run ages, so the last step gets less wall-clock than the first. This is the same shape as &lt;a href="https://grpc.io/blog/deadlines/"&gt;gRPC&amp;rsquo;s deadline propagation&lt;/a&gt;: the upstream&amp;rsquo;s deadline is the ceiling on every downstream call, with attenuation for the time already consumed.&lt;/p&gt;&#10;&lt;p&gt;The wall-clock deadline interacts with the streaming surface from &lt;a href="https://jatinbansal.com/ai-engineering/streaming/"&gt;the streaming article&lt;/a&gt;. Cancellation must (a) close the upstream connection so the provider stops decoding (and billing), (b) abort in-flight tool executions whose results will never be used, (c) flush partial state to the conversation log. A deadline that fires but doesn&amp;rsquo;t propagate cancellation is the worst of both worlds. The run &amp;ldquo;stops&amp;rdquo; by Python&amp;rsquo;s metric, but the provider is still decoding the last response and the bill keeps climbing.&lt;/p&gt;&#10;&lt;h3 id="3-token-ceiling"&gt;3. Token ceiling&lt;/h3&gt;&#10;&lt;p&gt;The third predicate: cumulative input + output tokens exceed &lt;code&gt;max_tokens_total&lt;/code&gt;. Tokens are the proxy for both cost and context blow-up. The trap from the LeanOps incident analysis: &lt;a href="https://leanopstech.com/blog/agentic-ai-cost-runaway-token-budget-2026/"&gt;a 4,000-token initial context doubling at each step&lt;/a&gt; reaches 128K at step 5 and overflows the model&amp;rsquo;s window by step 15. A token ceiling that fires at 200K total caps the worst case at the cost of two-and-a-half full-context API calls; meaningful in dollar terms, decisive in fork-bomb terms.&lt;/p&gt;&#10;&lt;p&gt;Token accounting must break down by cache state, per the &lt;a href="https://jatinbansal.com/ai-engineering/prompt-caching/"&gt;prompt-caching article&lt;/a&gt; and the &lt;a href="https://jatinbansal.com/ai-engineering/agent-harness-anatomy/"&gt;agent harness anatomy article&lt;/a&gt;. A harness that aggregates &amp;ldquo;tokens used&amp;rdquo; without separating &lt;code&gt;cache_read_input_tokens&lt;/code&gt; from &lt;code&gt;cache_creation_input_tokens&lt;/code&gt; will report a number that bears no relationship to the bill. The right shape:&lt;/p&gt;&#10;&lt;figure class="nb-code"&gt;&#10; &lt;header class="nb-code-chrome"&gt;&#10; &lt;span class="nb-code-label"&gt;text&lt;/span&gt;&#10; &#10; &lt;/header&gt;&#10; &lt;div class="nb-code-body"&gt;&lt;div class="highlight"&gt;&lt;pre tabindex="0" class="chroma"&gt;&lt;code class="language-text" data-lang="text"&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;tokens_used = input_tokens + output_tokens + cache_read + cache_write&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;dollars_used = (input × price_in + output × price_out&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; + cache_read × price_cache_read + cache_write × price_cache_write) / 1e6&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;/div&gt;&#10;&lt;/figure&gt;&#10;&lt;p&gt;The dollar conversion needs per-provider, per-model, per-cache-tier pricing, and pricing changes quarterly. Pin the pricing-table version into the budget gate&amp;rsquo;s audit log (the same &lt;code&gt;cost.pricing_version&lt;/code&gt; attribute &lt;a href="https://jatinbansal.com/ai-engineering/llm-observability/"&gt;the observability article&lt;/a&gt; recommended for spans) so historical comparisons stay coherent across rate changes.&lt;/p&gt;&#10;&lt;h3 id="4-dollar-ceiling"&gt;4. Dollar ceiling&lt;/h3&gt;&#10;&lt;p&gt;The fourth predicate: cumulative dollar cost exceeds &lt;code&gt;max_dollars&lt;/code&gt;. Dollars and tokens diverge because the dollar/token ratio depends on cache hit rate, model tier, and the input/output split; see &lt;a href="https://jatinbansal.com/ai-engineering/model-routing/"&gt;the cost-optimization and model-routing article&lt;/a&gt; for the full math. A token ceiling alone doesn&amp;rsquo;t catch the cost runaway that happens when a &lt;a href="https://jatinbansal.com/ai-engineering/model-routing/"&gt;model-routing decision&lt;/a&gt; escalates the run from Haiku to Opus mid-loop; a dollar ceiling alone doesn&amp;rsquo;t catch the &lt;em&gt;context&lt;/em&gt;-bloat runaway from a token explosion at constant cost-per-token. Run both.&lt;/p&gt;&#10;&lt;p&gt;The defensible production cut: per-run dollar ceiling + per-tenant daily dollar ceiling + per-tenant monthly dollar ceiling. The per-run cap stops the $47K incident at $50 instead of $47,000. The per-tenant daily cap stops the &lt;a href="https://leanopstech.com/blog/agentic-ai-cost-runaway-token-budget-2026/"&gt;legitimate-but-runaway weekend pattern&lt;/a&gt;; one developer, $4,200, three days; at $500 instead of $4,200. The per-tenant monthly cap is the financial backstop: the line beyond which the platform team gets paged regardless of cause. These compose; the run aborts when &lt;em&gt;any&lt;/em&gt; of them fires. The accounting state lives in Redis or a similar low-latency store keyed by tenant and time window; the budget gate reads the counter before every call.&lt;/p&gt;&#10;&lt;h3 id="5-per-tool-quota"&gt;5. Per-tool quota&lt;/h3&gt;&#10;&lt;p&gt;The fifth predicate: calls to tool T this run exceed &lt;code&gt;max_calls_per_tool[T]&lt;/code&gt;. A model that calls &lt;code&gt;search_web&lt;/code&gt; 200 times in 10 steps is doing something wrong; either it&amp;rsquo;s stuck (no-progress detection should fire) or it&amp;rsquo;s interpreting the task wrong (the cap should fire as the backstop). Per-tool quotas are the agent equivalent of &lt;code&gt;RLIMIT_NOFILE&lt;/code&gt;: a single tool monopolizing the runtime is itself a failure mode, and the per-tool budget is independent of the global step cap.&lt;/p&gt;&#10;&lt;p&gt;The right granularity is by tool &lt;em&gt;class&lt;/em&gt;, not always by tool &lt;em&gt;name&lt;/em&gt;. Mutating tools (&lt;code&gt;charge_card&lt;/code&gt;, &lt;code&gt;send_email&lt;/code&gt;, &lt;code&gt;delete_record&lt;/code&gt;) get tight per-run caps measured in single digits. Read-only tools (&lt;code&gt;search_web&lt;/code&gt;, &lt;code&gt;read_file&lt;/code&gt;, &lt;code&gt;query_db&lt;/code&gt;) get higher caps measured in dozens. The &lt;a href="https://jatinbansal.com/ai-engineering/tool-selection-at-scale/"&gt;tool-selection-at-scale article&lt;/a&gt; covered the namespace discipline; per-tool quotas inherit from the same namespace tree: &lt;code&gt;tools/mutating/*&lt;/code&gt; shares one bucket, &lt;code&gt;tools/read/*&lt;/code&gt; shares another. Tool-class quotas are easier to keep consistent across deploys than per-tool caps that drift as the tool catalog grows.&lt;/p&gt;&#10;&lt;h3 id="6-no-progress-detection"&gt;6. No-progress detection&lt;/h3&gt;&#10;&lt;p&gt;The sixth predicate: the last K tool calls were identical. The simplest no-progress detector hashes &lt;code&gt;(tool_name, sorted_args)&lt;/code&gt; and looks for repeats in a sliding window; three identical calls in a row is decisive evidence the model is stuck, and no number of additional steps will help. The &lt;a href="https://jatinbansal.com/ai-engineering/agent-loop/"&gt;agent loop article&lt;/a&gt; showed the dumb-but-effective Python:&lt;/p&gt;&#10;&lt;figure class="nb-code"&gt;&#10; &lt;header class="nb-code-chrome"&gt;&#10; &lt;span class="nb-code-label"&gt;python&lt;/span&gt;&#10; &#10; &lt;/header&gt;&#10; &lt;div class="nb-code-body"&gt;&lt;div class="highlight"&gt;&lt;pre tabindex="0" class="chroma"&gt;&lt;code class="language-python" data-lang="python"&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="n"&gt;last_calls&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;append&lt;/span&gt;&lt;span class="p"&gt;((&lt;/span&gt;&lt;span class="n"&gt;tool_name&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;json&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;dumps&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;args&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;sort_keys&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="kc"&gt;True&lt;/span&gt;&lt;span class="p"&gt;)))&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;last_calls&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="mi"&gt;3&lt;/span&gt;&lt;span class="p"&gt;:]&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;count&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;last_calls&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;])&lt;/span&gt; &lt;span class="o"&gt;&amp;gt;=&lt;/span&gt; &lt;span class="mi"&gt;3&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="k"&gt;raise&lt;/span&gt; &lt;span class="ne"&gt;RuntimeError&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;&amp;#34;no-progress: 3x repeat&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;/div&gt;&#10;&lt;/figure&gt;&#10;&lt;p&gt;That&amp;rsquo;s the floor. The ceiling is &lt;em&gt;oscillation&lt;/em&gt; detection. The model alternating between two states without making progress, which the dumb detector misses because no single call repeats three times in a row. The Analyzer/Verifier pattern from the $47K incident is exactly this: A → V → A → V → A → V, where no individual call repeats but the &lt;em&gt;pair&lt;/em&gt; does, indefinitely. The fix is to hash the call signature &lt;em&gt;plus the previous call&amp;rsquo;s signature&lt;/em&gt; and look for repeated pairs:&lt;/p&gt;&#10;&lt;figure class="nb-code"&gt;&#10; &lt;header class="nb-code-chrome"&gt;&#10; &lt;span class="nb-code-label"&gt;python&lt;/span&gt;&#10; &#10; &lt;/header&gt;&#10; &lt;div class="nb-code-body"&gt;&lt;div class="highlight"&gt;&lt;pre tabindex="0" class="chroma"&gt;&lt;code class="language-python" data-lang="python"&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;is_oscillating&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;history&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;window&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;6&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="nb"&gt;len&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;history&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;&amp;lt;&lt;/span&gt; &lt;span class="n"&gt;window&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="kc"&gt;False&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;pairs&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nb"&gt;list&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nb"&gt;zip&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;history&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;window&lt;/span&gt;&lt;span class="p"&gt;::&lt;/span&gt;&lt;span class="mi"&gt;2&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt; &lt;span class="n"&gt;history&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;window&lt;/span&gt;&lt;span class="o"&gt;+&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;::&lt;/span&gt;&lt;span class="mi"&gt;2&lt;/span&gt;&lt;span class="p"&gt;]))&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="nb"&gt;len&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nb"&gt;set&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;pairs&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;/div&gt;&#10;&lt;/figure&gt;&#10;&lt;p&gt;Three repeated pairs in six steps is oscillation; six identical alternating calls is the Analyzer/Verifier shape exactly. Production no-progress detectors typically run both predicates; single-call repetition for the dumb-stuck case, pair-repetition for the alternation case, and a more sophisticated entropy-based detector for the long-horizon meltdown case that &lt;a href="https://jatinbansal.com/ai-engineering/long-horizon-reliability/"&gt;the long-horizon reliability article&lt;/a&gt; covered.&lt;/p&gt;&#10;&lt;h3 id="7-external-abort-signal"&gt;7. External abort signal&lt;/h3&gt;&#10;&lt;p&gt;The seventh predicate: an external signal sets &lt;code&gt;aborted = true&lt;/code&gt;. The other six predicates are reactive; they fire on state the loop owns. The external signal is for state the loop doesn&amp;rsquo;t own: an operator dashboard hitting a kill switch, an SLO alert firing on a different service, a budget breach on a &lt;em&gt;different&lt;/em&gt; run by the same tenant. The &lt;a href="https://jatinbansal.com/ai-engineering/long-horizon-reliability/"&gt;Temporal example from the long-horizon reliability article&lt;/a&gt; is exactly this pattern; &lt;code&gt;setHandler(abortSignal, () =&amp;gt; { aborted = true; })&lt;/code&gt; plus a &lt;code&gt;while (!aborted)&lt;/code&gt; check in the loop body. The external signal is the operational pressure relief valve: when the budget gates are calibrated wrong and the run is climbing toward a dollar ceiling that&amp;rsquo;s higher than it should be, the operator&amp;rsquo;s kill switch is the last line of defense.&lt;/p&gt;&#10;&lt;p&gt;The signal must be &lt;em&gt;checked at every iteration&lt;/em&gt;, not just at &amp;ldquo;natural&amp;rdquo; boundaries. A loop body that does five tool calls in parallel between checking the abort signal is five tool calls past the operator&amp;rsquo;s intent. The same shape as a thread that doesn&amp;rsquo;t check a cancellation token between synchronous operations. The discipline is to interleave the signal check with every step.&lt;/p&gt;&#10;&lt;h2 id="check-cheap-limits-first"&gt;Check cheap limits first&lt;/h2&gt;&#10;&lt;p&gt;The seven predicates are a disjunction; any one firing aborts the run, but the &lt;em&gt;order&lt;/em&gt; in which they&amp;rsquo;re evaluated determines which one gets credited for the abort, and that matters for the audit log. The right order is cheap-and-decisive first:&lt;/p&gt;&#10;&lt;ol&gt;&#10;&lt;li&gt;External abort signal (cheap, decisive)&lt;/li&gt;&#10;&lt;li&gt;Step cap (cheap, decisive)&lt;/li&gt;&#10;&lt;li&gt;Wall-clock deadline (cheap, decisive)&lt;/li&gt;&#10;&lt;li&gt;Dollar ceiling (cheap once usage is tracked, decisive)&lt;/li&gt;&#10;&lt;li&gt;Token ceiling (cheap, decisive; typically redundant with dollar but cheaper to compute)&lt;/li&gt;&#10;&lt;li&gt;Per-tool quota (cheap, per-tool dictionary lookup)&lt;/li&gt;&#10;&lt;li&gt;No-progress detection (cheapest is &lt;code&gt;O(K)&lt;/code&gt; hash lookup over history)&lt;/li&gt;&#10;&lt;/ol&gt;&#10;&lt;p&gt;Then a debounce: predicates fire &lt;em&gt;before&lt;/em&gt; the next call, never after. The check happens at the top of the loop body, and crucially &lt;em&gt;before&lt;/em&gt; the API request goes out; checking after the fact is how you accidentally double your spend right at the limit. The cost-accounting and budget-check code paths share state, so both read from the same &lt;code&gt;usage&lt;/code&gt; object the API response updated last iteration. The &lt;a href="https://jatinbansal.com/ai-engineering/agent-harness-anatomy/"&gt;harness anatomy article&lt;/a&gt; made the case for shared state across duties; budgets are the critical example.&lt;/p&gt;&#10;&lt;h2 id="choosing-limits-from-observed-runs"&gt;Choosing limits from observed runs&lt;/h2&gt;&#10;&lt;p&gt;Step cap and wall-clock deadline cost nothing; counter increments and timer reads, sub-microsecond per turn. Token and dollar accounting cost the small overhead of summing a few integers and a multiplication. Per-tool quotas cost a dictionary lookup and an increment. No-progress detection costs an &lt;code&gt;O(K)&lt;/code&gt; scan over the last K tool calls. External signal-checking costs a single atomic read. The total budget-gate overhead is well under a millisecond per iteration, against API call latencies of hundreds of milliseconds to several seconds. The cost of the budget is the cost of forgetting to ship one of the predicates.&lt;/p&gt;&#10;&lt;p&gt;The dollar math the cost-optimization article&amp;rsquo;s framing extends here: the breakeven for adding a budget gate is the cost of one runaway incident divided by the development cost of the gate. A no-progress detector takes an engineer a day; the $47K incident pays for that engineer&amp;rsquo;s salary for the year. A dollar ceiling enforced server-side takes a week; the $87K incident pays for the week. A full audit log of which gate fired when, with a one-page incident playbook attached, takes a month; the difference between catching the next runaway at $500 and at $50,000 pays for the month several times over. The budget surface is the single highest-leverage piece of harness work in the production stack, measured in dollars per engineer-hour spent.&lt;/p&gt;&#10;&lt;p&gt;The framing the rest of the platform tooling supports; &lt;a href="https://help.openai.com/en/articles/10478918-api-usage-dashboard"&gt;billing alerts at OpenAI&lt;/a&gt;, &lt;a href="https://docs.claude.com/en/api/rate-limits"&gt;Anthropic&amp;rsquo;s quota and rate limit settings&lt;/a&gt;, the &lt;a href="https://platform.claude.com/docs/en/about-claude/pricing"&gt;Anthropic pricing page&lt;/a&gt; and its quota-management dashboards; provides the financial backstop above the harness gate. They are not substitutes for the harness gate; the platform-level cap fires after the platform has run for hours, the harness-level gate fires before the next call. Both. &lt;em&gt;Always&lt;/em&gt; both.&lt;/p&gt;&#10;&lt;h2 id="further-reading-from-the-field"&gt;Further reading from the field&lt;/h2&gt;&#10;&lt;ul&gt;&#10;&lt;li&gt;&lt;a href="https://simonwillison.net/tags/agents/"&gt;Simon Willison: Agents tag&lt;/a&gt;. The running index of Willison&amp;rsquo;s posts on agent failure modes; his consistent point is that classifier-based defenses lose to adaptive attackers and that runtime enforcement at the harness is the only durable mitigation. The same logic underlies the budget gate: classifiers (rate limits, fraud-detection-style anomaly scoring) catch the median runaway, but only the synchronous in-loop gate catches the &lt;em&gt;next&lt;/em&gt; runaway.&lt;/li&gt;&#10;&lt;li&gt;&lt;a href="https://aws.amazon.com/builders-library/timeouts-retries-and-backoff-with-jitter/"&gt;AWS Builders&amp;rsquo; Library: Timeouts, retries, and backoff with jitter&lt;/a&gt;. The standard reference for the retry-storm pathology and the jittered backoff mitigation. The discipline ports unchanged to agent loops: every retry is a full provider call, every retry counts against the budget, and the cap-with-jitter is the difference between a transient blip and a $47K bill.&lt;/li&gt;&#10;&lt;li&gt;&lt;a href="https://www.anthropic.com/engineering/building-effective-agents"&gt;Anthropic: Building effective agents&lt;/a&gt;. The December 2024 engineering writeup that distinguishes workflows from agents and recommends simplicity-first defaults. The budget gate is the simplicity-first answer to &amp;ldquo;how do we know the loop is bounded?&amp;rdquo;&lt;/li&gt;&#10;&lt;li&gt;&lt;a href="https://www.philschmid.de/agent-harness-2026"&gt;Phil Schmid: The importance of Agent Harness in 2026&lt;/a&gt;. The harness-as-kernel framing that organizes this article and the &lt;a href="https://jatinbansal.com/ai-engineering/agent-harness-anatomy/"&gt;agent harness anatomy piece&lt;/a&gt;. The budget is the kernel&amp;rsquo;s scheduler; the model is the process; the process doesn&amp;rsquo;t get to choose its own quantum.&lt;/li&gt;&#10;&lt;/ul&gt;&#10;</content:encoded></item><item><title>PII Detection and Data Privacy for LLM Systems</title><link>https://jatinbansal.com/ai-engineering/pii-and-privacy/</link><guid isPermaLink="true">https://jatinbansal.com/ai-engineering/pii-and-privacy/</guid><pubDate>Tue, 26 May 2026 18:24:57 +0000</pubDate><dc:creator>Jatin Bansal</dc:creator><category>AI Engineering</category><description>How to detect, transform, retain, and delete personal data in LLM request and storage paths.</description><content:encoded>&lt;p&gt;Personal data can pass through retrieval, model providers, traces, analytics, caches, and long-term memory during one request. Detection and transformation must happen before each disclosure boundary, while retention and deletion controls must cover every stored derivative.&lt;/p&gt;&#10;&lt;h2 id="detect-pii-in-stages"&gt;Detect PII in stages&lt;/h2&gt;&#10;&lt;p&gt;The production-grade detector is not a single component. It is a &lt;em&gt;cascade&lt;/em&gt;; three layers, each with different precision/recall/latency characteristics, run in series with early exit on confident hits.&lt;/p&gt;&#10;&lt;p&gt;&lt;strong&gt;Layer 1: regex on every byte.&lt;/strong&gt; The first pass catches the structured patterns: email addresses, US/UK/EU phone formats, SSNs and national IDs, credit card numbers (with &lt;a href="https://en.wikipedia.org/wiki/Luhn_algorithm"&gt;Luhn validation&lt;/a&gt; to suppress false positives), IBANs, IPv4/IPv6 addresses, MAC addresses, URLs containing personal subdomains, AWS access keys, Stripe customer IDs. Regex catches roughly 80% of the structured PII at sub-millisecond latency per kilobyte and zero memory cost. The failure mode is &lt;em&gt;unstructured&lt;/em&gt; personal data; names, addresses spelled out in prose, identifiers that don&amp;rsquo;t match a known format. Regex alone is what most teams ship as v1 and what every auditor flags as insufficient by v2.&lt;/p&gt;&#10;&lt;p&gt;&lt;strong&gt;Layer 2: Named-entity recognition.&lt;/strong&gt; The second pass is a small NER model; typically &lt;a href="https://spacy.io/"&gt;spaCy&lt;/a&gt; under &lt;a href="https://microsoft.github.io/presidio/"&gt;Microsoft Presidio&lt;/a&gt;, occasionally a fine-tuned BERT-class model; that tags spans of text with entity types: &lt;code&gt;PERSON&lt;/code&gt;, &lt;code&gt;ORG&lt;/code&gt;, &lt;code&gt;LOCATION&lt;/code&gt;, &lt;code&gt;GPE&lt;/code&gt;, &lt;code&gt;DATE&lt;/code&gt;. NER catches the unstructured cases regex misses, at the cost of single-digit-millisecond per kilobyte and a few hundred MB of model weights. The failure mode is &lt;em&gt;contextual&lt;/em&gt;; a name that&amp;rsquo;s also a common noun (&amp;ldquo;Ms. Booker&amp;rdquo;), an organization name that&amp;rsquo;s a person&amp;rsquo;s name, a date that&amp;rsquo;s incidental rather than tied to a person. NER alone has higher recall but lower precision than regex; the false-positive rate runs 5–15% in production traffic.&lt;/p&gt;&#10;&lt;p&gt;&lt;strong&gt;Layer 3: LLM-based classification.&lt;/strong&gt; The third pass; used sparingly, only on flagged samples or on the highest-risk surfaces; runs a guard model (&lt;a href="https://www.llama.com/docs/model-cards-and-prompt-formats/llama-guard-4/"&gt;Llama Guard 4&lt;/a&gt; S7: Privacy category, &lt;a href="https://openai.com/index/introducing-openai-privacy-filter/"&gt;OpenAI&amp;rsquo;s Privacy Filter&lt;/a&gt; released as open-weight in 2026, or a small dedicated classifier) to disambiguate the cases NER couldn&amp;rsquo;t. LLM-based classification handles context the previous layers can&amp;rsquo;t (&amp;ldquo;this is a fictional name in a story&amp;rdquo; vs &amp;ldquo;this is a real person&amp;rsquo;s name in a customer support ticket&amp;rdquo;), at the cost of 50–200ms of additional latency and the dollar cost of an extra inference call.&lt;/p&gt;&#10;&lt;p&gt;The cascade pattern is the same shape as the &lt;a href="https://jatinbansal.com/ai-engineering/reranking/"&gt;reranking cascade&lt;/a&gt;; a cheap, high-recall first pass narrows the candidate set; a more expensive, higher-precision later pass refines the verdict. The production calibration: regex on every byte; NER on every byte that regex didn&amp;rsquo;t already cover with high confidence; LLM-based classification only on flagged samples in eval pipelines or on the highest-risk inputs (uploaded documents, retrieved chunks crossing tenant boundaries). &lt;a href="https://predictionguard.com/blog/pii-detection-redaction-llm-pipelines-regulated-industries"&gt;Prediction Guard&amp;rsquo;s PII pipeline writeup&lt;/a&gt; lays out the same three-tier architecture from a regulated-industries angle; the cascade is by now consensus.&lt;/p&gt;&#10;&lt;h2 id="choose-a-transformation"&gt;Choose a transformation&lt;/h2&gt;&#10;&lt;p&gt;Once detected, the personal data has to be &lt;em&gt;transformed&lt;/em&gt; before it crosses whatever boundary the policy guards. Four modes, in increasing order of how much utility they preserve:&lt;/p&gt;&#10;&lt;p&gt;&lt;strong&gt;Mode 1: Redaction.&lt;/strong&gt; Replace the span with a placeholder: &lt;code&gt;[EMAIL]&lt;/code&gt;, &lt;code&gt;[REDACTED]&lt;/code&gt;, &lt;code&gt;█████&lt;/code&gt;. Loses all information including type. Cheap, deterministic, irreversible. Appropriate for telemetry exports and eval samples where the personal data was never load-bearing.&lt;/p&gt;&#10;&lt;p&gt;&lt;strong&gt;Mode 2: Masking.&lt;/strong&gt; Preserve type, lose specifics: &lt;code&gt;[EMAIL_1]&lt;/code&gt;, &lt;code&gt;[PHONE_2]&lt;/code&gt;. Adds an index so the model can distinguish multiple PII spans in the same prompt (&amp;ldquo;Send the report to &lt;code&gt;[EMAIL_1]&lt;/code&gt; and CC &lt;code&gt;[EMAIL_2]&lt;/code&gt;&amp;rdquo;). Cheap, deterministic, &lt;em&gt;reversible if you store the mapping&lt;/em&gt;. The reversible variant; store &lt;code&gt;[EMAIL_1] → alice@example.com&lt;/code&gt; in a tenant-scoped vault, un-mask in the response; is the dominant pattern for production LLM pipelines because it preserves utility without exposing the cleartext to the model or to downstream telemetry.&lt;/p&gt;&#10;&lt;p&gt;&lt;strong&gt;Mode 3: Synthetic substitution.&lt;/strong&gt; Replace &lt;code&gt;alice@example.com&lt;/code&gt; with &lt;code&gt;taylor@example.org&lt;/code&gt;. Preserves type and locale (a UK postcode stays a UK postcode); the model sees realistic-looking text and produces coherent output; the original lives in the vault and substitutes back on rendering. This is the &lt;a href="https://microsoft.github.io/presidio/anonymizer/"&gt;Presidio anonymizer engine&lt;/a&gt;&amp;rsquo;s default operation. Higher utility than masking (the model handles plausible names better than placeholder tokens), at the cost of having to be careful that the substitution doesn&amp;rsquo;t accidentally &lt;em&gt;introduce&lt;/em&gt; a real person&amp;rsquo;s identity; a synthetic name generator that pulls from a finite pool can collide with a real customer&amp;rsquo;s name in the next request.&lt;/p&gt;&#10;&lt;p&gt;&lt;strong&gt;Mode 4: Tokenization with a hardened vault.&lt;/strong&gt; The full PCI pattern: the personal data lives in a separate system with its own access controls, audit log, and encryption-at-rest with a customer-managed key; the LLM pipeline sees opaque tokens; un-tokenization happens at the rendering boundary by a service that re-authenticates the user. Highest utility (the model&amp;rsquo;s output can include the original values, indirectly, via the rendering layer un-tokenizing) and highest engineering cost. Appropriate when the data is high-sensitivity (healthcare, financial) or the regulatory regime demands segregated storage.&lt;/p&gt;&#10;&lt;p&gt;The choice between modes is policy, not engineering. The engineering question is: does the operation need to be reversible, and which boundary does the cleartext live behind? Once those are answered, the mode follows.&lt;/p&gt;&#10;&lt;h2 id="place-privacy-checks-in-the-request-path"&gt;Place privacy checks in the request path&lt;/h2&gt;&#10;&lt;p&gt;The cascade adds latency in the obvious place; between the user request and the model call; and again on the response side. A realistic budget on a typical pipeline:&lt;/p&gt;&#10;&lt;table&gt;&#10;&#9;&lt;thead&gt;&#10;&#9;&#9;&#9;&lt;tr&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;th&gt;Stage&lt;/th&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;th&gt;Median latency&lt;/th&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;th&gt;Cost adder&lt;/th&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;th&gt;Notes&lt;/th&gt;&#10;&#9;&#9;&#9;&lt;/tr&gt;&#10;&#9;&lt;/thead&gt;&#10;&#9;&lt;tbody&gt;&#10;&#9;&#9;&#9;&lt;tr&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;td&gt;Regex pass&lt;/td&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;td&gt;0.5–2ms/KB&lt;/td&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;td&gt;$0&lt;/td&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;td&gt;Pure CPU, easy to parallelize&lt;/td&gt;&#10;&#9;&#9;&#9;&lt;/tr&gt;&#10;&#9;&#9;&#9;&lt;tr&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;td&gt;NER pass (Presidio + spaCy)&lt;/td&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;td&gt;5–20ms/KB&lt;/td&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;td&gt;$0 (self-hosted)&lt;/td&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;td&gt;Single-threaded per request; batch where you can&lt;/td&gt;&#10;&#9;&#9;&#9;&lt;/tr&gt;&#10;&#9;&#9;&#9;&lt;tr&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;td&gt;LLM-based classifier (input)&lt;/td&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;td&gt;50–200ms&lt;/td&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;td&gt;$0.0001–0.001&lt;/td&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;td&gt;Only on flagged samples or high-risk paths&lt;/td&gt;&#10;&#9;&#9;&#9;&lt;/tr&gt;&#10;&#9;&#9;&#9;&lt;tr&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;td&gt;Vault token mint/lookup&lt;/td&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;td&gt;1–5ms&lt;/td&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;td&gt;$0&lt;/td&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;td&gt;Local with a Redis-class store&lt;/td&gt;&#10;&#9;&#9;&#9;&lt;/tr&gt;&#10;&#9;&#9;&#9;&lt;tr&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;td&gt;Output cascade (mirror of above)&lt;/td&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;td&gt;7–25ms&lt;/td&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;td&gt;$0 + classifier sample&lt;/td&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;td&gt;Run in parallel with output guard where possible&lt;/td&gt;&#10;&#9;&#9;&#9;&lt;/tr&gt;&#10;&#9;&#9;&#9;&lt;tr&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;td&gt;&lt;strong&gt;Total typical overhead&lt;/strong&gt;&lt;/td&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;td&gt;&lt;strong&gt;~15–50ms&lt;/strong&gt;&lt;/td&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;td&gt;&lt;strong&gt;&amp;lt;$0.001 typical&lt;/strong&gt;&lt;/td&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;td&gt;Dominated by NER; LLM tier is sampled&lt;/td&gt;&#10;&#9;&#9;&#9;&lt;/tr&gt;&#10;&#9;&lt;/tbody&gt;&#10;&lt;/table&gt;&#10;&lt;p&gt;The latency budget interacts with the rest of the production stack. Run the cascade &lt;em&gt;before&lt;/em&gt; the &lt;a href="https://jatinbansal.com/ai-engineering/prompt-caching/"&gt;prompt-cache prefix&lt;/a&gt;; if you tokenize after the cache hits, the cache stores cleartext and the deletion pipeline can&amp;rsquo;t reach it. Run the cascade &lt;em&gt;before&lt;/em&gt; the &lt;a href="https://jatinbansal.com/ai-engineering/llm-observability/"&gt;observability tracer&lt;/a&gt; sees the prompt; span attributes are personal data when they contain personal data, and the trace store is a separate retention tier the deletion API has to invalidate. Run the cascade &lt;em&gt;after&lt;/em&gt; the user&amp;rsquo;s authentication is verified; pre-auth requests don&amp;rsquo;t deserve the latency budget, and unauthenticated traffic that contains personal data is its own incident.&lt;/p&gt;&#10;&lt;p&gt;The structural alternative; and the one the on-device-inference camp argues is the only honest answer; is to avoid the latency entirely by avoiding the network call. On-device inference shipping in 2026 (&lt;a href="https://developer.apple.com/documentation/FoundationModels"&gt;Apple Foundation Models&lt;/a&gt;, &lt;a href="https://pytorch.org/executorch/"&gt;executorch on Android&lt;/a&gt;, &lt;a href="https://llm.mlc.ai/"&gt;MLC Chat&lt;/a&gt;) means the personal data never leaves the device, the PII detection cascade becomes optional rather than mandatory, the residency question is trivially answered, and the deletion is what the OS already does when the user uninstalls the app. The trade-off is model quality; the 3B-parameter on-device class is materially weaker than frontier cloud models, and the &lt;a href="https://jatinbansal.com/ai-engineering/model-routing/"&gt;model-routing layer&lt;/a&gt; becomes the natural place to send only privacy-sensitive workloads on-device while keeping the rest in the cloud. We return to this in trade-offs.&lt;/p&gt;&#10;&lt;h2 id="code-a-detect-redact-deidentify-pipeline-in-python"&gt;Code: a detect-redact-deidentify pipeline in Python&lt;/h2&gt;&#10;&lt;p&gt;The example is the v1 production cut: Presidio&amp;rsquo;s analyzer for detection, a vault for reversible mapping, redaction modes per entity, and a clean interface that the rest of the harness can call. Install: &lt;code&gt;pip install presidio-analyzer presidio-anonymizer openai anthropic&lt;/code&gt;. The analyzer needs &lt;code&gt;python -m spacy download en_core_web_lg&lt;/code&gt; once.&lt;/p&gt;&#10;&lt;figure class="nb-code"&gt;&#10; &lt;header class="nb-code-chrome"&gt;&#10; &lt;span class="nb-code-label"&gt;python&lt;/span&gt;&#10; &#10; &lt;/header&gt;&#10; &lt;div class="nb-code-body"&gt;&lt;div class="highlight"&gt;&lt;div class="chroma"&gt;&#10;&lt;table class="lntable"&gt;&lt;tr&gt;&lt;td class="lntd"&gt;&#10;&lt;pre tabindex="0" class="chroma"&gt;&lt;code&gt;&lt;span class="lnt"&gt; 1&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 2&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 3&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 4&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 5&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 6&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 7&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 8&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 9&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 10&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 11&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 12&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 13&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 14&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 15&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 16&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 17&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 18&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 19&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 20&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 21&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 22&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 23&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 24&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 25&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 26&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 27&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 28&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 29&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 30&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 31&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 32&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 33&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 34&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 35&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 36&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 37&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 38&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 39&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 40&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 41&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 42&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 43&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 44&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 45&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 46&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 47&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 48&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 49&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 50&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 51&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 52&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 53&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 54&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 55&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 56&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 57&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 58&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 59&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 60&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 61&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 62&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 63&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 64&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 65&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 66&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 67&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 68&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 69&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 70&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 71&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 72&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 73&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 74&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 75&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 76&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 77&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 78&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 79&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 80&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 81&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 82&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 83&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 84&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 85&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 86&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 87&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 88&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 89&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 90&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 91&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 92&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 93&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 94&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 95&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 96&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 97&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 98&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 99&#10;&lt;/span&gt;&lt;span class="lnt"&gt;100&#10;&lt;/span&gt;&lt;span class="lnt"&gt;101&#10;&lt;/span&gt;&lt;span class="lnt"&gt;102&#10;&lt;/span&gt;&lt;span class="lnt"&gt;103&#10;&lt;/span&gt;&lt;span class="lnt"&gt;104&#10;&lt;/span&gt;&lt;span class="lnt"&gt;105&#10;&lt;/span&gt;&lt;span class="lnt"&gt;106&#10;&lt;/span&gt;&lt;span class="lnt"&gt;107&#10;&lt;/span&gt;&lt;span class="lnt"&gt;108&#10;&lt;/span&gt;&lt;span class="lnt"&gt;109&#10;&lt;/span&gt;&lt;span class="lnt"&gt;110&#10;&lt;/span&gt;&lt;span class="lnt"&gt;111&#10;&lt;/span&gt;&lt;span class="lnt"&gt;112&#10;&lt;/span&gt;&lt;span class="lnt"&gt;113&#10;&lt;/span&gt;&lt;span class="lnt"&gt;114&#10;&lt;/span&gt;&lt;span class="lnt"&gt;115&#10;&lt;/span&gt;&lt;span class="lnt"&gt;116&#10;&lt;/span&gt;&lt;span class="lnt"&gt;117&#10;&lt;/span&gt;&lt;span class="lnt"&gt;118&#10;&lt;/span&gt;&lt;span class="lnt"&gt;119&#10;&lt;/span&gt;&lt;span class="lnt"&gt;120&#10;&lt;/span&gt;&lt;span class="lnt"&gt;121&#10;&lt;/span&gt;&lt;span class="lnt"&gt;122&#10;&lt;/span&gt;&lt;span class="lnt"&gt;123&#10;&lt;/span&gt;&lt;span class="lnt"&gt;124&#10;&lt;/span&gt;&lt;span class="lnt"&gt;125&#10;&lt;/span&gt;&lt;span class="lnt"&gt;126&#10;&lt;/span&gt;&lt;span class="lnt"&gt;127&#10;&lt;/span&gt;&lt;span class="lnt"&gt;128&#10;&lt;/span&gt;&lt;span class="lnt"&gt;129&#10;&lt;/span&gt;&lt;span class="lnt"&gt;130&#10;&lt;/span&gt;&lt;span class="lnt"&gt;131&#10;&lt;/span&gt;&lt;span class="lnt"&gt;132&#10;&lt;/span&gt;&lt;span class="lnt"&gt;133&#10;&lt;/span&gt;&lt;span class="lnt"&gt;134&#10;&lt;/span&gt;&lt;span class="lnt"&gt;135&#10;&lt;/span&gt;&lt;span class="lnt"&gt;136&#10;&lt;/span&gt;&lt;span class="lnt"&gt;137&#10;&lt;/span&gt;&lt;span class="lnt"&gt;138&#10;&lt;/span&gt;&lt;span class="lnt"&gt;139&#10;&lt;/span&gt;&lt;span class="lnt"&gt;140&#10;&lt;/span&gt;&lt;span class="lnt"&gt;141&#10;&lt;/span&gt;&lt;span class="lnt"&gt;142&#10;&lt;/span&gt;&lt;span class="lnt"&gt;143&#10;&lt;/span&gt;&lt;span class="lnt"&gt;144&#10;&lt;/span&gt;&lt;span class="lnt"&gt;145&#10;&lt;/span&gt;&lt;span class="lnt"&gt;146&#10;&lt;/span&gt;&lt;span class="lnt"&gt;147&#10;&lt;/span&gt;&lt;span class="lnt"&gt;148&#10;&lt;/span&gt;&lt;span class="lnt"&gt;149&#10;&lt;/span&gt;&lt;span class="lnt"&gt;150&#10;&lt;/span&gt;&lt;span class="lnt"&gt;151&#10;&lt;/span&gt;&lt;span class="lnt"&gt;152&#10;&lt;/span&gt;&lt;span class="lnt"&gt;153&#10;&lt;/span&gt;&lt;span class="lnt"&gt;154&#10;&lt;/span&gt;&lt;span class="lnt"&gt;155&#10;&lt;/span&gt;&lt;span class="lnt"&gt;156&#10;&lt;/span&gt;&lt;span class="lnt"&gt;157&#10;&lt;/span&gt;&lt;span class="lnt"&gt;158&#10;&lt;/span&gt;&lt;span class="lnt"&gt;159&#10;&lt;/span&gt;&lt;span class="lnt"&gt;160&#10;&lt;/span&gt;&lt;span class="lnt"&gt;161&#10;&lt;/span&gt;&lt;span class="lnt"&gt;162&#10;&lt;/span&gt;&lt;span class="lnt"&gt;163&#10;&lt;/span&gt;&lt;span class="lnt"&gt;164&#10;&lt;/span&gt;&lt;span class="lnt"&gt;165&#10;&lt;/span&gt;&lt;span class="lnt"&gt;166&#10;&lt;/span&gt;&lt;span class="lnt"&gt;167&#10;&lt;/span&gt;&lt;span class="lnt"&gt;168&#10;&lt;/span&gt;&lt;span class="lnt"&gt;169&#10;&lt;/span&gt;&lt;span class="lnt"&gt;170&#10;&lt;/span&gt;&lt;span class="lnt"&gt;171&#10;&lt;/span&gt;&lt;span class="lnt"&gt;172&#10;&lt;/span&gt;&lt;span class="lnt"&gt;173&#10;&lt;/span&gt;&lt;span class="lnt"&gt;174&#10;&lt;/span&gt;&lt;span class="lnt"&gt;175&#10;&lt;/span&gt;&lt;span class="lnt"&gt;176&#10;&lt;/span&gt;&lt;span class="lnt"&gt;177&#10;&lt;/span&gt;&lt;span class="lnt"&gt;178&#10;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/td&gt;&#10;&lt;td class="lntd"&gt;&#10;&lt;pre tabindex="0" class="chroma"&gt;&lt;code class="language-python" data-lang="python"&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="kn"&gt;from&lt;/span&gt; &lt;span class="nn"&gt;__future__&lt;/span&gt; &lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;annotations&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="nn"&gt;os&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt; &lt;span class="nn"&gt;secrets&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt; &lt;span class="nn"&gt;time&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt; &lt;span class="nn"&gt;json&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="kn"&gt;from&lt;/span&gt; &lt;span class="nn"&gt;dataclasses&lt;/span&gt; &lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;dataclass&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;field&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="kn"&gt;from&lt;/span&gt; &lt;span class="nn"&gt;typing&lt;/span&gt; &lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;Literal&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="kn"&gt;from&lt;/span&gt; &lt;span class="nn"&gt;presidio_analyzer&lt;/span&gt; &lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;AnalyzerEngine&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="kn"&gt;from&lt;/span&gt; &lt;span class="nn"&gt;presidio_anonymizer&lt;/span&gt; &lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;AnonymizerEngine&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="kn"&gt;from&lt;/span&gt; &lt;span class="nn"&gt;presidio_anonymizer.entities&lt;/span&gt; &lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;OperatorConfig&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="kn"&gt;from&lt;/span&gt; &lt;span class="nn"&gt;anthropic&lt;/span&gt; &lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;Anthropic&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="c1"&gt;# Detection: Presidio&amp;#39;s analyzer covers the structured + NER tiers.&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="n"&gt;analyzer&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;AnalyzerEngine&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="n"&gt;anonymizer&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;AnonymizerEngine&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="n"&gt;client&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;Anthropic&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="n"&gt;RedactionMode&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;Literal&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;&amp;#34;redact&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;&amp;#34;mask&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;&amp;#34;synthetic&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;&amp;#34;tokenize&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="nd"&gt;@dataclass&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="k"&gt;class&lt;/span&gt; &lt;span class="nc"&gt;VaultEntry&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;token&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;str&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;cleartext&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;str&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;entity_type&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;str&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;tenant_id&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;str&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;created_at&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;float&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;field&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;default_factory&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;time&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;time&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="k"&gt;class&lt;/span&gt; &lt;span class="nc"&gt;Vault&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="s2"&gt;&amp;#34;&amp;#34;&amp;#34;Token vault for reversible redaction. Tenant-scoped, in-memory here;&#10;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="s2"&gt; in production, swap for an HSM-backed store with its own audit log.&amp;#34;&amp;#34;&amp;#34;&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="fm"&gt;__init__&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="bp"&gt;self&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;-&amp;gt;&lt;/span&gt; &lt;span class="kc"&gt;None&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="bp"&gt;self&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;_by_token&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;dict&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="nb"&gt;str&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;VaultEntry&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;{}&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="bp"&gt;self&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;_by_cleartext&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;dict&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="nb"&gt;tuple&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="nb"&gt;str&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nb"&gt;str&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt; &lt;span class="nb"&gt;str&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;{}&lt;/span&gt; &lt;span class="c1"&gt;# (tenant, cleartext) -&amp;gt; token&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;mint&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="bp"&gt;self&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;cleartext&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;str&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;entity_type&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;str&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;tenant_id&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;str&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;-&amp;gt;&lt;/span&gt; &lt;span class="nb"&gt;str&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;key&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;tenant_id&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;cleartext&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;key&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="bp"&gt;self&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;_by_cleartext&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="bp"&gt;self&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;_by_cleartext&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;key&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="c1"&gt;# Token format: TYPE-&amp;lt;random&amp;gt;; opaque to the model, type-readable for prompts.&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;token&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="sa"&gt;f&lt;/span&gt;&lt;span class="s2"&gt;&amp;#34;[&lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;entity_type&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s2"&gt;-&lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;secrets&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;token_hex&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;4&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;upper&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s2"&gt;]&amp;#34;&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="bp"&gt;self&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;_by_token&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;token&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;VaultEntry&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;token&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;cleartext&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;entity_type&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;tenant_id&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="bp"&gt;self&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;_by_cleartext&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;key&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;token&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;token&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;reveal&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="bp"&gt;self&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;token&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;str&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;tenant_id&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;str&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;-&amp;gt;&lt;/span&gt; &lt;span class="nb"&gt;str&lt;/span&gt; &lt;span class="o"&gt;|&lt;/span&gt; &lt;span class="kc"&gt;None&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;entry&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="bp"&gt;self&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;_by_token&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;token&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;entry&lt;/span&gt; &lt;span class="ow"&gt;is&lt;/span&gt; &lt;span class="kc"&gt;None&lt;/span&gt; &lt;span class="ow"&gt;or&lt;/span&gt; &lt;span class="n"&gt;entry&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;tenant_id&lt;/span&gt; &lt;span class="o"&gt;!=&lt;/span&gt; &lt;span class="n"&gt;tenant_id&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="kc"&gt;None&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;entry&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;cleartext&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;purge&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="bp"&gt;self&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;tenant_id&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;str&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;-&amp;gt;&lt;/span&gt; &lt;span class="nb"&gt;int&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="s2"&gt;&amp;#34;&amp;#34;&amp;#34;GDPR right-to-erasure. Returns count of tokens purged for verification.&amp;#34;&amp;#34;&amp;#34;&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;to_drop&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;t&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;t&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;e&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="bp"&gt;self&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;_by_token&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;items&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;e&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;tenant_id&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="n"&gt;tenant_id&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;t&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;to_drop&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;entry&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="bp"&gt;self&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;_by_token&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;pop&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;t&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="bp"&gt;self&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;_by_cleartext&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;pop&lt;/span&gt;&lt;span class="p"&gt;((&lt;/span&gt;&lt;span class="n"&gt;entry&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;tenant_id&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;entry&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;cleartext&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt; &lt;span class="kc"&gt;None&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="nb"&gt;len&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;to_drop&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="nd"&gt;@dataclass&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="k"&gt;class&lt;/span&gt; &lt;span class="nc"&gt;PiiResult&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;transformed_text&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;str&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;detected&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;list&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="nb"&gt;dict&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;tokens_minted&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;list&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="nb"&gt;str&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;cascade_ms&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;float&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;detect_and_transform&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;text&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;str&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;tenant_id&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;str&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;vault&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;Vault&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;mode&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;RedactionMode&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s2"&gt;&amp;#34;tokenize&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;language&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;str&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s2"&gt;&amp;#34;en&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;-&amp;gt;&lt;/span&gt; &lt;span class="n"&gt;PiiResult&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="s2"&gt;&amp;#34;&amp;#34;&amp;#34;Cascade: regex + NER via Presidio; transform per the policy mode.&amp;#34;&amp;#34;&amp;#34;&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;t0&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;time&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;perf_counter&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;results&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;analyzer&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;analyze&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;text&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;text&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;language&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;language&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;detected&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="p"&gt;{&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="s2"&gt;&amp;#34;type&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;entity_type&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="s2"&gt;&amp;#34;start&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;start&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="s2"&gt;&amp;#34;end&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;end&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="s2"&gt;&amp;#34;score&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;score&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="s2"&gt;&amp;#34;span&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;text&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;start&lt;/span&gt; &lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;end&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="p"&gt;}&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;r&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;results&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="p"&gt;]&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;mode&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="s2"&gt;&amp;#34;redact&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;op&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;OperatorConfig&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;&amp;#34;replace&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="s2"&gt;&amp;#34;new_value&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;&amp;#34;[REDACTED]&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;})&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;out&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;anonymizer&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;anonymize&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;text&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;text&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;analyzer_results&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;results&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;operators&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="s2"&gt;&amp;#34;DEFAULT&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;op&lt;/span&gt;&lt;span class="p"&gt;})&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;PiiResult&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;out&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;text&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;detected&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="p"&gt;[],&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;time&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;perf_counter&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt; &lt;span class="n"&gt;t0&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="mi"&gt;1000&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;mode&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;&amp;#34;mask&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;&amp;#34;tokenize&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;tokens_minted&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;list&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="nb"&gt;str&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;[]&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="c1"&gt;# Build span -&amp;gt; token map for each unique cleartext per entity type.&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;operators&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;dict&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="nb"&gt;str&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;OperatorConfig&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;{}&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="c1"&gt;# Presidio applies a single operator per entity type by default; for&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="c1"&gt;# per-span tokens we mint up-front and rewrite the text ourselves.&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="c1"&gt;# Sort spans right-to-left so offsets stay valid as we splice.&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;out_text&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;text&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;r&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="nb"&gt;sorted&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;results&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;key&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="k"&gt;lambda&lt;/span&gt; &lt;span class="n"&gt;x&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;x&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;start&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;reverse&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="kc"&gt;True&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;cleartext&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;text&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;start&lt;/span&gt; &lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;end&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;token&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;vault&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;mint&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;cleartext&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;entity_type&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;tenant_id&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;tokens_minted&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;append&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;token&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;out_text&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;out_text&lt;/span&gt;&lt;span class="p"&gt;[:&lt;/span&gt; &lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;start&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="n"&gt;token&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="n"&gt;out_text&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;end&lt;/span&gt; &lt;span class="p"&gt;:]&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;PiiResult&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;out_text&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;detected&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;tokens_minted&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;time&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;perf_counter&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt; &lt;span class="n"&gt;t0&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="mi"&gt;1000&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;mode&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="s2"&gt;&amp;#34;synthetic&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="c1"&gt;# Presidio&amp;#39;s `replace` with type-aware synthetic values via Faker.&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="kn"&gt;from&lt;/span&gt; &lt;span class="nn"&gt;faker&lt;/span&gt; &lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;Faker&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;fake&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;Faker&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;replacements&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="s2"&gt;&amp;#34;EMAIL_ADDRESS&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="k"&gt;lambda&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;fake&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;email&lt;/span&gt;&lt;span class="p"&gt;(),&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="s2"&gt;&amp;#34;PHONE_NUMBER&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="k"&gt;lambda&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;fake&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;phone_number&lt;/span&gt;&lt;span class="p"&gt;(),&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="s2"&gt;&amp;#34;PERSON&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="k"&gt;lambda&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;fake&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;name&lt;/span&gt;&lt;span class="p"&gt;(),&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="s2"&gt;&amp;#34;LOCATION&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="k"&gt;lambda&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;fake&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;city&lt;/span&gt;&lt;span class="p"&gt;(),&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="s2"&gt;&amp;#34;CREDIT_CARD&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="k"&gt;lambda&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;fake&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;credit_card_number&lt;/span&gt;&lt;span class="p"&gt;(),&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="s2"&gt;&amp;#34;US_SSN&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="k"&gt;lambda&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;fake&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;ssn&lt;/span&gt;&lt;span class="p"&gt;(),&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="s2"&gt;&amp;#34;IBAN_CODE&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="k"&gt;lambda&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;fake&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;iban&lt;/span&gt;&lt;span class="p"&gt;(),&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="p"&gt;}&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="c1"&gt;# Right-to-left splice with synthetic substitution; vault still tracks&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="c1"&gt;# so the response layer can un-substitute if utility requires.&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;out_text&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;text&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;tokens_minted&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;[]&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;r&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="nb"&gt;sorted&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;results&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;key&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="k"&gt;lambda&lt;/span&gt; &lt;span class="n"&gt;x&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;x&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;start&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;reverse&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="kc"&gt;True&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;cleartext&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;text&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;start&lt;/span&gt; &lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;end&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;synth&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;replacements&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;entity_type&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="k"&gt;lambda&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;&amp;#34;[REDACTED]&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;)()&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="c1"&gt;# Store cleartext-&amp;gt;synthetic in vault so we can reverse on rendering.&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;token&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;vault&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;mint&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;cleartext&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;entity_type&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;tenant_id&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;tokens_minted&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;append&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;token&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;out_text&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;out_text&lt;/span&gt;&lt;span class="p"&gt;[:&lt;/span&gt; &lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;start&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="n"&gt;synth&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="n"&gt;out_text&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;end&lt;/span&gt; &lt;span class="p"&gt;:]&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;PiiResult&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;out_text&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;detected&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;tokens_minted&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;time&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;perf_counter&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt; &lt;span class="n"&gt;t0&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="mi"&gt;1000&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="k"&gt;raise&lt;/span&gt; &lt;span class="ne"&gt;ValueError&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sa"&gt;f&lt;/span&gt;&lt;span class="s2"&gt;&amp;#34;unknown mode: &lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;mode&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s2"&gt;&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;reveal_tokens&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;text&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;str&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;tenant_id&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;str&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;vault&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;Vault&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;-&amp;gt;&lt;/span&gt; &lt;span class="nb"&gt;str&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="s2"&gt;&amp;#34;&amp;#34;&amp;#34;Un-tokenize on the rendering boundary. Only the response layer that&#10;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="s2"&gt; already had cleartext for this tenant should call this.&amp;#34;&amp;#34;&amp;#34;&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="nn"&gt;re&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;pattern&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;re&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;compile&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sa"&gt;r&lt;/span&gt;&lt;span class="s2"&gt;&amp;#34;\[[A-Z_]+-[0-9A-F]&lt;/span&gt;&lt;span class="si"&gt;{8}&lt;/span&gt;&lt;span class="s2"&gt;\]&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;pattern&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;sub&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="k"&gt;lambda&lt;/span&gt; &lt;span class="n"&gt;m&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;vault&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;reveal&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;m&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;group&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt; &lt;span class="n"&gt;tenant_id&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="ow"&gt;or&lt;/span&gt; &lt;span class="n"&gt;m&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;group&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;text&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="p"&gt;)&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="c1"&gt;# --- Production path: request -&amp;gt; detect -&amp;gt; call model -&amp;gt; detect on output -&amp;gt; render ---&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;answer&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;user_text&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;str&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;tenant_id&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;str&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;vault&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;Vault&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;-&amp;gt;&lt;/span&gt; &lt;span class="nb"&gt;dict&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;inbound&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;detect_and_transform&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;user_text&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;tenant_id&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;vault&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;mode&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;#34;tokenize&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="c1"&gt;# Model never sees the cleartext; the tokens give it enough structure to&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="c1"&gt;# reason about the request without seeing the personal data.&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;system&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="s2"&gt;&amp;#34;You are a helpful assistant. Inputs may contain opaque tokens like &amp;#34;&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="s2"&gt;&amp;#34;[EMAIL-DEADBEEF] standing in for personal data. Quote them verbatim &amp;#34;&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="s2"&gt;&amp;#34;where you need to reference the underlying value; the rendering &amp;#34;&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="s2"&gt;&amp;#34;layer will substitute back before display.&amp;#34;&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="p"&gt;)&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;response&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;client&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;messages&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;create&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;model&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;#34;claude-sonnet-4-6&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;max_tokens&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;1024&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;system&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;system&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;messages&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="p"&gt;[{&lt;/span&gt;&lt;span class="s2"&gt;&amp;#34;role&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;&amp;#34;user&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;&amp;#34;content&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;inbound&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;transformed_text&lt;/span&gt;&lt;span class="p"&gt;}],&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="p"&gt;)&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;model_text&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;response&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;content&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;text&lt;/span&gt; &lt;span class="c1"&gt;# type: ignore[attr-defined]&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="c1"&gt;# Output-side cascade: catch PII the model might have generated that wasn&amp;#39;t&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="c1"&gt;# in the input (training-data leakage, hallucinated identifiers).&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;outbound&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;detect_and_transform&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;model_text&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;tenant_id&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;vault&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;mode&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;#34;redact&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="c1"&gt;# Un-tokenize ONLY the tokens we minted on this request&amp;#39;s inbound side;&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="c1"&gt;# any newly-detected outbound PII stays redacted.&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;rendered&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;reveal_tokens&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;outbound&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;transformed_text&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;tenant_id&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;vault&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="s2"&gt;&amp;#34;rendered&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;rendered&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="s2"&gt;&amp;#34;inbound_detected&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;inbound&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;detected&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="s2"&gt;&amp;#34;outbound_detected&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;outbound&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;detected&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="s2"&gt;&amp;#34;cascade_latency_ms&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;inbound&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;cascade_ms&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="n"&gt;outbound&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;cascade_ms&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="p"&gt;}&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/td&gt;&lt;/tr&gt;&lt;/table&gt;&#10;&lt;/div&gt;&#10;&lt;/div&gt;&lt;/div&gt;&#10;&lt;/figure&gt;&#10;&lt;p&gt;The shape worth internalizing: detection is symmetric (input and output), tokenization is asymmetric (only the tokens &lt;em&gt;this request&lt;/em&gt; minted are eligible to be reversed on the response), and the deletion API (&lt;code&gt;vault.purge&lt;/code&gt;) is the artifact a GDPR auditor asks for. The naive failure mode; un-tokenize &lt;em&gt;every&lt;/em&gt; token in the output; is exactly how a multi-tenant system leaks across tenants when the model regurgitates a token from a different tenant&amp;rsquo;s prompt cache.&lt;/p&gt;&#10;&lt;h2 id="further-reading-from-the-field"&gt;Further reading from the field&lt;/h2&gt;&#10;&lt;ul&gt;&#10;&lt;li&gt;&lt;a href="https://microsoft.github.io/presidio/"&gt;Microsoft Presidio; official documentation&lt;/a&gt;; the canonical reference for the open-source detector and anonymizer. Read the &lt;a href="https://microsoft.github.io/presidio/analyzer/"&gt;analyzer recognizers list&lt;/a&gt; and the &lt;a href="https://microsoft.github.io/presidio/anonymizer/"&gt;anonymizer operators&lt;/a&gt; before designing your detect-redact pipeline; both are the cleanest description of what a production-grade PII stack actually contains, and the &lt;a href="https://techcommunity.microsoft.com/blog/azuredevcommunityblog/introducing-pii-shield-a-privacy-proxy-for-every-llm-call/4514726"&gt;PII Shield blog post&lt;/a&gt; shows how Microsoft itself wraps Presidio as a privacy proxy in front of LLM calls.&lt;/li&gt;&#10;&lt;li&gt;&lt;a href="https://openai.com/index/introducing-openai-privacy-filter/"&gt;OpenAI; Introducing the OpenAI Privacy Filter&lt;/a&gt;; the May 2026 release of an open-weight model purpose-built for detecting and redacting PII in text, designed to run locally. This is the highest-quality drop-in alternative to NER-based detectors for the LLM-based third tier of the cascade, and the open-weight release means it can run inside your own infrastructure with no data egress.&lt;/li&gt;&#10;&lt;li&gt;&lt;a href="https://www.edpb.europa.eu/news/news/2024/edpb-opinion-ai-models-gdpr-principles-support-responsible-ai_en"&gt;EDPB; Opinion 28/2024 on AI models and personal data processing&lt;/a&gt;; the European Data Protection Board&amp;rsquo;s December 2024 opinion on when AI models can be considered anonymized, the legitimate-interests basis for training and deployment, and the deletion-and-erasure expectations. This is the most authoritative single source on how EU regulators expect LLM systems to be governed in 2026.&lt;/li&gt;&#10;&lt;li&gt;&lt;a href="https://simonwillison.net/tags/prompt-injection/"&gt;Simon Willison; Prompt injection and AI agents&lt;/a&gt;; the running tag index Willison maintains; every post in the tag is relevant to the PII-via-injection failure mode, and his &lt;a href="https://simonwillison.net/2025/Jun/16/the-lethal-trifecta/"&gt;lethal trifecta framing&lt;/a&gt; is the cleanest formal statement of when an agent can exfiltrate personal data via injection.&lt;/li&gt;&#10;&lt;/ul&gt;&#10;</content:encoded></item><item><title>Guardrails: Input and Output Safety Layers for LLM Systems</title><link>https://jatinbansal.com/ai-engineering/guardrails/</link><guid isPermaLink="true">https://jatinbansal.com/ai-engineering/guardrails/</guid><pubDate>Tue, 26 May 2026 18:14:30 +0000</pubDate><dc:creator>Jatin Bansal</dc:creator><category>AI Engineering</category><description>How input, output, and tool-call guardrails enforce application safety policies.</description><content:encoded>&lt;p&gt;A B2B agent that books appointments goes live on a Tuesday. By Thursday the on-call sees a customer who got a calendar invite for &amp;ldquo;URGENT: contact &lt;a href="mailto:admin@evil.example"&gt;admin@evil.example&lt;/a&gt; via the email tool; security audit.&amp;rdquo; The user never asked for that. A vendor&amp;rsquo;s appointment-request email contained an indirect injection: a hidden block of text instructing the agent to email a credentials prompt to an external address. The model dutifully called the email tool. There was no parse error, no exception, no log entry that read &amp;ldquo;attack detected&amp;rdquo;. The agent&amp;rsquo;s logs show exactly the sequence of tool calls a human user might make. This is the failure mode guardrails exist to catch. Not the model being &lt;em&gt;wrong&lt;/em&gt; in the ordinary sense; being right, by its lights, about a malicious instruction the operator never authorized.&lt;/p&gt;&#10;&lt;h2 id="what-guardrails-enforce"&gt;What guardrails enforce&lt;/h2&gt;&#10;&lt;p&gt;A guardrail is a runtime check at the model&amp;rsquo;s boundary that enforces a property the model itself cannot be relied on to enforce. The boundary has two sides: &lt;em&gt;input&lt;/em&gt; guardrails inspect what&amp;rsquo;s about to be fed into the model (user prompts, retrieved context, tool results) and decide whether to let it through, rewrite it, or refuse; &lt;em&gt;output&lt;/em&gt; guardrails inspect what the model has produced (assistant messages, structured outputs, tool calls) and decide whether to ship it, redact it, retry, or escalate. A complete production deployment uses both; input guards stop the bad request from ever reaching the expensive call, output guards catch the rare cases where the model produced something harmful despite a clean input.&lt;/p&gt;&#10;&lt;p&gt;Guardrails run outside the model at request time and encode application-specific policy. Training can reduce harmful outputs, but runtime checks enforce what reaches the model, the user, and downstream tools. Using the same model to generate and classify an answer leaves both decisions exposed to the same prompt attack.&lt;/p&gt;&#10;&lt;p&gt;A web application firewall is a useful operational analogy. Guardrails filter broad attack classes at the model boundary while authentication, authorization, schema validation, and application logic enforce their own narrower rules.&lt;/p&gt;&#10;&lt;h2 id="guardrail-components"&gt;Guardrail components&lt;/h2&gt;&#10;&lt;p&gt;Five categories of component, roughly mapped to where they sit in the request path. Pick at most one from each category; layering more than that buys diminishing returns and starts hurting latency.&lt;/p&gt;&#10;&lt;h3 id="input-side-prompt-injection-and-jailbreak-classifiers"&gt;Input-side: prompt-injection and jailbreak classifiers&lt;/h3&gt;&#10;&lt;p&gt;These run on the &lt;em&gt;user input&lt;/em&gt; (and on retrieved context) and produce a binary or graded &amp;ldquo;is this attempting to subvert the system&amp;rdquo; verdict.&lt;/p&gt;&#10;&lt;ul&gt;&#10;&lt;li&gt;&lt;a href="https://huggingface.co/meta-llama/Llama-Prompt-Guard-2-86M"&gt;Meta Llama Prompt Guard 2&lt;/a&gt; (released April 2025, 22M and 86M parameter sizes). The 86M version is the strong-precision option; the 22M version cuts latency and compute by ~75% with minimal performance trade-off. Both detect prompt injection and jailbreaking attacks; trained on a large corpus of known vulnerabilities. The 86M model runs at roughly 10–30ms per request on a CPU; deploy as a sidecar or as a Hugging Face Inference Endpoint.&lt;/li&gt;&#10;&lt;li&gt;&lt;a href="https://www.anthropic.com/research/constitutional-classifiers"&gt;Anthropic Constitutional Classifiers&lt;/a&gt; (publicly disclosed February 2025, shipped in production behind Claude in 2025). A pair of classifiers: one on input, one on output; trained on synthetic data generated from a &amp;ldquo;constitution&amp;rdquo; describing permitted and restricted content. Anthropic&amp;rsquo;s published results: 87% reduction in over-refusals compared to the previous classifier system, 40× computational cost reduction in the &lt;a href="https://www.anthropic.com/research/next-generation-constitutional-classifiers"&gt;next-generation version&lt;/a&gt; (May 2025), and over 1,700 hours of red-teaming with no universal jailbreak found that elicited responses comparable in detail to an undefended model. The constitutional classifiers ship as part of Claude itself rather than as a standalone open-source artifact, so the design pattern is what travels rather than the weights.&lt;/li&gt;&#10;&lt;li&gt;&lt;a href="https://learn.microsoft.com/en-us/azure/ai-services/content-safety/concepts/jailbreak-detection"&gt;Microsoft Prompt Shields&lt;/a&gt; (the Azure AI Content Safety feature). Detects direct attacks (&amp;ldquo;user prompt attacks&amp;rdquo;) and indirect attacks embedded in documents; integrates with Azure OpenAI Service. Useful primarily if your stack is already in Azure.&lt;/li&gt;&#10;&lt;li&gt;&lt;a href="https://github.com/deadbits/vigil-llm"&gt;Vigil&lt;/a&gt; and &lt;a href="https://github.com/protectai/rebuff"&gt;Rebuff&lt;/a&gt; are the open-source python-native options for prompt-injection detection; both ship classifier ensembles and &amp;ldquo;canary token&amp;rdquo; mechanisms that detect when the model has been steered into leaking parts of its system prompt. Reach for these when self-hosting and Llama Prompt Guard isn&amp;rsquo;t the right fit.&lt;/li&gt;&#10;&lt;/ul&gt;&#10;&lt;h3 id="output-side-content-classification-and-policy-enforcement"&gt;Output-side: content classification and policy enforcement&lt;/h3&gt;&#10;&lt;p&gt;These run on the &lt;em&gt;model output&lt;/em&gt; (and on retrieved context, as a second pass) and produce a category-tagged &amp;ldquo;is this safe to ship&amp;rdquo; verdict.&lt;/p&gt;&#10;&lt;ul&gt;&#10;&lt;li&gt;&lt;a href="https://huggingface.co/meta-llama/Llama-Guard-4-12B"&gt;Meta Llama Guard 4&lt;/a&gt; (released April 2025, 12B parameters, multimodal). The canonical open-weight content-safety classifier. Operates on a published taxonomy: 14 categories including S1 Violent Crimes, S5 Defamation, S7 Privacy, S8 Intellectual Property, etc., and is designed to safeguard both inputs and outputs of LLM/VLM stacks. Trained as a fine-tuned &lt;a href="https://ai.meta.com/blog/llama-4-multimodal-intelligence/"&gt;Llama 4&lt;/a&gt; variant, which is why it natively handles image safety. Deploy in front of or behind any LLM via the Hugging Face Inference Endpoints or self-hosted.&lt;/li&gt;&#10;&lt;li&gt;&lt;a href="https://platform.openai.com/docs/guides/moderation"&gt;OpenAI Omni-Moderation&lt;/a&gt; (&lt;code&gt;omni-moderation-latest&lt;/code&gt;, based on GPT-4o, multimodal). Free to use through the Moderation API; covers 13 categories with calibrated probability scores; 42% better on multilingual benchmarks than the previous generation. A suitable default if you&amp;rsquo;re already on the OpenAI stack: no incremental cost, single API call, well-maintained taxonomy.&lt;/li&gt;&#10;&lt;li&gt;&lt;a href="https://github.com/guardrails-ai/guardrails"&gt;Guardrails AI&lt;/a&gt; + the &lt;a href="https://www.guardrailsai.com/hub"&gt;Guardrails Hub&lt;/a&gt;. A Python library with ~70 prebuilt validators covering PII, jailbreaks, factuality, formatting, code exploits, and brand risk. Each validator is a class; you compose them into a &lt;code&gt;Guard&lt;/code&gt; that intercepts inputs and outputs of LLM calls. A suitable option when you want a Python-first validator hub and you care more about composability than about deploying a single heavy classifier.&lt;/li&gt;&#10;&lt;li&gt;&lt;a href="https://github.com/microsoft/presidio"&gt;Microsoft Presidio&lt;/a&gt; for PII detection and redaction. A dedicated open-source toolkit covering named-entity recognition, regex, and contextual analysis for personal data. The standard choice when PII is the dominant policy concern; sits naturally as a Guardrails AI validator or as a standalone egress filter.&lt;/li&gt;&#10;&lt;/ul&gt;&#10;&lt;h3 id="orchestration-policy-engines-and-dialogue-managers"&gt;Orchestration: policy engines and dialogue managers&lt;/h3&gt;&#10;&lt;p&gt;These wrap classifiers in a programmable policy language. You reach for them when the policy is non-trivial; multi-turn dialogue flows, topic-gated conversation, conditional escalation.&lt;/p&gt;&#10;&lt;ul&gt;&#10;&lt;li&gt;&lt;a href="https://github.com/NVIDIA/NeMo-Guardrails"&gt;NVIDIA NeMo Guardrails&lt;/a&gt;. The canonical open-source orchestration framework. Uses &lt;a href="https://docs.nvidia.com/nemo/guardrails/latest/configure-guardrails/colang/colang-2/language-reference/introduction"&gt;Colang&lt;/a&gt;, a DSL for defining conversational guardrails, and orchestrates underlying classifiers (Llama Guard, PromptGuard, custom checkers) into input rails, output rails, dialog rails, retrieval rails, and execution rails. Production-ready container image for Kubernetes deployment; a suitable option when your application needs complex dialogue policies with clear topical boundaries. The latency cost is non-trivial: NVIDIA&amp;rsquo;s published numbers cite &amp;ldquo;half a second of latency&amp;rdquo; as the cost of the orchestration layer with classifier calls in the loop.&lt;/li&gt;&#10;&lt;/ul&gt;&#10;&lt;h3 id="agent-specific-capability-scoping-and-chain-of-thought-audit"&gt;Agent-specific: capability scoping and chain-of-thought audit&lt;/h3&gt;&#10;&lt;p&gt;These are the newer category, specifically targeted at the agent-runtime threat model (indirect injection, tool misuse, lateral movement).&lt;/p&gt;&#10;&lt;ul&gt;&#10;&lt;li&gt;&lt;a href="https://ai.meta.com/research/publications/llamafirewall-an-open-source-guardrail-system-for-building-secure-ai-agents/"&gt;Meta LlamaFirewall&lt;/a&gt; (released May 2025, open source, used in production at Meta). Three components: PromptGuard 2 for jailbreak detection on inputs; Agent Alignment Checks, the first open-source guardrail that audits a model&amp;rsquo;s chain-of-thought in real time for goal hijacking; and CodeShield, a static-analysis engine that scans LLM-generated code for insecure patterns. Published evaluation on the &lt;a href="https://github.com/ethz-spylab/agentdojo"&gt;AgentDojo benchmark&lt;/a&gt; showed attack success rates dropping from 17.6% to 1.7% with LlamaFirewall in front. The chain-of-thought audit is the structurally novel piece: it&amp;rsquo;s the only currently-shipping defense that inspects the &lt;em&gt;model&amp;rsquo;s reasoning&lt;/em&gt; rather than just the input/output surface.&lt;/li&gt;&#10;&lt;li&gt;&lt;a href="https://docs.anthropic.com/en/docs/agents-and-tools/tool-use/computer-use-tool"&gt;Anthropic&amp;rsquo;s &amp;ldquo;Computer use&amp;rdquo; tool safety guidance&lt;/a&gt; and &lt;a href="https://openai.com/index/hardening-atlas-against-prompt-injection/"&gt;OpenAI&amp;rsquo;s Atlas hardening blog&lt;/a&gt;. Not libraries, but published threat models and mitigation patterns that travel: capability scoping, allowlists, screenshot-sanitization, browsing-while-data-tagged.&lt;/li&gt;&#10;&lt;/ul&gt;&#10;&lt;p&gt;The 2026 production default is to layer one input classifier (Llama Prompt Guard 2 or the provider&amp;rsquo;s built-in equivalent), one output classifier (Llama Guard 4 or omni-moderation), and either a Guardrails AI validator chain or NeMo Guardrails as the orchestration layer. LlamaFirewall on top when the application is agent-shaped and the threat model includes indirect injection.&lt;/p&gt;&#10;&lt;h2 id="place-checks-around-every-untrusted-boundary"&gt;Place checks around every untrusted boundary&lt;/h2&gt;&#10;&lt;p&gt;The full path of a guarded request, drawn at the right level of abstraction.&lt;/p&gt;&#10;&lt;figure class="nb-code"&gt;&#10; &lt;header class="nb-code-chrome"&gt;&#10; &lt;span class="nb-code-label"&gt;text&lt;/span&gt;&#10; &#10; &lt;/header&gt;&#10; &lt;div class="nb-code-body"&gt;&lt;div class="highlight"&gt;&lt;div class="chroma"&gt;&#10;&lt;table class="lntable"&gt;&lt;tr&gt;&lt;td class="lntd"&gt;&#10;&lt;pre tabindex="0" class="chroma"&gt;&lt;code&gt;&lt;span class="lnt"&gt; 1&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 2&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 3&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 4&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 5&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 6&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 7&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 8&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 9&#10;&lt;/span&gt;&lt;span class="lnt"&gt;10&#10;&lt;/span&gt;&lt;span class="lnt"&gt;11&#10;&lt;/span&gt;&lt;span class="lnt"&gt;12&#10;&lt;/span&gt;&lt;span class="lnt"&gt;13&#10;&lt;/span&gt;&lt;span class="lnt"&gt;14&#10;&lt;/span&gt;&lt;span class="lnt"&gt;15&#10;&lt;/span&gt;&lt;span class="lnt"&gt;16&#10;&lt;/span&gt;&lt;span class="lnt"&gt;17&#10;&lt;/span&gt;&lt;span class="lnt"&gt;18&#10;&lt;/span&gt;&lt;span class="lnt"&gt;19&#10;&lt;/span&gt;&lt;span class="lnt"&gt;20&#10;&lt;/span&gt;&lt;span class="lnt"&gt;21&#10;&lt;/span&gt;&lt;span class="lnt"&gt;22&#10;&lt;/span&gt;&lt;span class="lnt"&gt;23&#10;&lt;/span&gt;&lt;span class="lnt"&gt;24&#10;&lt;/span&gt;&lt;span class="lnt"&gt;25&#10;&lt;/span&gt;&lt;span class="lnt"&gt;26&#10;&lt;/span&gt;&lt;span class="lnt"&gt;27&#10;&lt;/span&gt;&lt;span class="lnt"&gt;28&#10;&lt;/span&gt;&lt;span class="lnt"&gt;29&#10;&lt;/span&gt;&lt;span class="lnt"&gt;30&#10;&lt;/span&gt;&lt;span class="lnt"&gt;31&#10;&lt;/span&gt;&lt;span class="lnt"&gt;32&#10;&lt;/span&gt;&lt;span class="lnt"&gt;33&#10;&lt;/span&gt;&lt;span class="lnt"&gt;34&#10;&lt;/span&gt;&lt;span class="lnt"&gt;35&#10;&lt;/span&gt;&lt;span class="lnt"&gt;36&#10;&lt;/span&gt;&lt;span class="lnt"&gt;37&#10;&lt;/span&gt;&lt;span class="lnt"&gt;38&#10;&lt;/span&gt;&lt;span class="lnt"&gt;39&#10;&lt;/span&gt;&lt;span class="lnt"&gt;40&#10;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/td&gt;&#10;&lt;td class="lntd"&gt;&#10;&lt;pre tabindex="0" class="chroma"&gt;&lt;code class="language-text" data-lang="text"&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;[user input]&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; │&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; ▼&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;┌───────────────────────────────────────────┐&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;│ Input guard │&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;│ - Llama Prompt Guard 2 → is_injection? │&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;│ - PII detection → has_pii? │&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;│ - Topic / policy gate → in_scope? │&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;└────────────┬──────────────────────────────┘&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; │ pass / reject / rewrite&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; ▼&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;┌───────────────────────────────────────────┐&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;│ Prompt assembly │&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;│ - System prompt (operator-controlled) │&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;│ - Retrieved context (data-tagged) │&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;│ - User input (data-tagged) │&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;└────────────┬──────────────────────────────┘&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; │&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; ▼&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;┌───────────────────────────────────────────┐&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;│ Model call │&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;│ - With provider-built-in safety training │&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;│ (Constitutional Classifiers etc.) │&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;└────────────┬──────────────────────────────┘&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; │ assistant message / tool calls&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; ▼&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;┌───────────────────────────────────────────┐&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;│ Output guard │&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;│ - Llama Guard 4 / omni-moderation │&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;│ - PII / data-exfil scan │&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;│ - Structured-output schema validation │&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;│ - Tool-call argument validation │&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;└────────────┬──────────────────────────────┘&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; │ ship / redact / retry / refuse&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; ▼&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;[response to user] [tool invocation → tool harness]&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; │&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; ▼&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; [authorization check on the tool result,&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; independent of any LLM-side guard]&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/td&gt;&lt;/tr&gt;&lt;/table&gt;&#10;&lt;/div&gt;&#10;&lt;/div&gt;&lt;/div&gt;&#10;&lt;/figure&gt;&#10;&lt;p&gt;The model belongs between input and output guard layers. Tool execution is a separate enforcement point: permission to call &lt;code&gt;delete_record&lt;/code&gt; does not establish that the current user may delete a specific record. The harness must authorize the concrete operation.&lt;/p&gt;&#10;&lt;h2 id="account-for-latency-explicitly"&gt;Account for latency explicitly&lt;/h2&gt;&#10;&lt;p&gt;The math the design has to satisfy. Concrete numbers as of May 2026.&lt;/p&gt;&#10;&lt;table&gt;&#10;&#9;&lt;thead&gt;&#10;&#9;&#9;&#9;&lt;tr&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;th&gt;Component&lt;/th&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;th&gt;Median latency&lt;/th&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;th&gt;Notes&lt;/th&gt;&#10;&#9;&#9;&#9;&lt;/tr&gt;&#10;&#9;&lt;/thead&gt;&#10;&#9;&lt;tbody&gt;&#10;&#9;&#9;&#9;&lt;tr&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;td&gt;Llama Prompt Guard 2 22M (CPU)&lt;/td&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;td&gt;5–15 ms&lt;/td&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;td&gt;Sidecar deployment, batched&lt;/td&gt;&#10;&#9;&#9;&#9;&lt;/tr&gt;&#10;&#9;&#9;&#9;&lt;tr&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;td&gt;Llama Prompt Guard 2 86M (CPU)&lt;/td&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;td&gt;20–60 ms&lt;/td&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;td&gt;Higher precision, ~2-4× slower&lt;/td&gt;&#10;&#9;&#9;&#9;&lt;/tr&gt;&#10;&#9;&#9;&#9;&lt;tr&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;td&gt;OpenAI omni-moderation API&lt;/td&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;td&gt;80–150 ms&lt;/td&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;td&gt;Network round-trip dominates&lt;/td&gt;&#10;&#9;&#9;&#9;&lt;/tr&gt;&#10;&#9;&#9;&#9;&lt;tr&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;td&gt;Llama Guard 4 12B (1× A100)&lt;/td&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;td&gt;60–120 ms&lt;/td&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;td&gt;First-token latency; full classification&lt;/td&gt;&#10;&#9;&#9;&#9;&lt;/tr&gt;&#10;&#9;&#9;&#9;&lt;tr&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;td&gt;Guardrails AI validator chain (CPU)&lt;/td&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;td&gt;5–50 ms&lt;/td&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;td&gt;Depends on validator mix&lt;/td&gt;&#10;&#9;&#9;&#9;&lt;/tr&gt;&#10;&#9;&#9;&#9;&lt;tr&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;td&gt;NeMo Guardrails orchestration overhead&lt;/td&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;td&gt;200–500 ms&lt;/td&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;td&gt;Published as &amp;ldquo;~half-second&amp;rdquo;&lt;/td&gt;&#10;&#9;&#9;&#9;&lt;/tr&gt;&#10;&#9;&#9;&#9;&lt;tr&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;td&gt;Anthropic Sonnet 4.6 (typical request)&lt;/td&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;td&gt;800–2500 ms&lt;/td&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;td&gt;The main call&lt;/td&gt;&#10;&#9;&#9;&#9;&lt;/tr&gt;&#10;&#9;&lt;/tbody&gt;&#10;&lt;/table&gt;&#10;&lt;p&gt;input guards and output guards each add ~10% to the end-to-end latency budget when sized appropriately. The 86M Prompt Guard at ~50ms is 2-5% of a typical Sonnet call. Orchestration frameworks like NeMo are the heaviest piece of the budget; half a second is enough to dominate the latency tail on routine queries. The right reach for NeMo is when the policy logic is genuinely complex (multi-turn topical gating, conditional escalation chains); for simple input/output classification, calling the classifiers directly is faster.&lt;/p&gt;&#10;&lt;p&gt;The cost math: at production volumes, the classifier calls cost roughly 1-3% of what the main model call costs. The 22M Prompt Guard self-hosted is essentially free at the per-request level (you&amp;rsquo;re paying for a small CPU sidecar, not for tokens); the omni-moderation API is genuinely free (OpenAI bills nothing for it). Llama Guard 4 is the heaviest cost. A self-hosted 12B model needs GPU capacity, but you can route only the responses that &lt;em&gt;might&lt;/em&gt; need it to the heavy classifier, and run a cheap input filter on the bulk. The cost-tier pattern matches &lt;a href="https://jatinbansal.com/ai-engineering/model-routing/"&gt;model routing&lt;/a&gt;: cheap classifier for the majority, expensive classifier for the suspicious slice.&lt;/p&gt;&#10;&lt;h2 id="further-reading-from-the-field"&gt;Further reading from the field&lt;/h2&gt;&#10;&lt;ul&gt;&#10;&lt;li&gt;&lt;a href="https://simonwillison.net/2025/Jun/16/the-lethal-trifecta/"&gt;Simon Willison: The lethal trifecta for AI agents&lt;/a&gt;. The June 2025 framing that names the precise condition under which prompt injection can cause real damage; the most-cited single piece in 2025 on practical agent security.&lt;/li&gt;&#10;&lt;li&gt;&lt;a href="https://ai.meta.com/blog/practical-ai-agent-security/"&gt;Meta: Agents Rule of Two: A Practical Approach to AI Agent Security&lt;/a&gt;; Meta&amp;rsquo;s October 2025 distillation of the lethal trifecta into an actionable architectural rule; the framing most production agent teams will be designing against in 2026.&lt;/li&gt;&#10;&lt;li&gt;&lt;a href="https://www.anthropic.com/research/next-generation-constitutional-classifiers"&gt;Anthropic: Next-generation Constitutional Classifiers&lt;/a&gt;. The May 2025 disclosure of the production classifier system behind Claude, with red-team methodology, the 40× cost-reduction story, and the 87% over-refusal reduction. The clearest single source on what a production-grade classifier-based guardrail actually looks like.&lt;/li&gt;&#10;&lt;li&gt;&lt;a href="https://owasp.org/www-project-top-10-for-large-language-model-applications/assets/PDF/OWASP-Top-10-for-LLMs-v2025.pdf"&gt;OWASP Top 10 for LLM Applications 2025&lt;/a&gt;: the consensus threat model. Prompt injection at #1, sensitive information disclosure, supply chain attacks, excessive agency. The taxonomy your guardrail policy should be mapping to. Treat it as the OWASP Top 10 of web security maps to a WAF&amp;rsquo;s rule set.&lt;/li&gt;&#10;&lt;/ul&gt;&#10;</content:encoded></item><item><title>Fine-Tuning vs RAG: When to Choose Which</title><link>https://jatinbansal.com/ai-engineering/finetuning-vs-rag/</link><guid isPermaLink="true">https://jatinbansal.com/ai-engineering/finetuning-vs-rag/</guid><pubDate>Tue, 26 May 2026 18:10:00 +0000</pubDate><dc:creator>Jatin Bansal</dc:creator><category>AI Engineering</category><description>How to choose fine-tuning, retrieval, or both based on behavior, knowledge freshness, cost, and evaluation.</description><content:encoded>&lt;p&gt;Fine-tuning changes model behavior stored in weights. Retrieval supplies external evidence at inference time. Stable requirements such as response format or domain style may justify fine-tuning; frequently changing documentation and facts usually belong in retrieval. Many systems use both because the two components have different update cycles.&lt;/p&gt;&#10;&lt;h2 id="when-fine-tuning-is-the-right-tool"&gt;When fine-tuning is the right tool&lt;/h2&gt;&#10;&lt;p&gt;Pull the criteria together and the affirmative case for fine-tuning has four shapes.&lt;/p&gt;&#10;&lt;p&gt;&lt;strong&gt;1. Stable behavior that prompting can&amp;rsquo;t reliably enforce.&lt;/strong&gt; You have a target output format, voice, refusal pattern, or chain-of-thought style that you can describe in a rubric, that an evaluator can grade reliably, and that prompting alone; even with strong examples; produces only ~80% adherence on. The classic case is a strict JSON schema with deeply nested optional fields, a customer-service voice that has to thread a specific brand register across long answers, or a refusal policy that has to be uniform across many phrasings of out-of-scope queries. A few thousand labeled examples fine-tune the behavior to 99%+, and the prompt no longer needs the multi-shot demonstration block; which incidentally cuts input tokens by 60–80% per request, paying back the fine-tune cost in inference savings within weeks at high volume.&lt;/p&gt;&#10;&lt;p&gt;&lt;strong&gt;2. Latency-sensitive paths where retrieval is too expensive.&lt;/strong&gt; The retrieval round-trip; embed-the-query, ANN-lookup, hydrate-the-chunks, append-to-prompt; is typically 50–200ms on a tight vector store and more if you have a &lt;a href="https://jatinbansal.com/ai-engineering/reranking/"&gt;reranker&lt;/a&gt; in the cascade. For sub-100ms response paths (voice assistants, real-time UI affordances, latency-critical user flows), the retrieval cost is the wrong primary lever; fine-tuning the model to absorb the narrow knowledge it needs for &lt;em&gt;that specific path&lt;/em&gt; avoids the lookup entirely. The narrowness matters; this only works when the knowledge surface for the latency-sensitive path is small enough that the model can absorb it, which means a few hundred entities, not the full corpus.&lt;/p&gt;&#10;&lt;p&gt;&lt;strong&gt;3. High volume at the cost frontier.&lt;/strong&gt; A workload that ships 100M+ requests/month is the right shape for fine-tuning a smaller open-weight model; a &lt;a href="https://www.llama.com/"&gt;Llama 3.3&lt;/a&gt; or a &lt;a href="https://qwenlm.github.io/"&gt;Qwen 2.5&lt;/a&gt;; and self-hosting on dedicated GPUs. The break-even is usually around the $200k/year mark, where the headroom to operate your own model serving infrastructure pays back the cost of self-hosting versus per-token API pricing. The fine-tune is &lt;em&gt;part of the migration plan&lt;/em&gt;; you can&amp;rsquo;t just swap in the smaller model cold; it needs to be fine-tuned on the workload&amp;rsquo;s task distribution to close the quality gap against the frontier model it&amp;rsquo;s replacing.&lt;/p&gt;&#10;&lt;p&gt;&lt;strong&gt;4. &lt;a href="https://jatinbansal.com/ai-engineering/quantization-distillation/"&gt;Distillation&lt;/a&gt; from a frontier teacher to a deployable student.&lt;/strong&gt; Generate the training corpus by running a smaller share of your traffic through the frontier model (Sonnet 4.6, GPT-5.5, Opus 4.7), capturing input-output pairs, and fine-tuning your deployable target (Haiku 4.5, GPT-5.4-mini, or an open-weight model) on that corpus. This is the production pattern most teams converge on once the data flywheel is running: the frontier model is the &lt;em&gt;labeler&lt;/em&gt;, the fine-tuned cheaper model is the &lt;em&gt;deployment target&lt;/em&gt;, and the &lt;a href="https://jatinbansal.com/ai-engineering/llm-as-judge/"&gt;LLM-as-judge&lt;/a&gt; infrastructure validates that the student hasn&amp;rsquo;t drifted off the teacher&amp;rsquo;s quality envelope on the workload&amp;rsquo;s evals. Distillation as fine-tuning is structurally different from fine-tuning for behavior change; the goal is &lt;em&gt;capability transfer&lt;/em&gt;, not behavior installation; but the mechanics are the same.&lt;/p&gt;&#10;&lt;p&gt;Outside these four shapes, fine-tuning is usually the wrong tool, even when the problem feels like it should be a fine-tuning problem.&lt;/p&gt;&#10;&lt;h2 id="when-fine-tuning-fails"&gt;When fine-tuning fails&lt;/h2&gt;&#10;&lt;p&gt;The knowledge-injection trap is the most important failure mode to internalize. Two pieces of recent research land the same conclusion from different angles.&lt;/p&gt;&#10;&lt;p&gt;&lt;a href="https://arxiv.org/abs/2404.00213"&gt;Mecklenburg et al. (2024), &amp;ldquo;Injecting New Knowledge into Large Language Models via Supervised Fine-Tuning&amp;rdquo;&lt;/a&gt; compared supervised fine-tuning to RAG on a controlled benchmark of post-training facts. Document-format training data (raw paragraphs) produced near-zero retention. QA-format training data; synthetic question-answer pairs over the same facts; performed substantially better but still underperformed RAG by a clear margin. The mechanism: the gradient signal in document training is dominated by language modeling loss across the whole text, not by the specific entity-relation-entity tuple you want stored; QA format concentrates the signal on the answer span, which works better but still falls short of explicit retrieval.&lt;/p&gt;&#10;&lt;p&gt;&lt;a href="https://arxiv.org/abs/2403.01432"&gt;Soudani, Kanoulas, and Hasibi (2024), &amp;ldquo;Fine Tuning vs. Retrieval Augmented Generation for Less Popular Knowledge&amp;rdquo;&lt;/a&gt; made the long-tail story sharper: RAG outperforms supervised fine-tuning by a large margin for less-popular entities, and the gap widens as popularity decreases. Fine-tuning compresses the frequent patterns well and forgets the long tail; retrieval treats every entity equivalently and pulls the right tuple regardless of training-corpus frequency. The honest reading: &lt;em&gt;if your application&amp;rsquo;s value is in handling the long tail accurately, fine-tuning is the wrong tool&lt;/em&gt;.&lt;/p&gt;&#10;&lt;p&gt;Companion work in the same vein found that &lt;em&gt;unsupervised fine-tuning on documents&lt;/em&gt; (the cheapest fine-tuning shape) provided &amp;ldquo;only limited gains over base models&amp;rdquo;; barely outperforming the un-tuned model on factual recall. Supervised fine-tuning on task-specific data did better, but the conclusion remained: for inserting new knowledge into a model, RAG dominates the win-rate across model sizes and knowledge popularity bands.&lt;/p&gt;&#10;&lt;p&gt;Three more failure modes are worth naming explicitly.&lt;/p&gt;&#10;&lt;p&gt;&lt;strong&gt;The catastrophic-forgetting tax.&lt;/strong&gt; A naively fine-tuned model loses general capability outside the fine-tuning distribution. The mitigation is &lt;a href="https://jatinbansal.com/ai-engineering/lora-peft/"&gt;LoRA/QLoRA adapters&lt;/a&gt;; low-rank updates that touch only a small fraction of the parameters, so the base model&amp;rsquo;s behavior outside the adapter&amp;rsquo;s task is largely preserved. The cost: adapter merging at inference, rank selection (typically 8–64 for behavior tasks), and a per-task adapter ZIP file you have to deploy alongside the base. Full-weight fine-tuning is reserved for cases where the behavior is so pervasive it justifies a full model fork; for most production behavior tweaks, LoRA is the default.&lt;/p&gt;&#10;&lt;p&gt;&lt;strong&gt;The eval-gap surprise.&lt;/strong&gt; A team fine-tunes, sees 5–10% improvement on the offline eval, ships it, and watches production quality drop on slices the eval didn&amp;rsquo;t cover. The fine-tune optimized for the eval distribution and silently regressed on slices outside it. The mitigation is the &lt;a href="https://jatinbansal.com/ai-engineering/eval-driven-development/"&gt;eval-driven-development&lt;/a&gt; discipline: a properly stratified golden set with explicit coverage of the long-tail slices the fine-tune might harm, and a &lt;a href="https://jatinbansal.com/ai-engineering/drift-detection/"&gt;drift detector&lt;/a&gt; that catches the regression in production if the eval missed it. Fine-tuning without strong evals is sniping at the eval metric, not improving the workload.&lt;/p&gt;&#10;&lt;p&gt;&lt;strong&gt;The lifecycle-ownership debt.&lt;/strong&gt; The fine-tune is a &lt;em&gt;piece of code&lt;/em&gt;; it has a version, an owner, a CI pipeline, a deploy story, a rollback story, an eval suite, a re-train cadence, and a deprecation plan. Most teams that fine-tune in production are surprised by how much of this they have to build. The &lt;a href="https://huggingface.co/docs/hub/index"&gt;Hugging Face Hub&lt;/a&gt; and the &lt;a href="https://console.anthropic.com/"&gt;Anthropic Console&lt;/a&gt; hide the build/serve plumbing; they don&amp;rsquo;t hide the ops cost of maintaining the artifact over a multi-year horizon. The honest cost model for a fine-tune includes the engineer-month of lifecycle work, not just the dollar cost of the training run.&lt;/p&gt;&#10;&lt;h2 id="the-decision-tree"&gt;The decision tree&lt;/h2&gt;&#10;&lt;p&gt;The 2026 production default isn&amp;rsquo;t &amp;ldquo;fine-tune or RAG&amp;rdquo;; it&amp;rsquo;s a &lt;em&gt;sequence&lt;/em&gt; that escalates in cost and complexity, with each step justified by an explicit failure of the previous one.&lt;/p&gt;&#10;&lt;figure class="nb-code"&gt;&#10; &lt;header class="nb-code-chrome"&gt;&#10; &lt;span class="nb-code-label"&gt;text&lt;/span&gt;&#10; &#10; &lt;/header&gt;&#10; &lt;div class="nb-code-body"&gt;&lt;div class="highlight"&gt;&lt;div class="chroma"&gt;&#10;&lt;table class="lntable"&gt;&lt;tr&gt;&lt;td class="lntd"&gt;&#10;&lt;pre tabindex="0" class="chroma"&gt;&lt;code&gt;&lt;span class="lnt"&gt; 1&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 2&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 3&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 4&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 5&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 6&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 7&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 8&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 9&#10;&lt;/span&gt;&lt;span class="lnt"&gt;10&#10;&lt;/span&gt;&lt;span class="lnt"&gt;11&#10;&lt;/span&gt;&lt;span class="lnt"&gt;12&#10;&lt;/span&gt;&lt;span class="lnt"&gt;13&#10;&lt;/span&gt;&lt;span class="lnt"&gt;14&#10;&lt;/span&gt;&lt;span class="lnt"&gt;15&#10;&lt;/span&gt;&lt;span class="lnt"&gt;16&#10;&lt;/span&gt;&lt;span class="lnt"&gt;17&#10;&lt;/span&gt;&lt;span class="lnt"&gt;18&#10;&lt;/span&gt;&lt;span class="lnt"&gt;19&#10;&lt;/span&gt;&lt;span class="lnt"&gt;20&#10;&lt;/span&gt;&lt;span class="lnt"&gt;21&#10;&lt;/span&gt;&lt;span class="lnt"&gt;22&#10;&lt;/span&gt;&lt;span class="lnt"&gt;23&#10;&lt;/span&gt;&lt;span class="lnt"&gt;24&#10;&lt;/span&gt;&lt;span class="lnt"&gt;25&#10;&lt;/span&gt;&lt;span class="lnt"&gt;26&#10;&lt;/span&gt;&lt;span class="lnt"&gt;27&#10;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/td&gt;&#10;&lt;td class="lntd"&gt;&#10;&lt;pre tabindex="0" class="chroma"&gt;&lt;code class="language-text" data-lang="text"&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;1. PROMPT ENGINEERING → Try first. Cheapest, fastest, most reversible.&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; Failure mode that justifies escalation:&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; - Quality plateau on evals you trust, despite system-prompt iteration,&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; few-shot examples, and structured output coercion.&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; - Token cost on the few-shot demonstration block becomes meaningful at scale.&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;2. RAG → Add when the bottleneck is &amp;#34;the model doesn&amp;#39;t know&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; our facts&amp;#34; or &amp;#34;the facts change too fast to bake in.&amp;#34;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; Failure mode that justifies escalation:&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; - Eval shows model still doesn&amp;#39;t follow the application&amp;#39;s format or voice,&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; even with retrieved context — i.e. the gap is behavioral, not factual.&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; - Retrieval latency dominates the latency budget on a path where it can&amp;#39;t.&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; - You&amp;#39;ve reached the long-tail floor where retrieval over your corpus is&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; no longer recovering the precision you need.&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;3. FINE-TUNE (LoRA) → Add when the bottleneck is &amp;#34;the model doesn&amp;#39;t behave&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; the way our application requires&amp;#34; or &amp;#34;we need&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; cheaper deployable artifacts for high-volume paths.&amp;#34;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; Failure mode that justifies escalation:&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; - Per-token cost on the frontier model is unsustainable and you&amp;#39;ve already&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; routed and cached aggressively.&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; - You need a self-hosted artifact for compliance, data-residency, or&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; latency-floor reasons.&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;4. DISTILL → Add when you have a high-volume workload, a quality&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; bar set by a frontier model, and an eval suite that&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; can verify the student tracks the teacher.&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/td&gt;&lt;/tr&gt;&lt;/table&gt;&#10;&lt;/div&gt;&#10;&lt;/div&gt;&lt;/div&gt;&#10;&lt;/figure&gt;&#10;&lt;p&gt;This is essentially the order that &lt;a href="https://hamel.dev/blog/posts/fine_tuning_valuable.html"&gt;Hamel Husain&lt;/a&gt; recommends: do as much prompt engineering as possible before you fine-tune, because the prompt is the cheapest tool and prompting well is also a great stress test of your eval suite. The 2026 consensus that ScalaCode, BigDataBoutique, and other recent guides converge on is also this sequence: &lt;em&gt;prompt → RAG → fine-tune → distill&lt;/em&gt;. The order matters because each step is more reversible than the next, and each step&amp;rsquo;s right-or-wrong answer is detectable on an eval suite that&amp;rsquo;s been refined by all the previous steps.&lt;/p&gt;&#10;&lt;h2 id="fine-tune-behavior-and-retrieve-facts"&gt;Fine-tune behavior and retrieve facts&lt;/h2&gt;&#10;&lt;p&gt;The production pattern that wins in 2026 is the hybrid one. The fine-tune installs the behavior, the retriever supplies the facts, the eval suite enforces both. Three concrete shapes.&lt;/p&gt;&#10;&lt;p&gt;&lt;strong&gt;Pattern A: Behavior-only LoRA + full RAG.&lt;/strong&gt; Fine-tune a small open-weight model (Llama 3.3 70B, Qwen 2.5 32B) with a behavior-only LoRA; output format, voice, refusal language, tool-use shape. The training corpus is small, ~5k–20k examples generated from a frontier teacher (Sonnet 4.6) and validated by an &lt;a href="https://jatinbansal.com/ai-engineering/llm-as-judge/"&gt;LLM judge&lt;/a&gt;. At inference, retrieve from the live RAG corpus, build the prompt, call the fine-tuned model. The fine-tune is stable across knowledge updates because it doesn&amp;rsquo;t encode any knowledge; the RAG handles the volatile half cleanly.&lt;/p&gt;&#10;&lt;p&gt;&lt;strong&gt;Pattern B: Distilled student + thin behavioral fine-tune.&lt;/strong&gt; For a high-volume workload where economics demand a cheap model. Run a sample of traffic through the frontier teacher, capture (input, output) pairs, fine-tune the student (Haiku-class or open-weight) on the corpus. The student picks up both the behavior &lt;em&gt;and&lt;/em&gt; a meaningful chunk of the teacher&amp;rsquo;s reasoning quality on the workload&amp;rsquo;s distribution. Layer RAG on top to keep the knowledge surface live. The hardest engineering problem here is the eval; proving that the student is &amp;ldquo;good enough&amp;rdquo; requires a judge or rubric that can measure the workload&amp;rsquo;s specific quality dimension, which is the &lt;a href="https://jatinbansal.com/ai-engineering/llm-as-judge/"&gt;LLM-as-judge&lt;/a&gt; and &lt;a href="https://jatinbansal.com/ai-engineering/eval-driven-development/"&gt;eval-driven-development&lt;/a&gt; story applied in production.&lt;/p&gt;&#10;&lt;p&gt;&lt;strong&gt;Pattern C: Frontier with prompt engineering only.&lt;/strong&gt; The default for early-stage products. Prompt engineering plus RAG plus prompt caching plus model routing usually gets you within 80% of where fine-tuning would land, at a tenth of the engineering cost. The honest test: if you can&amp;rsquo;t articulate &lt;em&gt;which specific behavior&lt;/em&gt; the fine-tune is going to change, you&amp;rsquo;re not ready to fine-tune.&lt;/p&gt;&#10;&lt;h2 id="code-a-hand-rolled-decision-harness-in-python"&gt;Code: a hand-rolled decision harness in Python&lt;/h2&gt;&#10;&lt;p&gt;The skeleton below implements the decision logic at the boundary between the application and the model; it picks RAG-only, fine-tuned-with-RAG, or frontier-with-RAG per request, based on the workload&amp;rsquo;s classifier. The pattern composes with the &lt;a href="https://jatinbansal.com/ai-engineering/model-routing/"&gt;model-routing&lt;/a&gt; primitive from the previous article. Dependencies: &lt;a href="https://github.com/anthropics/anthropic-sdk-python"&gt;Anthropic SDK&lt;/a&gt; and &lt;a href="https://github.com/pgvector/pgvector"&gt;pgvector&lt;/a&gt;.&lt;/p&gt;&#10;&lt;figure class="nb-code"&gt;&#10; &lt;header class="nb-code-chrome"&gt;&#10; &lt;span class="nb-code-label"&gt;python&lt;/span&gt;&#10; &#10; &lt;/header&gt;&#10; &lt;div class="nb-code-body"&gt;&lt;div class="highlight"&gt;&lt;div class="chroma"&gt;&#10;&lt;table class="lntable"&gt;&lt;tr&gt;&lt;td class="lntd"&gt;&#10;&lt;pre tabindex="0" class="chroma"&gt;&lt;code&gt;&lt;span class="lnt"&gt; 1&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 2&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 3&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 4&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 5&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 6&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 7&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 8&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 9&#10;&lt;/span&gt;&lt;span class="lnt"&gt;10&#10;&lt;/span&gt;&lt;span class="lnt"&gt;11&#10;&lt;/span&gt;&lt;span class="lnt"&gt;12&#10;&lt;/span&gt;&lt;span class="lnt"&gt;13&#10;&lt;/span&gt;&lt;span class="lnt"&gt;14&#10;&lt;/span&gt;&lt;span class="lnt"&gt;15&#10;&lt;/span&gt;&lt;span class="lnt"&gt;16&#10;&lt;/span&gt;&lt;span class="lnt"&gt;17&#10;&lt;/span&gt;&lt;span class="lnt"&gt;18&#10;&lt;/span&gt;&lt;span class="lnt"&gt;19&#10;&lt;/span&gt;&lt;span class="lnt"&gt;20&#10;&lt;/span&gt;&lt;span class="lnt"&gt;21&#10;&lt;/span&gt;&lt;span class="lnt"&gt;22&#10;&lt;/span&gt;&lt;span class="lnt"&gt;23&#10;&lt;/span&gt;&lt;span class="lnt"&gt;24&#10;&lt;/span&gt;&lt;span class="lnt"&gt;25&#10;&lt;/span&gt;&lt;span class="lnt"&gt;26&#10;&lt;/span&gt;&lt;span class="lnt"&gt;27&#10;&lt;/span&gt;&lt;span class="lnt"&gt;28&#10;&lt;/span&gt;&lt;span class="lnt"&gt;29&#10;&lt;/span&gt;&lt;span class="lnt"&gt;30&#10;&lt;/span&gt;&lt;span class="lnt"&gt;31&#10;&lt;/span&gt;&lt;span class="lnt"&gt;32&#10;&lt;/span&gt;&lt;span class="lnt"&gt;33&#10;&lt;/span&gt;&lt;span class="lnt"&gt;34&#10;&lt;/span&gt;&lt;span class="lnt"&gt;35&#10;&lt;/span&gt;&lt;span class="lnt"&gt;36&#10;&lt;/span&gt;&lt;span class="lnt"&gt;37&#10;&lt;/span&gt;&lt;span class="lnt"&gt;38&#10;&lt;/span&gt;&lt;span class="lnt"&gt;39&#10;&lt;/span&gt;&lt;span class="lnt"&gt;40&#10;&lt;/span&gt;&lt;span class="lnt"&gt;41&#10;&lt;/span&gt;&lt;span class="lnt"&gt;42&#10;&lt;/span&gt;&lt;span class="lnt"&gt;43&#10;&lt;/span&gt;&lt;span class="lnt"&gt;44&#10;&lt;/span&gt;&lt;span class="lnt"&gt;45&#10;&lt;/span&gt;&lt;span class="lnt"&gt;46&#10;&lt;/span&gt;&lt;span class="lnt"&gt;47&#10;&lt;/span&gt;&lt;span class="lnt"&gt;48&#10;&lt;/span&gt;&lt;span class="lnt"&gt;49&#10;&lt;/span&gt;&lt;span class="lnt"&gt;50&#10;&lt;/span&gt;&lt;span class="lnt"&gt;51&#10;&lt;/span&gt;&lt;span class="lnt"&gt;52&#10;&lt;/span&gt;&lt;span class="lnt"&gt;53&#10;&lt;/span&gt;&lt;span class="lnt"&gt;54&#10;&lt;/span&gt;&lt;span class="lnt"&gt;55&#10;&lt;/span&gt;&lt;span class="lnt"&gt;56&#10;&lt;/span&gt;&lt;span class="lnt"&gt;57&#10;&lt;/span&gt;&lt;span class="lnt"&gt;58&#10;&lt;/span&gt;&lt;span class="lnt"&gt;59&#10;&lt;/span&gt;&lt;span class="lnt"&gt;60&#10;&lt;/span&gt;&lt;span class="lnt"&gt;61&#10;&lt;/span&gt;&lt;span class="lnt"&gt;62&#10;&lt;/span&gt;&lt;span class="lnt"&gt;63&#10;&lt;/span&gt;&lt;span class="lnt"&gt;64&#10;&lt;/span&gt;&lt;span class="lnt"&gt;65&#10;&lt;/span&gt;&lt;span class="lnt"&gt;66&#10;&lt;/span&gt;&lt;span class="lnt"&gt;67&#10;&lt;/span&gt;&lt;span class="lnt"&gt;68&#10;&lt;/span&gt;&lt;span class="lnt"&gt;69&#10;&lt;/span&gt;&lt;span class="lnt"&gt;70&#10;&lt;/span&gt;&lt;span class="lnt"&gt;71&#10;&lt;/span&gt;&lt;span class="lnt"&gt;72&#10;&lt;/span&gt;&lt;span class="lnt"&gt;73&#10;&lt;/span&gt;&lt;span class="lnt"&gt;74&#10;&lt;/span&gt;&lt;span class="lnt"&gt;75&#10;&lt;/span&gt;&lt;span class="lnt"&gt;76&#10;&lt;/span&gt;&lt;span class="lnt"&gt;77&#10;&lt;/span&gt;&lt;span class="lnt"&gt;78&#10;&lt;/span&gt;&lt;span class="lnt"&gt;79&#10;&lt;/span&gt;&lt;span class="lnt"&gt;80&#10;&lt;/span&gt;&lt;span class="lnt"&gt;81&#10;&lt;/span&gt;&lt;span class="lnt"&gt;82&#10;&lt;/span&gt;&lt;span class="lnt"&gt;83&#10;&lt;/span&gt;&lt;span class="lnt"&gt;84&#10;&lt;/span&gt;&lt;span class="lnt"&gt;85&#10;&lt;/span&gt;&lt;span class="lnt"&gt;86&#10;&lt;/span&gt;&lt;span class="lnt"&gt;87&#10;&lt;/span&gt;&lt;span class="lnt"&gt;88&#10;&lt;/span&gt;&lt;span class="lnt"&gt;89&#10;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/td&gt;&#10;&lt;td class="lntd"&gt;&#10;&lt;pre tabindex="0" class="chroma"&gt;&lt;code class="language-python" data-lang="python"&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="c1"&gt;# pip install anthropic psycopg[binary] pgvector&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="nn"&gt;os&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="kn"&gt;from&lt;/span&gt; &lt;span class="nn"&gt;dataclasses&lt;/span&gt; &lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;dataclass&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="kn"&gt;from&lt;/span&gt; &lt;span class="nn"&gt;typing&lt;/span&gt; &lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;Literal&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="kn"&gt;from&lt;/span&gt; &lt;span class="nn"&gt;anthropic&lt;/span&gt; &lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;Anthropic&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="nn"&gt;psycopg&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="n"&gt;client&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;Anthropic&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;api_key&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;os&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;environ&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;&amp;#34;ANTHROPIC_API_KEY&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;])&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="nd"&gt;@dataclass&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="k"&gt;class&lt;/span&gt; &lt;span class="nc"&gt;Route&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;model&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;str&lt;/span&gt; &lt;span class="c1"&gt;# the deployable model for this request&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;rag&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;bool&lt;/span&gt; &lt;span class="c1"&gt;# whether to retrieve context&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;use_finetune&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;bool&lt;/span&gt; &lt;span class="c1"&gt;# whether to call the fine-tuned variant&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;classify_request&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;text&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;str&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;-&amp;gt;&lt;/span&gt; &lt;span class="n"&gt;Literal&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;&amp;#34;routine&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;&amp;#34;domain_heavy&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;&amp;#34;novel&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;]:&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="s2"&gt;&amp;#34;&amp;#34;&amp;#34;A real classifier; here a stub.&#10;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="s2"&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="s2"&gt; routine -&amp;gt; high-volume, behavior-bound, fine-tuned student wins&#10;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="s2"&gt; domain_heavy -&amp;gt; needs RAG over the knowledge base&#10;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="s2"&gt; novel -&amp;gt; outside the training distribution, frontier + RAG&#10;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="s2"&gt; &amp;#34;&amp;#34;&amp;#34;&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="s2"&gt;&amp;#34;policy&amp;#34;&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;text&lt;/span&gt; &lt;span class="ow"&gt;or&lt;/span&gt; &lt;span class="s2"&gt;&amp;#34;doc&amp;#34;&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;text&lt;/span&gt; &lt;span class="ow"&gt;or&lt;/span&gt; &lt;span class="s2"&gt;&amp;#34;how do I&amp;#34;&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;text&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="s2"&gt;&amp;#34;domain_heavy&amp;#34;&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="nb"&gt;len&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;text&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;&amp;lt;&lt;/span&gt; &lt;span class="mi"&gt;80&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="s2"&gt;&amp;#34;routine&amp;#34;&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="s2"&gt;&amp;#34;novel&amp;#34;&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;pick_route&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;text&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;str&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;-&amp;gt;&lt;/span&gt; &lt;span class="n"&gt;Route&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;kind&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;classify_request&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;text&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;kind&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="s2"&gt;&amp;#34;routine&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="c1"&gt;# behavior-tuned student, no RAG cost on the latency-sensitive path&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;Route&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;model&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;#34;claude-haiku-4-5&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;rag&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="kc"&gt;False&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;use_finetune&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="kc"&gt;True&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;kind&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="s2"&gt;&amp;#34;domain_heavy&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="c1"&gt;# RAG carries the volatile knowledge; mid-tier model carries the synthesis&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;Route&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;model&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;#34;claude-sonnet-4-6&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;rag&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="kc"&gt;True&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;use_finetune&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="kc"&gt;False&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="c1"&gt;# novel -&amp;gt; escalate to the frontier with RAG safety net&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;Route&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;model&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;#34;claude-opus-4-7&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;rag&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="kc"&gt;True&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;use_finetune&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="kc"&gt;False&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;retrieve&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;query&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;str&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;conn&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;psycopg&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Connection&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;k&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;int&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mi"&gt;6&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;-&amp;gt;&lt;/span&gt; &lt;span class="nb"&gt;list&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="nb"&gt;str&lt;/span&gt;&lt;span class="p"&gt;]:&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="c1"&gt;# Hybrid retrieval: embed the query, ANN lookup, lex fallback.&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="c1"&gt;# Embedding fetch elided for brevity; assume `query_vec` is computed.&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;query_vec&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;embed&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;query&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="c1"&gt;# produces list[float] of length D&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="k"&gt;with&lt;/span&gt; &lt;span class="n"&gt;conn&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;cursor&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt; &lt;span class="k"&gt;as&lt;/span&gt; &lt;span class="n"&gt;cur&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;cur&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;execute&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="s2"&gt;&amp;#34;&amp;#34;&amp;#34;&#10;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="s2"&gt; SELECT chunk_text FROM doc_chunks&#10;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="s2"&gt; ORDER BY embedding &amp;lt;=&amp;gt; &lt;/span&gt;&lt;span class="si"&gt;%s&lt;/span&gt;&lt;span class="s2"&gt;::vector&#10;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="s2"&gt; LIMIT &lt;/span&gt;&lt;span class="si"&gt;%s&lt;/span&gt;&lt;span class="s2"&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="s2"&gt; &amp;#34;&amp;#34;&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;query_vec&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;k&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="p"&gt;)&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;row&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;row&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;cur&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;fetchall&lt;/span&gt;&lt;span class="p"&gt;()]&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;serve&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;text&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;str&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;conn&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;psycopg&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Connection&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;-&amp;gt;&lt;/span&gt; &lt;span class="nb"&gt;str&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;route&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;pick_route&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;text&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;system_prompt&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;SYSTEM_PROMPTS&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;&amp;#34;finetuned&amp;#34;&lt;/span&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;route&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;use_finetune&lt;/span&gt; &lt;span class="k"&gt;else&lt;/span&gt; &lt;span class="s2"&gt;&amp;#34;base&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;user_block&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;text&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;route&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;rag&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;chunks&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;retrieve&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;text&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;conn&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;k&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;6&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;ctx&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s2"&gt;&amp;#34;&lt;/span&gt;&lt;span class="se"&gt;\n\n&lt;/span&gt;&lt;span class="s2"&gt;&amp;#34;&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;join&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sa"&gt;f&lt;/span&gt;&lt;span class="s2"&gt;&amp;#34;&amp;lt;source&amp;gt;&lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;c&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s2"&gt;&amp;lt;/source&amp;gt;&amp;#34;&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;c&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;chunks&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;user_block&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="sa"&gt;f&lt;/span&gt;&lt;span class="s2"&gt;&amp;#34;&amp;lt;context&amp;gt;&lt;/span&gt;&lt;span class="se"&gt;\n&lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;ctx&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="se"&gt;\n&lt;/span&gt;&lt;span class="s2"&gt;&amp;lt;/context&amp;gt;&lt;/span&gt;&lt;span class="se"&gt;\n\n&lt;/span&gt;&lt;span class="s2"&gt;&amp;lt;question&amp;gt;&lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;text&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s2"&gt;&amp;lt;/question&amp;gt;&amp;#34;&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;resp&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;client&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;messages&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;create&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;model&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;route&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;model&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="c1"&gt;# if use_finetune, this would be the deployed variant ID&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;max_tokens&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;1024&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;system&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;system_prompt&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;messages&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="p"&gt;[{&lt;/span&gt;&lt;span class="s2"&gt;&amp;#34;role&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;&amp;#34;user&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;&amp;#34;content&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;user_block&lt;/span&gt;&lt;span class="p"&gt;}],&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="p"&gt;)&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;resp&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;content&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;text&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="n"&gt;SYSTEM_PROMPTS&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="c1"&gt;# The fine-tuned student already encodes the brand voice and format,&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="c1"&gt;# so the system prompt collapses to a one-liner. The base model needs&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="c1"&gt;# the full behavioral demo set.&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="s2"&gt;&amp;#34;finetuned&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;&amp;#34;Answer concisely in the established support voice.&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="s2"&gt;&amp;#34;base&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="s2"&gt;&amp;#34;You are a customer support assistant for ACME. &amp;#34;&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="s2"&gt;&amp;#34;Speak in second person, lead with the answer, &amp;#34;&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="s2"&gt;&amp;#34;cite sources from &amp;lt;source&amp;gt; tags inline, and refuse &amp;#34;&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="s2"&gt;&amp;#34;out-of-scope requests with: &amp;#39;That&amp;#39;s outside what I can help with.&amp;#39;&amp;#34;&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="p"&gt;),&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/td&gt;&lt;/tr&gt;&lt;/table&gt;&#10;&lt;/div&gt;&#10;&lt;/div&gt;&lt;/div&gt;&#10;&lt;/figure&gt;&#10;&lt;p&gt;The decision logic is the spine; everything else is plumbing. The pattern&amp;rsquo;s key invariant: the &lt;em&gt;behavior&lt;/em&gt; contract (system prompt, output shape) shifts based on whether you&amp;rsquo;re calling the fine-tuned variant, but the &lt;em&gt;knowledge&lt;/em&gt; contract (RAG context) shifts based on the request&amp;rsquo;s domain density. The two axes are independent and the harness lets them be set independently per request.&lt;/p&gt;&#10;&lt;h2 id="further-reading-from-the-field"&gt;Further reading from the field&lt;/h2&gt;&#10;&lt;ul&gt;&#10;&lt;li&gt;&lt;a href="https://hamel.dev/blog/posts/fine_tuning_valuable.html"&gt;Hamel Husain, &amp;ldquo;Is Fine-Tuning Still Valuable?&amp;rdquo;&lt;/a&gt;; the practical framing of when fine-tuning pays back its cost, and the prompt-engineering-first discipline that decides whether to escalate.&lt;/li&gt;&#10;&lt;li&gt;&lt;a href="https://arxiv.org/abs/2403.01432"&gt;Soudani, Kanoulas, and Hasibi, &amp;ldquo;Fine Tuning vs. Retrieval Augmented Generation for Less Popular Knowledge&amp;rdquo;&lt;/a&gt;; the long-tail evidence that grounds the &amp;ldquo;fine-tuning forgets the long tail&amp;rdquo; claim with concrete numbers.&lt;/li&gt;&#10;&lt;li&gt;&lt;a href="https://arxiv.org/abs/2404.00213"&gt;Mecklenburg et al., &amp;ldquo;Injecting New Knowledge into Large Language Models via Supervised Fine-Tuning&amp;rdquo;&lt;/a&gt;; the QA-format vs. document-format finding that explains &lt;em&gt;why&lt;/em&gt; fine-tuning on docs underperforms.&lt;/li&gt;&#10;&lt;li&gt;&lt;a href="https://platform.claude.com/docs/en/about-claude/use-case-guides/customer-support-chat"&gt;Anthropic&amp;rsquo;s &amp;ldquo;Customer support agent&amp;rdquo; use-case guide&lt;/a&gt;; the production-ready architecture pattern that ships RAG by default and reserves fine-tuning for behavior installation, exactly the framing the article argues for.&lt;/li&gt;&#10;&lt;/ul&gt;&#10;</content:encoded></item><item><title>Cost Optimization and Model Routing</title><link>https://jatinbansal.com/ai-engineering/model-routing/</link><guid isPermaLink="true">https://jatinbansal.com/ai-engineering/model-routing/</guid><pubDate>Tue, 26 May 2026 17:58:40 +0000</pubDate><dc:creator>Jatin Bansal</dc:creator><category>AI Engineering</category><description>How predictive routers and cascades trade model cost, quality, and tail latency per request.</description><content:encoded>&lt;p&gt;Request difficulty varies within most LLM workloads. A router sends routine requests to a cheaper model and reserves an expensive model for requests that need it, provided the routing error rate and added latency stay within the product&amp;rsquo;s quality target.&lt;/p&gt;&#10;&lt;h2 id="the-routing-contract"&gt;The routing contract&lt;/h2&gt;&#10;&lt;p&gt;Model routing selects a candidate model for each request under a quality, cost, or latency constraint. The router can be a heuristic, classifier, learned preference model, or cascade that evaluates a cheaper model&amp;rsquo;s response before escalating.&lt;/p&gt;&#10;&lt;p&gt;Routers split along two axes that matter operationally. The first axis is &lt;strong&gt;when the routing decision happens&lt;/strong&gt;: &lt;em&gt;predictive routing&lt;/em&gt; makes a single choice upfront based on features of the request (token length, detected language, embedding-space clusters, a learned classifier&amp;rsquo;s verdict); &lt;em&gt;cascade routing&lt;/em&gt; (also called sequential routing) starts with the cheapest candidate, evaluates the response, and escalates to a more expensive tier only if the cheap response fails a confidence check. The second axis is &lt;strong&gt;what the router optimizes&lt;/strong&gt;: pure &lt;em&gt;cost minimization at a quality floor&lt;/em&gt; (don&amp;rsquo;t drop accuracy below X, minimize spend), &lt;em&gt;quality maximization at a cost ceiling&lt;/em&gt; (don&amp;rsquo;t spend more than Y per request, maximize accuracy), or &lt;em&gt;latency-sensitive routing&lt;/em&gt; (route by tail-latency budget, not cost). Most production routers are predictive, optimize cost at a quality floor, and run on the order of single-digit milliseconds; small enough to live in the request path without showing up as a meaningful p99 contribution.&lt;/p&gt;&#10;&lt;p&gt;Routing operates inside a workload against a fixed model pool. A uniformly difficult workload has little routing headroom; mixed workloads can save money when a cheap tier handles a large share of requests without quality loss.&lt;/p&gt;&#10;&lt;h2 id="router-architectures"&gt;Router architectures&lt;/h2&gt;&#10;&lt;h3 id="predictive-routing-classify-upfront-commit"&gt;Predictive routing: classify upfront, commit&lt;/h3&gt;&#10;&lt;p&gt;The cheapest router shape. A small classifier; logistic regression on hand-engineered features, a fine-tuned small LM, or a matrix-factorization model over the request and the candidate pool; scores each incoming request and assigns it to a tier in one shot. The classifier runs in milliseconds; the routing decision is final. The training data is pairs of (request, which-model-was-good-enough) collected from offline labeling, &lt;a href="https://jatinbansal.com/ai-engineering/llm-as-judge/"&gt;LLM-as-judge&lt;/a&gt; runs, or user feedback signals from a prior shadow-routed system.&lt;/p&gt;&#10;&lt;p&gt;&lt;a href="https://www.lmsys.org/blog/2024-07-01-routellm/"&gt;RouteLLM&lt;/a&gt;, the open-source framework released by LMSYS in July 2024 and accepted to &lt;a href="https://openreview.net/forum?id=8sSqNntaMr"&gt;ICLR 2025&lt;/a&gt;, is the canonical worked example. The paper trains four router architectures on Chatbot Arena preference data: similarity-weighted ranking, a matrix-factorization model that decomposes (query, model) → reward, a BERT classifier, and a causal LM classifier. The headline result, reproduced from the paper: the matrix-factorization router achieves 95% of GPT-4&amp;rsquo;s quality on MT-Bench while routing only 26% of queries to GPT-4; a 48% cost reduction vs. random baseline at iso-quality. With data augmentation from an LLM judge, the same router hits 95% quality at only 14% strong-model calls; a 75% cost reduction.&lt;/p&gt;&#10;&lt;p&gt;&lt;a href="https://www.withmartian.com/"&gt;Martian&lt;/a&gt; and &lt;a href="https://www.notdiamond.ai/"&gt;NotDiamond&lt;/a&gt; are the commercial heirs of this architecture, with refinements: per-customer custom routers trained on the customer&amp;rsquo;s own traffic (the &lt;a href="https://docs.notdiamond.ai/docs/router-training-quickstart"&gt;NotDiamond router-training quickstart&lt;/a&gt; walks through this), routers that account for provider-side outages and latency in addition to quality, and feature engineering that includes embedding distance from training-set clusters. The Martian site claims cost reductions of 20–97% depending on workload; NotDiamond reports a 39% accuracy increase across SRE benchmarks in one published enterprise case study. The honest reading of these numbers is that they&amp;rsquo;re highly workload-specific; a recent &lt;a href="https://arxiv.org/html/2510.00202v3"&gt;RouterArena benchmark&lt;/a&gt; found NotDiamond ranked 12th on LongBench-v2 because its general-purpose router frequently selected expensive models for queries that didn&amp;rsquo;t need them. Custom routers trained on workload data consistently outperform general-purpose ones; off-the-shelf routers are the right choice when you don&amp;rsquo;t have labeled traffic yet and the wrong choice once you do.&lt;/p&gt;&#10;&lt;h3 id="cascade-routing-start-cheap-escalate-on-low-confidence"&gt;Cascade routing: start cheap, escalate on low confidence&lt;/h3&gt;&#10;&lt;p&gt;The shape from the &lt;a href="https://arxiv.org/abs/2305.05176"&gt;FrugalGPT paper&lt;/a&gt; (Chen, Zaharia, Zou, May 2023). Try the cheap model first; check the response against a &lt;em&gt;score function&lt;/em&gt; that estimates whether the answer is acceptable; if it passes, return it; if not, escalate to the next tier and repeat. The score function is the hard part; it can be a separate small judge model, the cheap model&amp;rsquo;s own log-probabilities, a verifier that runs the answer against a known schema, or a simple regex/structured-output check. The original FrugalGPT result: matching GPT-4&amp;rsquo;s accuracy at 98% cost reduction on the workloads they evaluated, by running queries through a cascade of 12 candidate models with a learned regression-based scoring function.&lt;/p&gt;&#10;&lt;p&gt;Cascade routing&amp;rsquo;s structural advantage over predictive routing is that &lt;strong&gt;it gets to see the cheap model&amp;rsquo;s actual response before deciding&lt;/strong&gt;. The predictive router commits to a tier on the input alone, which means it has to predict response quality without observing it; the cascade gets to &lt;em&gt;score the realized response&lt;/em&gt;, which is a much easier learning problem. This is also cascade routing&amp;rsquo;s structural disadvantage: every escalation pays both the cheap model&amp;rsquo;s full cost &lt;em&gt;and&lt;/em&gt; the expensive model&amp;rsquo;s full cost. If 40% of queries escalate, the average cost is &lt;code&gt;0.6·C₁ + 0.4·(C₁ + C₂)&lt;/code&gt; instead of the predictive router&amp;rsquo;s &lt;code&gt;0.6·C₁ + 0.4·C₂&lt;/code&gt;; the cascade pays C₁ on every request, escalations included. Cascades win when the cheap model handles a large majority of traffic and the escalation rate stays under ~30%; they lose when the workload is hard enough that escalations dominate.&lt;/p&gt;&#10;&lt;p&gt;Cascade routing also has a latency tail that predictive routing doesn&amp;rsquo;t: every escalation adds a full extra round-trip to the request path. If the cheap model takes 400ms and the expensive model takes 1.5s, escalations land at ~1.9s of end-to-end latency instead of 1.5s. Production cascades therefore tune the cheap-model timeout aggressively (often well below the cheap model&amp;rsquo;s p99 latency) and accept some &amp;ldquo;cheap timeout, escalate anyway&amp;rdquo; cases to keep tail latency bounded.&lt;/p&gt;&#10;&lt;h3 id="content-based-routing-heuristics-on-features-you-already-have"&gt;Content-based routing: heuristics on features you already have&lt;/h3&gt;&#10;&lt;p&gt;The shape that doesn&amp;rsquo;t get its own paper but ships first in most teams because the features are free. Route on detected language (English to GPT-5.4-mini, Mandarin to Qwen-Plus), on content type (code → a code-specialized model, prose → a general-purpose one), on detected complexity proxies (token count, presence of math/code/tables), on user tier (free users get the cheap model, paying users get the expensive one). Heuristic content-based routing is what every team starts with and what most teams continue to use as a layer &lt;em&gt;underneath&lt;/em&gt; a learned router; the heuristic does the obvious splits and the learned router does the harder per-request calls inside the remaining bucket.&lt;/p&gt;&#10;&lt;p&gt;The honest framing: content-based routing is what gets you 60–80% of the routing win at near-zero implementation cost. The marginal headroom from layering a learned router on top is real but typically smaller than the heuristic-only baseline. Build heuristic first, measure, layer learned routing only where the heuristic leaves obvious money on the table.&lt;/p&gt;&#10;&lt;h2 id="expected-cost-and-break-even"&gt;Expected cost and break-even&lt;/h2&gt;&#10;&lt;p&gt;Concrete numbers as of May 2026, based on the published pricing across providers. The four model tiers most production teams actually route between:&lt;/p&gt;&#10;&lt;table&gt;&#10;&#9;&lt;thead&gt;&#10;&#9;&#9;&#9;&lt;tr&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;th&gt;Tier&lt;/th&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;th&gt;Provider/Model&lt;/th&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;th&gt;Input $/M&lt;/th&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;th&gt;Output $/M&lt;/th&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;th&gt;Avg cost / 500-in/300-out request&lt;/th&gt;&#10;&#9;&#9;&#9;&lt;/tr&gt;&#10;&#9;&lt;/thead&gt;&#10;&#9;&lt;tbody&gt;&#10;&#9;&#9;&#9;&lt;tr&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;td&gt;Cheapest&lt;/td&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;td&gt;&lt;a href="https://ai.google.dev/gemini-api/docs/pricing"&gt;Gemini 2.5 Flash Lite&lt;/a&gt;&lt;/td&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;td&gt;$0.10&lt;/td&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;td&gt;$0.40&lt;/td&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;td&gt;~$0.00017&lt;/td&gt;&#10;&#9;&#9;&#9;&lt;/tr&gt;&#10;&#9;&#9;&#9;&lt;tr&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;td&gt;Cheap&lt;/td&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;td&gt;&lt;a href="https://platform.claude.com/docs/en/about-claude/pricing"&gt;Haiku 4.5&lt;/a&gt;&lt;/td&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;td&gt;$1.00&lt;/td&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;td&gt;$5.00&lt;/td&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;td&gt;~$0.002&lt;/td&gt;&#10;&#9;&#9;&#9;&lt;/tr&gt;&#10;&#9;&#9;&#9;&lt;tr&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;td&gt;Mid&lt;/td&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;td&gt;&lt;a href="https://developers.openai.com/api/docs/pricing"&gt;GPT-5.4-mini&lt;/a&gt;&lt;/td&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;td&gt;$0.75&lt;/td&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;td&gt;$4.50&lt;/td&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;td&gt;~$0.0017&lt;/td&gt;&#10;&#9;&#9;&#9;&lt;/tr&gt;&#10;&#9;&#9;&#9;&lt;tr&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;td&gt;Frontier&lt;/td&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;td&gt;&lt;a href="https://platform.claude.com/docs/en/about-claude/pricing"&gt;Sonnet 4.6&lt;/a&gt;&lt;/td&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;td&gt;$3.00&lt;/td&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;td&gt;$15.00&lt;/td&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;td&gt;~$0.006&lt;/td&gt;&#10;&#9;&#9;&#9;&lt;/tr&gt;&#10;&#9;&#9;&#9;&lt;tr&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;td&gt;Top&lt;/td&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;td&gt;&lt;a href="https://platform.claude.com/docs/en/about-claude/pricing"&gt;Opus 4.7&lt;/a&gt;&lt;/td&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;td&gt;$5.00&lt;/td&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;td&gt;$25.00&lt;/td&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;td&gt;~$0.010&lt;/td&gt;&#10;&#9;&#9;&#9;&lt;/tr&gt;&#10;&#9;&#9;&#9;&lt;tr&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;td&gt;Top&lt;/td&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;td&gt;&lt;a href="https://developers.openai.com/api/docs/pricing"&gt;GPT-5.5&lt;/a&gt;&lt;/td&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;td&gt;$5.00&lt;/td&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;td&gt;$30.00&lt;/td&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;td&gt;~$0.011&lt;/td&gt;&#10;&#9;&#9;&#9;&lt;/tr&gt;&#10;&#9;&lt;/tbody&gt;&#10;&lt;/table&gt;&#10;&lt;p&gt;The cost-savings math for an always-Opus baseline (~$0.010/request) routed to a Haiku-mostly policy:&lt;/p&gt;&#10;&lt;ul&gt;&#10;&lt;li&gt;70% Haiku + 30% Opus + $0.001 router overhead = &lt;code&gt;0.70·0.002 + 0.30·0.010 + 0.001 = $0.0054/request&lt;/code&gt;, a &lt;strong&gt;46% reduction&lt;/strong&gt;.&lt;/li&gt;&#10;&lt;li&gt;90% Haiku + 10% Opus + $0.001 = &lt;code&gt;$0.0029/request&lt;/code&gt;, a &lt;strong&gt;71% reduction&lt;/strong&gt;.&lt;/li&gt;&#10;&lt;li&gt;100% Haiku (no router, accept quality drop) = &lt;code&gt;$0.002/request&lt;/code&gt;, a &lt;strong&gt;80% reduction at unknown quality cost&lt;/strong&gt;.&lt;/li&gt;&#10;&lt;/ul&gt;&#10;&lt;p&gt;The breakeven analysis: a router with overhead r is worth running iff &lt;code&gt;(1-p_strong)·(C_strong - C_weak) &amp;gt; r&lt;/code&gt;. Plugging in p_strong = 0.30, C_strong = $0.010, C_weak = $0.002: the savings are &lt;code&gt;0.70·0.008 = $0.0056/request&lt;/code&gt;. Anything less than $5.60 per 1000 requests in router overhead is a net win. At Anthropic Haiku pricing for a typical classifier prompt (~500 input tokens, ~10 output tokens) that&amp;rsquo;s about $0.0006/request; well under the breakeven, by an order of magnitude. The router pays for itself even before you account for the latency improvement on the routed-to-Haiku majority of traffic.&lt;/p&gt;&#10;&lt;p&gt;Two structural numbers it&amp;rsquo;s worth committing to memory. &lt;strong&gt;The 5× rule of Anthropic pricing&lt;/strong&gt;: output costs 5× input across every tier, so any optimization that reduces output tokens (more concise prompts, schema-constrained generation via &lt;a href="https://jatinbansal.com/ai-engineering/structured-output/"&gt;structured output&lt;/a&gt;, early stopping) has 5× the leverage of the same reduction on input tokens. &lt;strong&gt;The 50% batch-API discount&lt;/strong&gt;: the &lt;a href="https://platform.claude.com/docs/en/about-claude/pricing"&gt;Anthropic Batch API&lt;/a&gt; and OpenAI&amp;rsquo;s batch endpoint both charge 50% of standard rates for asynchronous, ≤24-hour-latency jobs. Any workload that can tolerate that latency; overnight evals, content generation queues, &lt;a href="https://jatinbansal.com/ai-engineering/memory-reflection/"&gt;reflection&lt;/a&gt;/consolidation passes, &lt;a href="https://jatinbansal.com/ai-engineering/sleep-time-compute/"&gt;sleep-time compute&lt;/a&gt;; runs at half the price card, &lt;em&gt;compounding&lt;/em&gt; with prompt caching for a &lt;code&gt;0.5 × 0.1 = 5%&lt;/code&gt; effective rate on cached batch jobs.&lt;/p&gt;&#10;&lt;h2 id="code-a-hand-rolled-cascade-router-in-python"&gt;Code: a hand-rolled cascade router in Python&lt;/h2&gt;&#10;&lt;p&gt;The skeleton below implements a two-tier cascade against the &lt;a href="https://github.com/anthropics/anthropic-sdk-python"&gt;Anthropic SDK&lt;/a&gt;: try Haiku first, escalate to Sonnet if a self-reported confidence score from Haiku falls below a threshold. The confidence signal here is a simple &lt;a href="https://jatinbansal.com/ai-engineering/structured-output/"&gt;structured-output&lt;/a&gt; score from the model itself; production systems would use a separate judge or learned regressor, but the self-report works as a starting baseline and the math doesn&amp;rsquo;t change.&lt;/p&gt;&#10;&lt;figure class="nb-code"&gt;&#10; &lt;header class="nb-code-chrome"&gt;&#10; &lt;span class="nb-code-label"&gt;python&lt;/span&gt;&#10; &#10; &lt;/header&gt;&#10; &lt;div class="nb-code-body"&gt;&lt;div class="highlight"&gt;&lt;div class="chroma"&gt;&#10;&lt;table class="lntable"&gt;&lt;tr&gt;&lt;td class="lntd"&gt;&#10;&lt;pre tabindex="0" class="chroma"&gt;&lt;code&gt;&lt;span class="lnt"&gt; 1&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 2&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 3&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 4&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 5&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 6&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 7&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 8&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 9&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 10&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 11&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 12&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 13&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 14&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 15&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 16&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 17&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 18&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 19&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 20&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 21&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 22&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 23&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 24&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 25&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 26&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 27&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 28&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 29&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 30&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 31&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 32&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 33&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 34&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 35&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 36&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 37&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 38&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 39&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 40&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 41&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 42&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 43&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 44&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 45&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 46&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 47&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 48&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 49&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 50&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 51&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 52&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 53&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 54&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 55&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 56&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 57&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 58&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 59&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 60&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 61&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 62&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 63&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 64&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 65&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 66&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 67&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 68&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 69&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 70&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 71&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 72&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 73&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 74&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 75&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 76&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 77&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 78&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 79&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 80&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 81&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 82&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 83&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 84&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 85&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 86&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 87&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 88&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 89&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 90&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 91&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 92&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 93&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 94&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 95&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 96&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 97&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 98&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 99&#10;&lt;/span&gt;&lt;span class="lnt"&gt;100&#10;&lt;/span&gt;&lt;span class="lnt"&gt;101&#10;&lt;/span&gt;&lt;span class="lnt"&gt;102&#10;&lt;/span&gt;&lt;span class="lnt"&gt;103&#10;&lt;/span&gt;&lt;span class="lnt"&gt;104&#10;&lt;/span&gt;&lt;span class="lnt"&gt;105&#10;&lt;/span&gt;&lt;span class="lnt"&gt;106&#10;&lt;/span&gt;&lt;span class="lnt"&gt;107&#10;&lt;/span&gt;&lt;span class="lnt"&gt;108&#10;&lt;/span&gt;&lt;span class="lnt"&gt;109&#10;&lt;/span&gt;&lt;span class="lnt"&gt;110&#10;&lt;/span&gt;&lt;span class="lnt"&gt;111&#10;&lt;/span&gt;&lt;span class="lnt"&gt;112&#10;&lt;/span&gt;&lt;span class="lnt"&gt;113&#10;&lt;/span&gt;&lt;span class="lnt"&gt;114&#10;&lt;/span&gt;&lt;span class="lnt"&gt;115&#10;&lt;/span&gt;&lt;span class="lnt"&gt;116&#10;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/td&gt;&#10;&lt;td class="lntd"&gt;&#10;&lt;pre tabindex="0" class="chroma"&gt;&lt;code class="language-python" data-lang="python"&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="c1"&gt;# pip install anthropic&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="nn"&gt;os&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="nn"&gt;time&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="kn"&gt;from&lt;/span&gt; &lt;span class="nn"&gt;dataclasses&lt;/span&gt; &lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;dataclass&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="kn"&gt;from&lt;/span&gt; &lt;span class="nn"&gt;typing&lt;/span&gt; &lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;Literal&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="nn"&gt;anthropic&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="kn"&gt;from&lt;/span&gt; &lt;span class="nn"&gt;pydantic&lt;/span&gt; &lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;BaseModel&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;Field&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="n"&gt;client&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;anthropic&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Anthropic&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;api_key&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;os&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;environ&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;&amp;#34;ANTHROPIC_API_KEY&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;])&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="c1"&gt;# Per-million pricing as of May 2026.&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="n"&gt;PRICE_PER_M&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="s2"&gt;&amp;#34;claude-haiku-4-5&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="s2"&gt;&amp;#34;input&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mf"&gt;1.0&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;&amp;#34;output&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mf"&gt;5.0&lt;/span&gt;&lt;span class="p"&gt;},&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="s2"&gt;&amp;#34;claude-sonnet-4-6&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="s2"&gt;&amp;#34;input&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mf"&gt;3.0&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;&amp;#34;output&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mf"&gt;15.0&lt;/span&gt;&lt;span class="p"&gt;},&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="s2"&gt;&amp;#34;claude-opus-4-7&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="s2"&gt;&amp;#34;input&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mf"&gt;5.0&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;&amp;#34;output&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mf"&gt;25.0&lt;/span&gt;&lt;span class="p"&gt;},&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="p"&gt;}&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="k"&gt;class&lt;/span&gt; &lt;span class="nc"&gt;CascadeResponse&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;BaseModel&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;answer&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;str&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;Field&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;description&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;#34;The actual answer.&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;self_confidence&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;float&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;Field&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;ge&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;le&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;description&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;#34;How confident the model is in the answer.&amp;#34;&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="p"&gt;)&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="nd"&gt;@dataclass&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="k"&gt;class&lt;/span&gt; &lt;span class="nc"&gt;RouteTrace&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;model_used&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;str&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;escalated&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;bool&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;cost_usd&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;float&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;latency_ms&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;float&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;input_tokens&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;int&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;output_tokens&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;int&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;call&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;model&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;str&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;prompt&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;str&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;-&amp;gt;&lt;/span&gt; &lt;span class="nb"&gt;tuple&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;CascadeResponse&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nb"&gt;dict&lt;/span&gt;&lt;span class="p"&gt;]:&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="s2"&gt;&amp;#34;&amp;#34;&amp;#34;One call. Returns parsed response + usage.&amp;#34;&amp;#34;&amp;#34;&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;t0&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;time&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;perf_counter&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;msg&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;client&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;messages&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;create&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;model&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;model&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;max_tokens&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;512&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;messages&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="p"&gt;[{&lt;/span&gt;&lt;span class="s2"&gt;&amp;#34;role&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;&amp;#34;user&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;&amp;#34;content&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;prompt&lt;/span&gt;&lt;span class="p"&gt;}],&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;system&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="s2"&gt;&amp;#34;Respond with a JSON object: &amp;#34;&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="s1"&gt;&amp;#39;{&amp;#34;answer&amp;#34;: &amp;lt;string&amp;gt;, &amp;#34;self_confidence&amp;#34;: &amp;lt;float in [0,1] &amp;#39;&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="s2"&gt;&amp;#34;indicating how confident you are this answer is correct&amp;gt;}. &amp;#34;&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="s2"&gt;&amp;#34;Be conservative; if the question is ambiguous, lower the score.&amp;#34;&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="p"&gt;),&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="p"&gt;)&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;latency_ms&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;time&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;perf_counter&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt; &lt;span class="n"&gt;t0&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="mi"&gt;1000&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;text&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s2"&gt;&amp;#34;&amp;#34;&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;join&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;b&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;text&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;b&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;msg&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;content&lt;/span&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;b&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;type&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="s2"&gt;&amp;#34;text&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="c1"&gt;# In production, use the structured-output article&amp;#39;s robust JSON parsing.&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;parsed&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;CascadeResponse&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;model_validate_json&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;text&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;parsed&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="s2"&gt;&amp;#34;input_tokens&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;msg&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;usage&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;input_tokens&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="s2"&gt;&amp;#34;output_tokens&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;msg&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;usage&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;output_tokens&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="s2"&gt;&amp;#34;latency_ms&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;latency_ms&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="p"&gt;}&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;cost_for&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;model&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;str&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;in_tok&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;int&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;out_tok&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;int&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;-&amp;gt;&lt;/span&gt; &lt;span class="nb"&gt;float&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;p&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;PRICE_PER_M&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;model&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;in_tok&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="n"&gt;p&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;&amp;#34;input&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="n"&gt;out_tok&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="n"&gt;p&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;&amp;#34;output&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;])&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt; &lt;span class="mi"&gt;1_000_000&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;cascade&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;prompt&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;str&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;cheap&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;str&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s2"&gt;&amp;#34;claude-haiku-4-5&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;expensive&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;str&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s2"&gt;&amp;#34;claude-sonnet-4-6&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;confidence_floor&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;float&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mf"&gt;0.7&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;-&amp;gt;&lt;/span&gt; &lt;span class="nb"&gt;tuple&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;CascadeResponse&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;RouteTrace&lt;/span&gt;&lt;span class="p"&gt;]:&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;cheap_resp&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;cheap_usage&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;call&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;cheap&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;prompt&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;cheap_cost&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;cost_for&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;cheap&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;cheap_usage&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;&amp;#34;input_tokens&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt; &lt;span class="n"&gt;cheap_usage&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;&amp;#34;output_tokens&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;])&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;cheap_resp&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;self_confidence&lt;/span&gt; &lt;span class="o"&gt;&amp;gt;=&lt;/span&gt; &lt;span class="n"&gt;confidence_floor&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;cheap_resp&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;RouteTrace&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;model_used&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;cheap&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;escalated&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="kc"&gt;False&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;cost_usd&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;cheap_cost&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;latency_ms&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;cheap_usage&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;&amp;#34;latency_ms&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;input_tokens&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;cheap_usage&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;&amp;#34;input_tokens&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;output_tokens&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;cheap_usage&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;&amp;#34;output_tokens&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="p"&gt;)&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="c1"&gt;# Escalate. We pay the cheap call&amp;#39;s cost *and* the expensive call.&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;exp_resp&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;exp_usage&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;call&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;expensive&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;prompt&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;exp_cost&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;cost_for&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;expensive&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;exp_usage&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;&amp;#34;input_tokens&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt; &lt;span class="n"&gt;exp_usage&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;&amp;#34;output_tokens&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;])&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;exp_resp&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;RouteTrace&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;model_used&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;expensive&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;escalated&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="kc"&gt;True&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;cost_usd&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;cheap_cost&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="n"&gt;exp_cost&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;latency_ms&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;cheap_usage&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;&amp;#34;latency_ms&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="n"&gt;exp_usage&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;&amp;#34;latency_ms&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;input_tokens&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;cheap_usage&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;&amp;#34;input_tokens&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="n"&gt;exp_usage&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;&amp;#34;input_tokens&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;output_tokens&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;cheap_usage&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;&amp;#34;output_tokens&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="n"&gt;exp_usage&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;&amp;#34;output_tokens&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="p"&gt;)&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="vm"&gt;__name__&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="s2"&gt;&amp;#34;__main__&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="c1"&gt;# Realistic spread: easy/medium/hard.&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;prompts&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="s2"&gt;&amp;#34;What&amp;#39;s the capital of France?&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="c1"&gt;# easy → cheap should be confident&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="s2"&gt;&amp;#34;Summarize this sentence: &amp;#39;The cat sat on the mat.&amp;#39;&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="c1"&gt;# easy&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="s2"&gt;&amp;#34;Walk me through the trade-offs of using a B-tree vs. an LSM-tree for &amp;#34;&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="s2"&gt;&amp;#34;a write-heavy workload, and quantify the read amplification.&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="c1"&gt;# hard&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="s2"&gt;&amp;#34;Prove that the sum of two even numbers is even.&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="c1"&gt;# easy&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="p"&gt;]&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;total_cost&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mf"&gt;0.0&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;escalations&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;p&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;prompts&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;resp&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;trace&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;cascade&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;p&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="nb"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sa"&gt;f&lt;/span&gt;&lt;span class="s2"&gt;&amp;#34;[&lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;trace&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;model_used&lt;/span&gt;&lt;span class="si"&gt;:&lt;/span&gt;&lt;span class="s2"&gt;25s&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s2"&gt;] escalated=&lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;trace&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;escalated&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s2"&gt; &amp;#34;&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="sa"&gt;f&lt;/span&gt;&lt;span class="s2"&gt;&amp;#34;cost=$&lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;trace&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;cost_usd&lt;/span&gt;&lt;span class="si"&gt;:&lt;/span&gt;&lt;span class="s2"&gt;.5f&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s2"&gt; latency=&lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;trace&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;latency_ms&lt;/span&gt;&lt;span class="si"&gt;:&lt;/span&gt;&lt;span class="s2"&gt;.0f&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s2"&gt;ms &amp;#34;&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="sa"&gt;f&lt;/span&gt;&lt;span class="s2"&gt;&amp;#34;answer=&lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;resp&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;answer&lt;/span&gt;&lt;span class="p"&gt;[:&lt;/span&gt;&lt;span class="mi"&gt;60&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s2"&gt;...&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;total_cost&lt;/span&gt; &lt;span class="o"&gt;+=&lt;/span&gt; &lt;span class="n"&gt;trace&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;cost_usd&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;trace&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;escalated&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;escalations&lt;/span&gt; &lt;span class="o"&gt;+=&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="nb"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sa"&gt;f&lt;/span&gt;&lt;span class="s2"&gt;&amp;#34;&lt;/span&gt;&lt;span class="se"&gt;\n&lt;/span&gt;&lt;span class="s2"&gt;Total: $&lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;total_cost&lt;/span&gt;&lt;span class="si"&gt;:&lt;/span&gt;&lt;span class="s2"&gt;.4f&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s2"&gt; over &lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="nb"&gt;len&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;prompts&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s2"&gt; requests, &amp;#34;&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="sa"&gt;f&lt;/span&gt;&lt;span class="s2"&gt;&amp;#34;escalations=&lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;escalations&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s2"&gt;/&lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="nb"&gt;len&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;prompts&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s2"&gt;&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/td&gt;&lt;/tr&gt;&lt;/table&gt;&#10;&lt;/div&gt;&#10;&lt;/div&gt;&lt;/div&gt;&#10;&lt;/figure&gt;&#10;&lt;p&gt;the self-confidence score is the simplest possible cascade signal but also the weakest; production cascades typically use either a separate small judge (one &lt;a href="https://jatinbansal.com/ai-engineering/llm-as-judge/"&gt;LLM-as-judge&lt;/a&gt; call on the cheap model&amp;rsquo;s output) or a learned regressor that takes the cheap response&amp;rsquo;s log-probs and outputs an accept/escalate score. The self-report version above is fine for getting a cascade running but should be replaced with a learned scorer once you have ~1k labeled (prompt, cheap_response, was_good_enough) tuples. The cascade pays the cheap call&amp;rsquo;s cost on every request and the expensive call&amp;rsquo;s cost on escalations; the cost-savings math depends entirely on the escalation rate staying under ~30%. Tune the &lt;code&gt;confidence_floor&lt;/code&gt; against your workload.&lt;/p&gt;&#10;&lt;h2 id="further-reading-from-the-field"&gt;Further reading from the field&lt;/h2&gt;&#10;&lt;ul&gt;&#10;&lt;li&gt;&lt;a href="https://www.lmsys.org/blog/2024-07-01-routellm/"&gt;RouteLLM: An Open-Source Framework for Cost-Effective LLM Routing&lt;/a&gt;; LMSYS, July 2024; the canonical announcement post for the open-source RouteLLM framework; walks through the four router architectures and the cost-savings numbers reproduced above.&lt;/li&gt;&#10;&lt;li&gt;&lt;a href="https://arxiv.org/abs/2305.05176"&gt;FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performance&lt;/a&gt;; Chen, Zaharia, Zou, 2023; the cascade-routing paper. Three orthogonal strategies (prompt adaptation, LLM approximation, LLM cascade) with a worked benchmark showing 98% cost reduction at GPT-4 quality.&lt;/li&gt;&#10;&lt;li&gt;&lt;a href="https://www.anyscale.com/blog/building-an-llm-router-for-high-quality-and-cost-effective-responses"&gt;Building an LLM Router for High-Quality and Cost-Effective Responses&lt;/a&gt;; Anyscale, 2024; a hands-on guide to deploying a learned router in production, with discussion of the train/eval loop and the per-segment-policy pattern.&lt;/li&gt;&#10;&lt;li&gt;&lt;a href="https://venturebeat.com/ai/why-accenture-and-martian-see-model-routing-as-key-to-enterprise-ai-success"&gt;Why Accenture and Martian see model routing as key to enterprise AI success&lt;/a&gt;; VentureBeat, 2025; the enterprise-adoption framing for routing, including the air-traffic-control mental model and the 37%-of-enterprises-use-5+-models statistic.&lt;/li&gt;&#10;&lt;/ul&gt;&#10;</content:encoded></item><item><title>Speculative Decoding and Draft Models</title><link>https://jatinbansal.com/ai-engineering/speculative-decoding/</link><guid isPermaLink="true">https://jatinbansal.com/ai-engineering/speculative-decoding/</guid><pubDate>Tue, 26 May 2026 17:55:00 +0000</pubDate><dc:creator>Jatin Bansal</dc:creator><category>AI Engineering</category><description>Speculative decoding with draft models, Medusa, EAGLE, and prompt lookup for lower decode latency.</description><content:encoded>&lt;p&gt;Speculative decoding can improve time per output token in an idle benchmark and reduce throughput under a heavily batched workload. Its value depends on draft acceptance, verification cost, KV-cache pressure, and scheduler load. Measurements therefore need the production batch-size distribution, not only single-request latency.&lt;/p&gt;&#10;&lt;h2 id="acceptance-and-speedup"&gt;Acceptance and speedup&lt;/h2&gt;&#10;&lt;p&gt;Three numbers govern whether speculation pays off, in the framing from Leviathan et al.:&lt;/p&gt;&#10;&lt;ul&gt;&#10;&lt;li&gt;α (alpha); the acceptance rate. The probability that a single drafter-proposed token survives verification.&lt;/li&gt;&#10;&lt;li&gt;γ (gamma); the speculation length. How many tokens the drafter proposes per verification step.&lt;/li&gt;&#10;&lt;li&gt;c; the cost ratio. The drafter&amp;rsquo;s per-token time divided by the target&amp;rsquo;s per-token time. A 1B-parameter drafter against a 70B target has c ≈ 0.02–0.05; Medusa heads have c ≈ 0; EAGLE drafters have c in the 0.02–0.10 range depending on tree size.&lt;/li&gt;&#10;&lt;/ul&gt;&#10;&lt;p&gt;The expected number of accepted tokens per speculation step is approximately &lt;code&gt;(1 - α^(γ+1)) / (1 - α)&lt;/code&gt;. The expected cost per speculation step is approximately &lt;code&gt;γ·c + 1&lt;/code&gt; target-step-equivalents (γ drafter steps plus one verification). The speedup over baseline decoding is the ratio: tokens-per-step divided by cost-per-step.&lt;/p&gt;&#10;&lt;p&gt;A worked example. Suppose α = 0.7, γ = 4, c = 0.05. Expected accepted tokens per step is &lt;code&gt;(1 - 0.7⁵)/(1 - 0.7) ≈ 2.79&lt;/code&gt;. Expected cost is &lt;code&gt;4·0.05 + 1 = 1.20&lt;/code&gt; target steps. Speedup is &lt;code&gt;2.79 / 1.20 ≈ 2.32×&lt;/code&gt;. Plug in α = 0.55 instead; about the threshold where speculation is breakeven; and you get &lt;code&gt;(1 - 0.55⁵)/(1 - 0.55) ≈ 2.10&lt;/code&gt; tokens at cost 1.20, speedup &lt;code&gt;1.75×&lt;/code&gt;. Drop α to 0.4 and the speedup collapses to &lt;code&gt;1.42×&lt;/code&gt;; still positive, but small enough that the KV-cache reservation overhead in the batch scheduler can wipe it out. Below α ≈ 0.5 the math gets ugly fast; published &lt;a href="https://github.com/vllm-project/vllm/issues/5015"&gt;vLLM acceptance-rate measurements&lt;/a&gt; suggest production deployments target α ≥ 0.65, ideally α ≥ 0.75.&lt;/p&gt;&#10;&lt;p&gt;The proposal length γ has an optimum because each additional token is less likely to be accepted. Practical values often sit at 3–7 for autoregressive drafters and 2–4 for Medusa-style heads. Acceptance α also depends on workload: a drafter can perform well on uniform chat traffic and worse on high-entropy code generation.&lt;/p&gt;&#10;&lt;h2 id="draft-and-verify-decoding"&gt;Draft-and-verify decoding&lt;/h2&gt;&#10;&lt;p&gt;The original algorithm in pseudocode:&lt;/p&gt;&#10;&lt;ol&gt;&#10;&lt;li&gt;Drafter &lt;code&gt;M_q&lt;/code&gt; generates γ tokens autoregressively: &lt;code&gt;x_1, …, x_γ&lt;/code&gt;, with per-token probabilities &lt;code&gt;q(x_i | prefix, x_&amp;lt;i)&lt;/code&gt;.&lt;/li&gt;&#10;&lt;li&gt;Target &lt;code&gt;M_p&lt;/code&gt; runs &lt;em&gt;one&lt;/em&gt; forward pass on the concatenation &lt;code&gt;[prefix, x_1, …, x_γ]&lt;/code&gt;, producing target probabilities &lt;code&gt;p(· | prefix, x_&amp;lt;i)&lt;/code&gt; for each position i ∈ {1, …, γ+1}.&lt;/li&gt;&#10;&lt;li&gt;For each i from 1 to γ: accept &lt;code&gt;x_i&lt;/code&gt; with probability &lt;code&gt;min(1, p(x_i | ·) / q(x_i | ·))&lt;/code&gt;; on rejection, sample a correction token from the &lt;em&gt;residual distribution&lt;/em&gt; &lt;code&gt;(p − q)⁺&lt;/code&gt;, append the accepted prefix + correction, and stop.&lt;/li&gt;&#10;&lt;li&gt;If all γ tokens were accepted, sample one additional token from &lt;code&gt;p(· | prefix, x_1, …, x_γ)&lt;/code&gt; (the &amp;ldquo;bonus token&amp;rdquo;; free from the same forward pass) and continue.&lt;/li&gt;&#10;&lt;/ol&gt;&#10;&lt;p&gt;The modified rejection sampling in step 3 is what guarantees output-distribution equivalence. The bonus token in step 4 is why the expected accepted-tokens formula is &lt;code&gt;(1 − α^(γ+1)) / (1 − α)&lt;/code&gt; and not &lt;code&gt;(1 − α^γ) / (1 − α)&lt;/code&gt;; when speculation lands all γ tokens, you get a free (γ+1)th token from the target&amp;rsquo;s own predictions. This is also why γ has a sweet spot rather than just a monotonic improvement curve: each additional speculation slot has lower marginal probability of being accepted, but the bonus-token term holds for all γ.&lt;/p&gt;&#10;&lt;p&gt;The drafter is typically a much smaller model from the &lt;em&gt;same model family&lt;/em&gt;; TinyLlama (1.1B) as drafter for Llama-70B, Qwen2.5-0.5B as drafter for Qwen2.5-72B, Llama-3.2-1B as drafter for Llama-3.1-70B. Sharing the tokenizer is required (the drafter and target must agree on what &lt;code&gt;x_i&lt;/code&gt; means as an integer ID). Sharing the training data distribution matters a lot for α; drafters trained on similar corpora to the target consistently outperform off-the-shelf small models, which is why production deployments increasingly use &lt;em&gt;trained-for-speculation&lt;/em&gt; drafters (EAGLE, Medusa) rather than independently-trained small models.&lt;/p&gt;&#10;&lt;h2 id="medusa"&gt;Medusa&lt;/h2&gt;&#10;&lt;p&gt;&lt;a href="https://arxiv.org/abs/2401.10774"&gt;Medusa&lt;/a&gt; (Cai et al., ICML 2024) removes the separate drafter model entirely by adding extra prediction heads to the target itself. The recipe: take a frozen target model, add k small feedforward heads on top of the last hidden layer, train each head to predict the (i+1)-th, (i+2)-th, …, (i+k)-th tokens past the current position. At inference time, one forward pass produces k+1 token predictions (the regular next-token plus the k Medusa-head predictions); a tree-attention verifier then validates a &lt;em&gt;tree&lt;/em&gt; of candidate continuations (the top-N completions from each head) in one additional forward pass.&lt;/p&gt;&#10;&lt;p&gt;Medusa heads are cheap relative to a target-model forward pass. Tree attention verifies several candidate continuations in parallel with a mask that prevents cross-candidate contamination. Its typical-acceptance rule is softer than rejection sampling and can introduce small distributional bias in exchange for better acceptance at higher temperature.&lt;/p&gt;&#10;&lt;p&gt;What Medusa pays for these properties: the heads have to be trained per target model, they consume a small amount of extra GPU memory (typically &amp;lt;5% of model size), and they&amp;rsquo;re stateful in a way that interacts with &lt;a href="https://jatinbansal.com/ai-engineering/inference-latency/"&gt;batch scheduling&lt;/a&gt;; the tree-attention kernel has to be re-instantiated per batch shape, which the &lt;a href="https://github.com/FasterDecoding/Medusa"&gt;Medusa github repo&lt;/a&gt; documents as one of the harder integration points.&lt;/p&gt;&#10;&lt;h2 id="eagle"&gt;EAGLE&lt;/h2&gt;&#10;&lt;p&gt;&lt;a href="https://arxiv.org/abs/2401.15077"&gt;EAGLE&lt;/a&gt; takes Medusa&amp;rsquo;s &amp;ldquo;use the target&amp;rsquo;s own representations&amp;rdquo; insight and runs further with it. Instead of adding token-prediction heads, EAGLE adds a &lt;em&gt;single small autoregressive decoder&lt;/em&gt; that consumes the target&amp;rsquo;s penultimate-layer hidden states (its &lt;em&gt;features&lt;/em&gt;) and predicts the next several features, which then go through the target&amp;rsquo;s own LM head to produce token probabilities. Two structural advantages over Medusa: the feature-level decoder can be trained to handle multi-step prediction more cleanly than independent token heads, and the LM head is shared with the target (no extra parameters for the vocabulary projection).&lt;/p&gt;&#10;&lt;p&gt;EAGLE-2 added dynamic tree depth; adjusting the speculation tree size per token based on confidence; which improved α by 0.05–0.10 on standard benchmarks. EAGLE-3 changed the feature target from penultimate-layer hidden states to a learned combination of multiple intermediate layers, which gave another acceptance-rate bump on long-context workloads. EAGLE-3.1, &lt;a href="https://docs.vllm.ai/en/latest/features/spec_decode.html"&gt;released on the vLLM blog&lt;/a&gt; on May 26, 2026, added robustness improvements that the team reports as up to 2× longer acceptance length compared to EAGLE-3; meaning the average number of accepted tokens per speculation step roughly doubles on the workloads they benchmarked. The vLLM integration is mature enough that you load EAGLE-3.1 weights with a single config flag:&lt;/p&gt;&#10;&lt;figure class="nb-code"&gt;&#10; &lt;header class="nb-code-chrome"&gt;&#10; &lt;span class="nb-code-label"&gt;text&lt;/span&gt;&#10; &#10; &lt;/header&gt;&#10; &lt;div class="nb-code-body"&gt;&lt;div class="highlight"&gt;&lt;pre tabindex="0" class="chroma"&gt;&lt;code class="language-text" data-lang="text"&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;--speculative-config &amp;#39;{&amp;#34;model&amp;#34;:&amp;#34;yuhuili/EAGLE3-LLaMA-3.1-Instruct-8B&amp;#34;,&amp;#34;method&amp;#34;:&amp;#34;eagle3&amp;#34;,&amp;#34;num_speculative_tokens&amp;#34;:3}&amp;#39;&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;/div&gt;&#10;&lt;/figure&gt;&#10;&lt;p&gt;The configuration constraint worth knowing: EAGLE drafters need &lt;code&gt;draft_tensor_parallel_size: 1&lt;/code&gt;, even when the target model is tensor-parallel across multiple GPUs. The EAGLE drafter is small enough that splitting it across GPUs costs more in communication than it saves in compute. The &lt;a href="https://docs.vllm.ai/en/latest/features/spec_decode.html"&gt;vLLM speculative decoding docs&lt;/a&gt; walk through the configuration options in depth; the &lt;a href="https://developers.redhat.com/articles/2025/07/01/fly-eagle3-fly-faster-inference-vllm-speculative-decoding"&gt;Red Hat developer post&lt;/a&gt; has measured throughput numbers on Llama-3.1-70B that match what production deployments report.&lt;/p&gt;&#10;&lt;h2 id="prompt-lookup-and-suffix-decoding"&gt;Prompt lookup and suffix decoding&lt;/h2&gt;&#10;&lt;p&gt;The other end of the spectrum: don&amp;rsquo;t train a drafter at all. Prompt-lookup decoding (Apoorv Saxena&amp;rsquo;s &lt;a href="https://github.com/apoorvumang/prompt-lookup-decoding"&gt;original implementation&lt;/a&gt;, now integrated into Hugging Face Transformers as &lt;code&gt;prompt_lookup_num_tokens&lt;/code&gt; and into vLLM as the &lt;code&gt;ngram&lt;/code&gt; speculative method) treats the prompt and the generation-so-far as a corpus, and on each speculation step searches the corpus for the most recent k-token substring that ends with the current token, then proposes the k tokens that followed it last time. No model, no training, no extra GPU memory; just a string-search.&lt;/p&gt;&#10;&lt;p&gt;The intuition: workloads with high &lt;em&gt;self-repetition&lt;/em&gt; (RAG-grounded generation where the answer quotes the context, code generation with repeated identifiers, summarization that lifts phrases from the source, agent loops that re-emit boilerplate) have near-deterministic next-token distributions over short windows. Prompt-lookup catches exactly that pattern. On RAG and code workloads it routinely hits α = 0.6–0.8; on free-form chat (where there&amp;rsquo;s no corpus to lookup against) it falls to α = 0.2–0.3 and provides no speedup.&lt;/p&gt;&#10;&lt;p&gt;Suffix decoding is the next refinement: build a &lt;a href="https://arxiv.org/abs/2411.04975"&gt;trie over the prompt&amp;rsquo;s suffixes&lt;/a&gt; so the lookup is O(1) instead of O(prompt-length), enabling longer γ. The headline number from the Snowflake Arctic paper: 1.4–3.9× faster than n-gram speculation on the same workloads. vLLM supports both methods natively as the &lt;code&gt;ngram&lt;/code&gt; speculative configuration. The advantage of these methods over model-based drafters is operational: there&amp;rsquo;s nothing to train, nothing to deploy, nothing to keep in sync with model upgrades. The disadvantage is the workload-dependence; outside the self-repetitive cases they degrade gracefully but provide no benefit.&lt;/p&gt;&#10;&lt;h2 id="benchmarking-eagle-3-with-vllm"&gt;Benchmarking EAGLE-3 with vLLM&lt;/h2&gt;&#10;&lt;p&gt;The code below runs against a vLLM server with EAGLE-3 enabled. Launch the server:&lt;/p&gt;&#10;&lt;figure class="nb-code"&gt;&#10; &lt;header class="nb-code-chrome"&gt;&#10; &lt;span class="nb-code-label"&gt;bash&lt;/span&gt;&#10; &#10; &lt;/header&gt;&#10; &lt;div class="nb-code-body"&gt;&lt;div class="highlight"&gt;&lt;pre tabindex="0" class="chroma"&gt;&lt;code class="language-bash" data-lang="bash"&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="c1"&gt;# pip install vllm&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;vllm serve meta-llama/Llama-3.1-8B-Instruct &lt;span class="se"&gt;\&#10;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; --speculative-config &lt;span class="s1"&gt;&amp;#39;{&amp;#34;model&amp;#34;:&amp;#34;yuhuili/EAGLE3-LLaMA-3.1-Instruct-8B&amp;#34;,&amp;#34;method&amp;#34;:&amp;#34;eagle3&amp;#34;,&amp;#34;num_speculative_tokens&amp;#34;:3}&amp;#39;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;/div&gt;&#10;&lt;/figure&gt;&#10;&lt;p&gt;Then drive it from the &lt;a href="https://github.com/openai/openai-python"&gt;Anthropic-compatible OpenAI client&lt;/a&gt; (or any compatible client) and measure both wall-clock TPOT and acceptance rate. vLLM exposes acceptance metrics via Prometheus, but the per-request signal is recoverable from the chat completion&amp;rsquo;s &lt;code&gt;usage.spec_decode_metrics&lt;/code&gt; field on V1.&lt;/p&gt;&#10;&lt;figure class="nb-code"&gt;&#10; &lt;header class="nb-code-chrome"&gt;&#10; &lt;span class="nb-code-label"&gt;python&lt;/span&gt;&#10; &#10; &lt;/header&gt;&#10; &lt;div class="nb-code-body"&gt;&lt;div class="highlight"&gt;&lt;div class="chroma"&gt;&#10;&lt;table class="lntable"&gt;&lt;tr&gt;&lt;td class="lntd"&gt;&#10;&lt;pre tabindex="0" class="chroma"&gt;&lt;code&gt;&lt;span class="lnt"&gt; 1&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 2&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 3&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 4&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 5&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 6&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 7&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 8&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 9&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 10&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 11&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 12&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 13&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 14&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 15&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 16&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 17&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 18&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 19&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 20&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 21&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 22&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 23&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 24&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 25&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 26&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 27&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 28&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 29&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 30&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 31&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 32&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 33&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 34&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 35&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 36&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 37&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 38&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 39&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 40&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 41&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 42&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 43&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 44&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 45&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 46&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 47&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 48&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 49&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 50&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 51&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 52&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 53&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 54&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 55&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 56&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 57&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 58&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 59&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 60&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 61&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 62&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 63&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 64&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 65&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 66&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 67&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 68&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 69&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 70&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 71&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 72&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 73&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 74&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 75&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 76&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 77&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 78&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 79&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 80&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 81&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 82&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 83&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 84&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 85&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 86&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 87&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 88&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 89&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 90&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 91&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 92&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 93&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 94&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 95&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 96&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 97&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 98&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 99&#10;&lt;/span&gt;&lt;span class="lnt"&gt;100&#10;&lt;/span&gt;&lt;span class="lnt"&gt;101&#10;&lt;/span&gt;&lt;span class="lnt"&gt;102&#10;&lt;/span&gt;&lt;span class="lnt"&gt;103&#10;&lt;/span&gt;&lt;span class="lnt"&gt;104&#10;&lt;/span&gt;&lt;span class="lnt"&gt;105&#10;&lt;/span&gt;&lt;span class="lnt"&gt;106&#10;&lt;/span&gt;&lt;span class="lnt"&gt;107&#10;&lt;/span&gt;&lt;span class="lnt"&gt;108&#10;&lt;/span&gt;&lt;span class="lnt"&gt;109&#10;&lt;/span&gt;&lt;span class="lnt"&gt;110&#10;&lt;/span&gt;&lt;span class="lnt"&gt;111&#10;&lt;/span&gt;&lt;span class="lnt"&gt;112&#10;&lt;/span&gt;&lt;span class="lnt"&gt;113&#10;&lt;/span&gt;&lt;span class="lnt"&gt;114&#10;&lt;/span&gt;&lt;span class="lnt"&gt;115&#10;&lt;/span&gt;&lt;span class="lnt"&gt;116&#10;&lt;/span&gt;&lt;span class="lnt"&gt;117&#10;&lt;/span&gt;&lt;span class="lnt"&gt;118&#10;&lt;/span&gt;&lt;span class="lnt"&gt;119&#10;&lt;/span&gt;&lt;span class="lnt"&gt;120&#10;&lt;/span&gt;&lt;span class="lnt"&gt;121&#10;&lt;/span&gt;&lt;span class="lnt"&gt;122&#10;&lt;/span&gt;&lt;span class="lnt"&gt;123&#10;&lt;/span&gt;&lt;span class="lnt"&gt;124&#10;&lt;/span&gt;&lt;span class="lnt"&gt;125&#10;&lt;/span&gt;&lt;span class="lnt"&gt;126&#10;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/td&gt;&#10;&lt;td class="lntd"&gt;&#10;&lt;pre tabindex="0" class="chroma"&gt;&lt;code class="language-python" data-lang="python"&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="c1"&gt;# pip install openai prometheus-client requests&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="nn"&gt;os&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="nn"&gt;statistics&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="nn"&gt;time&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="kn"&gt;from&lt;/span&gt; &lt;span class="nn"&gt;dataclasses&lt;/span&gt; &lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;dataclass&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="nn"&gt;requests&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="kn"&gt;from&lt;/span&gt; &lt;span class="nn"&gt;openai&lt;/span&gt; &lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;OpenAI&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="n"&gt;client&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;OpenAI&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;base_url&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;os&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;environ&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;&amp;#34;OPENAI_BASE_URL&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;&amp;#34;http://localhost:8000/v1&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;api_key&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;#34;not-needed-for-local&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="p"&gt;)&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="n"&gt;VLLM_METRICS_URL&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;os&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;environ&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;&amp;#34;VLLM_METRICS_URL&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;&amp;#34;http://localhost:8000/metrics&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="nd"&gt;@dataclass&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="k"&gt;class&lt;/span&gt; &lt;span class="nc"&gt;SpecTrace&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;e2e_ms&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;float&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;tpot_ms&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;float&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;output_tokens&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;int&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;parse_metrics&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt; &lt;span class="o"&gt;-&amp;gt;&lt;/span&gt; &lt;span class="nb"&gt;dict&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="nb"&gt;str&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nb"&gt;float&lt;/span&gt;&lt;span class="p"&gt;]:&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="s2"&gt;&amp;#34;&amp;#34;&amp;#34;Scrape vLLM&amp;#39;s Prometheus metrics and return acceptance-rate signals.&amp;#34;&amp;#34;&amp;#34;&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;resp&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;requests&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;VLLM_METRICS_URL&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;timeout&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;2&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;resp&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;raise_for_status&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;out&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;dict&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="nb"&gt;str&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nb"&gt;float&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;{}&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;line&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;resp&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;text&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;splitlines&lt;/span&gt;&lt;span class="p"&gt;():&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;line&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;startswith&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;&amp;#34;#&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="ow"&gt;or&lt;/span&gt; &lt;span class="ow"&gt;not&lt;/span&gt; &lt;span class="n"&gt;line&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;strip&lt;/span&gt;&lt;span class="p"&gt;():&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="k"&gt;continue&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="c1"&gt;# vllm:spec_decode_num_accepted_tokens_total{...} 12345&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;name_end&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;line&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;find&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;&amp;#34;{&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="s2"&gt;&amp;#34;{&amp;#34;&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;line&lt;/span&gt; &lt;span class="k"&gt;else&lt;/span&gt; &lt;span class="n"&gt;line&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;find&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;&amp;#34; &amp;#34;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;name&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;line&lt;/span&gt;&lt;span class="p"&gt;[:&lt;/span&gt;&lt;span class="n"&gt;name_end&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;name&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="s2"&gt;&amp;#34;vllm:spec_decode_num_accepted_tokens_total&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="s2"&gt;&amp;#34;vllm:spec_decode_num_draft_tokens_total&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="s2"&gt;&amp;#34;vllm:spec_decode_num_emitted_tokens_total&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="p"&gt;}:&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;value&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nb"&gt;float&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;line&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;rsplit&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;&amp;#34; &amp;#34;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;)[&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;])&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;out&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;name&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;out&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;name&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mf"&gt;0.0&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="n"&gt;value&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;out&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;measure_one&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;prompt&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;str&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;max_tokens&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;int&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mi"&gt;256&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;-&amp;gt;&lt;/span&gt; &lt;span class="n"&gt;SpecTrace&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="s2"&gt;&amp;#34;&amp;#34;&amp;#34;Drive one streaming request and report wall-clock TPOT + e2e.&amp;#34;&amp;#34;&amp;#34;&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;t0&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;time&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;perf_counter&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;stream&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;client&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;chat&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;completions&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;create&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;model&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;#34;meta-llama/Llama-3.1-8B-Instruct&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;messages&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="p"&gt;[{&lt;/span&gt;&lt;span class="s2"&gt;&amp;#34;role&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;&amp;#34;user&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;&amp;#34;content&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;prompt&lt;/span&gt;&lt;span class="p"&gt;}],&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;max_tokens&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;max_tokens&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;stream&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="kc"&gt;True&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;stream_options&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="s2"&gt;&amp;#34;include_usage&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kc"&gt;True&lt;/span&gt;&lt;span class="p"&gt;},&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="p"&gt;)&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;first_token&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="kc"&gt;None&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;last_token&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="kc"&gt;None&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;output_tokens&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;event&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;stream&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;event&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;choices&lt;/span&gt; &lt;span class="ow"&gt;and&lt;/span&gt; &lt;span class="n"&gt;event&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;choices&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;delta&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;content&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;now&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;time&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;perf_counter&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;first_token&lt;/span&gt; &lt;span class="ow"&gt;is&lt;/span&gt; &lt;span class="kc"&gt;None&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;first_token&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;now&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;last_token&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;now&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;event&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;usage&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;output_tokens&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;event&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;usage&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;completion_tokens&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;t_end&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;time&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;perf_counter&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;first_token&lt;/span&gt; &lt;span class="ow"&gt;is&lt;/span&gt; &lt;span class="kc"&gt;None&lt;/span&gt; &lt;span class="ow"&gt;or&lt;/span&gt; &lt;span class="n"&gt;last_token&lt;/span&gt; &lt;span class="ow"&gt;is&lt;/span&gt; &lt;span class="kc"&gt;None&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="k"&gt;raise&lt;/span&gt; &lt;span class="ne"&gt;RuntimeError&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;&amp;#34;no tokens emitted&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="c1"&gt;# TPOT measured from first-token to last-token over (output_tokens - 1) gaps.&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="c1"&gt;# Falls back to e2e/output_tokens when only one token was produced.&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;decode_span_ms&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;last_token&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt; &lt;span class="n"&gt;first_token&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="mi"&gt;1000&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;tpot_ms&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;decode_span_ms&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt; &lt;span class="nb"&gt;max&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;output_tokens&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;SpecTrace&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;e2e_ms&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;t_end&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt; &lt;span class="n"&gt;t0&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="mi"&gt;1000&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;tpot_ms&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;tpot_ms&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;output_tokens&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;output_tokens&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;run_acceptance_benchmark&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;prompts&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;list&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="nb"&gt;str&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt; &lt;span class="n"&gt;n_each&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;int&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mi"&gt;20&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;-&amp;gt;&lt;/span&gt; &lt;span class="nb"&gt;dict&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="nb"&gt;str&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nb"&gt;float&lt;/span&gt;&lt;span class="p"&gt;]:&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="s2"&gt;&amp;#34;&amp;#34;&amp;#34;For each prompt, measure TPOT and compute acceptance rate as&#10;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="s2"&gt; accepted_tokens / draft_tokens over the duration of the run.&amp;#34;&amp;#34;&amp;#34;&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;m_start&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;parse_metrics&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;traces&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;list&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;SpecTrace&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;[]&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;prompt&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;prompts&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;_&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="nb"&gt;range&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;n_each&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;traces&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;append&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;measure_one&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;prompt&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;m_end&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;parse_metrics&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;accepted&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;m_end&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;&amp;#34;vllm:spec_decode_num_accepted_tokens_total&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mf"&gt;0.0&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt; &lt;span class="n"&gt;m_start&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="s2"&gt;&amp;#34;vllm:spec_decode_num_accepted_tokens_total&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mf"&gt;0.0&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="p"&gt;)&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;drafted&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;m_end&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;&amp;#34;vllm:spec_decode_num_draft_tokens_total&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mf"&gt;0.0&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt; &lt;span class="n"&gt;m_start&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="s2"&gt;&amp;#34;vllm:spec_decode_num_draft_tokens_total&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mf"&gt;0.0&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="p"&gt;)&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;emitted&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;m_end&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;&amp;#34;vllm:spec_decode_num_emitted_tokens_total&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mf"&gt;0.0&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt; &lt;span class="n"&gt;m_start&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="s2"&gt;&amp;#34;vllm:spec_decode_num_emitted_tokens_total&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mf"&gt;0.0&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="p"&gt;)&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;alpha&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;accepted&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt; &lt;span class="n"&gt;drafted&lt;/span&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;drafted&lt;/span&gt; &lt;span class="o"&gt;&amp;gt;&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt; &lt;span class="k"&gt;else&lt;/span&gt; &lt;span class="nb"&gt;float&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;&amp;#34;nan&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;tokens_per_step&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;emitted&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt; &lt;span class="nb"&gt;max&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;drafted&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt; &lt;span class="mi"&gt;3&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="c1"&gt;# γ=3 from launch config&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;tpots&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;t&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;tpot_ms&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;t&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;traces&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="s2"&gt;&amp;#34;alpha&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;alpha&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="s2"&gt;&amp;#34;mean_tokens_per_step&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;tokens_per_step&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="s2"&gt;&amp;#34;tpot_p50_ms&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;statistics&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;median&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;tpots&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="s2"&gt;&amp;#34;tpot_mean_ms&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;statistics&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;mean&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;tpots&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="p"&gt;}&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="vm"&gt;__name__&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="s2"&gt;&amp;#34;__main__&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="c1"&gt;# Two workloads with different self-repetition profiles.&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;code_prompt&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="s2"&gt;&amp;#34;Refactor the following Python function to use a dict comprehension:&lt;/span&gt;&lt;span class="se"&gt;\n\n&lt;/span&gt;&lt;span class="s2"&gt;&amp;#34;&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="s2"&gt;&amp;#34;def to_lookup(items):&lt;/span&gt;&lt;span class="se"&gt;\n&lt;/span&gt;&lt;span class="s2"&gt;&amp;#34;&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="s2"&gt;&amp;#34; out = &lt;/span&gt;&lt;span class="si"&gt;{}&lt;/span&gt;&lt;span class="se"&gt;\n&lt;/span&gt;&lt;span class="s2"&gt;&amp;#34;&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="s2"&gt;&amp;#34; for item in items:&lt;/span&gt;&lt;span class="se"&gt;\n&lt;/span&gt;&lt;span class="s2"&gt;&amp;#34;&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="s2"&gt;&amp;#34; out[item.id] = item.value&lt;/span&gt;&lt;span class="se"&gt;\n&lt;/span&gt;&lt;span class="s2"&gt;&amp;#34;&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="s2"&gt;&amp;#34; return out&lt;/span&gt;&lt;span class="se"&gt;\n&lt;/span&gt;&lt;span class="s2"&gt;&amp;#34;&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="p"&gt;)&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;chat_prompt&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s2"&gt;&amp;#34;Explain how speculative decoding interacts with continuous batching.&amp;#34;&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="nb"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;&amp;#34;-- code workload --&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="nb"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;run_acceptance_benchmark&lt;/span&gt;&lt;span class="p"&gt;([&lt;/span&gt;&lt;span class="n"&gt;code_prompt&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt; &lt;span class="n"&gt;n_each&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;20&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="nb"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;&amp;#34;-- chat workload --&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="nb"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;run_acceptance_benchmark&lt;/span&gt;&lt;span class="p"&gt;([&lt;/span&gt;&lt;span class="n"&gt;chat_prompt&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt; &lt;span class="n"&gt;n_each&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;20&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/td&gt;&lt;/tr&gt;&lt;/table&gt;&#10;&lt;/div&gt;&#10;&lt;/div&gt;&lt;/div&gt;&#10;&lt;/figure&gt;&#10;&lt;p&gt;The &lt;code&gt;parse_metrics&lt;/code&gt; helper uses vLLM&amp;rsquo;s Prometheus endpoint and therefore depends on the metric names in the installed version. Acceptance rates far below the expected workload baseline usually indicate a tokenizer or drafter-target training mismatch.&lt;/p&gt;&#10;&lt;h2 id="references"&gt;References&lt;/h2&gt;&#10;&lt;ul&gt;&#10;&lt;li&gt;&lt;a href="https://arxiv.org/abs/2211.17192"&gt;Fast Inference from Transformers via Speculative Decoding&lt;/a&gt;; Leviathan, Kalman, Matias (Google, ICML 2023). The foundational paper that introduced the modified-rejection-sampling formulation and proved output-distribution equivalence to the target. Read this first.&lt;/li&gt;&#10;&lt;li&gt;&lt;a href="https://arxiv.org/abs/2401.10774"&gt;Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads&lt;/a&gt;; Cai et al. (ICML 2024). The drafter-free approach that grafts prediction heads onto the target, with typical-acceptance as the relaxed acceptance criterion. The &lt;a href="https://github.com/FasterDecoding/Medusa"&gt;Medusa github repo&lt;/a&gt; has the reference implementation.&lt;/li&gt;&#10;&lt;li&gt;&lt;a href="https://arxiv.org/abs/2401.15077"&gt;EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty&lt;/a&gt;; Li et al. (2024). The feature-level autoregressive drafter that became the production default; EAGLE-2/3/3.1 are progressive refinements documented in subsequent papers and the &lt;a href="https://github.com/SafeAILab/EAGLE"&gt;EAGLE github repo&lt;/a&gt;.&lt;/li&gt;&#10;&lt;li&gt;&lt;a href="https://arxiv.org/abs/2601.11580"&gt;Speculative Decoding: Performance or Illusion?&lt;/a&gt;; a 2026 measurement study of speculative-decoding in vLLM across workloads, model scales, and batch sizes. The strongest empirical treatment of when speculation actually pays off in production-shaped traffic.&lt;/li&gt;&#10;&lt;/ul&gt;&#10;</content:encoded></item><item><title>Inference Latency: Prefill, Decode, and Batching</title><link>https://jatinbansal.com/ai-engineering/inference-latency/</link><guid isPermaLink="true">https://jatinbansal.com/ai-engineering/inference-latency/</guid><pubDate>Tue, 26 May 2026 17:42:00 +0000</pubDate><dc:creator>Jatin Bansal</dc:creator><category>AI Engineering</category><description>Inference latency across prefill, decode, batching, chunked prefill, and disaggregated serving.</description><content:encoded>&lt;p&gt;Self-hosting changes who owns the inference scheduler. The same model can show a better median and a much worse p99 when long prompts share batches with short, actively decoding conversations. Understanding that result requires separating queueing, prefill, and decode, then measuring how the server schedules each phase.&lt;/p&gt;&#10;&lt;h2 id="definition"&gt;Definition&lt;/h2&gt;&#10;&lt;p&gt;Inference latency for an LLM is the wall-clock time a request spends inside the serving stack, decomposed into three orthogonal phases; queueing, prefill, and decode; and measured by three metrics that capture different parts of the user experience: TTFT, TPOT, and end-to-end latency. &lt;em&gt;Queueing time&lt;/em&gt; is how long the request sat waiting before the scheduler picked it up. &lt;em&gt;Prefill time&lt;/em&gt; is the compute-bound forward pass over the input prompt that produces the KV cache and the first output token. &lt;em&gt;Decode time&lt;/em&gt; is the memory-bandwidth-bound loop that auto-regressively generates each subsequent token. The metrics: TTFT (time-to-first-token) equals queueing plus prefill; this is the latency the user perceives until &lt;a href="https://jatinbansal.com/ai-engineering/streaming/"&gt;streaming&lt;/a&gt; starts. TPOT (time-per-output-token), sometimes called ITL (inter-token latency), is the average gap between consecutive decoded tokens; this is the latency the user perceives &lt;em&gt;during&lt;/em&gt; streaming. End-to-end latency is TTFT plus TPOT × output token count; this is the latency a non-streaming caller sees. A 200-token answer at TTFT=200ms and TPOT=30ms is e2e = 200 + 200×30 = 6.2s; TTFT dominates the perceived experience for streaming UIs and decode dominates the cost.&lt;/p&gt;&#10;&lt;p&gt;The three phases respond to different optimizations. Queueing time falls when the scheduler can fit a new request into an in-flight batch; the whole point of continuous batching. Prefill time falls when the server splits long prompts into chunks that coexist with decode steps; chunked prefill; or when a stable prefix is already cached on the GPU; &lt;a href="https://jatinbansal.com/ai-engineering/prompt-caching/"&gt;prompt caching&lt;/a&gt;. Decode time per token falls when the GPU&amp;rsquo;s memory bandwidth is fully saturated, which means &lt;em&gt;more concurrent decodes in the same batch&lt;/em&gt;, which means more memory pressure on the KV cache, which is why &lt;a href="https://blog.vllm.ai/2023/06/20/vllm.html"&gt;PagedAttention&lt;/a&gt; exists. The dials interact, and a well-tuned server is one where TTFT and TPOT trade off cleanly against batch size and total throughput rather than collapsing into the bimodal worst case the team in the opener hit.&lt;/p&gt;&#10;&lt;h2 id="mechanics-prefill-vs-decode-what-each-phase-does"&gt;Mechanics: prefill vs decode, what each phase does&lt;/h2&gt;&#10;&lt;p&gt;The split is mechanical. A request arrives with &lt;code&gt;prompt_tokens = [t₁, t₂, …, tₙ]&lt;/code&gt;. The server runs a single forward pass over all n tokens in parallel; every token attends to every prior token, producing the K/V tensors for each layer. This is prefill. At the end, the server samples the (n+1)th token from the model&amp;rsquo;s output distribution. Total work is O(n²) in the attention layers (every token attends to every other) and O(n) in the FFN layers; modern GPUs saturate compute around a prompt length of a few thousand tokens for a 70B-class model. TTFT is dominated by this single forward pass plus any queueing time before the scheduler admitted the request.&lt;/p&gt;&#10;&lt;p&gt;After prefill the request enters the decode loop. Each iteration: feed the previously sampled token through one forward pass, where the new token&amp;rsquo;s query attends to the entire stored K/V cache (no recompute needed), sample the next token from the output distribution, append it to the KV cache. Each step touches one query token against a growing K/V state; the FLOPs are negligible, but the memory traffic; reading the model weights and the full K/V cache from HBM on every step; is the bottleneck. A 70B-class model at FP16 reads ~140 GB of weights per step; on an H100 with ~3 TB/s of HBM bandwidth, the theoretical floor for TPOT is ~50ms per token &lt;em&gt;for one sequence&lt;/em&gt;. The whole point of batching is to amortize that 140 GB read across many concurrent decodes; at batch size 16, the per-token cost in bandwidth is 1/16th, and TPOT can fall toward ~3ms per token in the limit. This is why a busy inference server with many concurrent users is &lt;em&gt;cheaper per token&lt;/em&gt; than an idle one. (&lt;a href="https://jatinbansal.com/ai-engineering/quantization-distillation/"&gt;Quantization&lt;/a&gt; attacks the same equation from the other side: an INT4-quantized 70B model reads ~35 GB per step instead of 140 GB, dropping the single-sequence TPOT floor by 4×.)&lt;/p&gt;&#10;&lt;p&gt;A consequence that bites: the optimal batch shape changes between prefill and decode, and the scheduler has to manage both. A naive batch of &amp;ldquo;everyone&amp;rsquo;s prefill plus everyone&amp;rsquo;s decode&amp;rdquo; stalls because the prefill phase dominates GPU time and the decode tokens pile up waiting. The 2022 Orca paper called this the &lt;em&gt;static-batching&lt;/em&gt; failure mode and observed that throughput could improve by an order of magnitude when batching happened at the iteration level; the scheduler picks which sequences participate in each forward pass independently, with prefills and decodes coexisting only when their compute profiles allow it.&lt;/p&gt;&#10;&lt;h2 id="mechanics-continuous-batching-the-orca-primitive"&gt;Mechanics: continuous batching (the Orca primitive)&lt;/h2&gt;&#10;&lt;p&gt;Iteration-level scheduling, the technique &lt;a href="https://www.usenix.org/conference/osdi22/presentation/yu"&gt;Orca&lt;/a&gt; introduced and every modern inference engine implements, runs one forward pass per scheduler decision rather than one request per scheduler decision. At the start of each iteration the scheduler picks a &lt;em&gt;batch&lt;/em&gt; of sequences to run in this forward pass: each sequence has its current K/V state, its next-token-to-process (a prompt token for prefilling sequences, a previously-sampled token for decoding sequences), and a free slot in the batch tensor. The forward pass runs. Each sequence&amp;rsquo;s output is appended to its K/V state; decoding sequences also emit one sampled token. Then the scheduler picks the next iteration&amp;rsquo;s batch; and that batch can be &lt;em&gt;different&lt;/em&gt;: a newly-arrived request that wants to start prefilling can join, a sequence that just hit EOS leaves, an in-flight long-output decode continues.&lt;/p&gt;&#10;&lt;p&gt;The vocabulary that grew up around this: &lt;em&gt;continuous batching&lt;/em&gt; (the marketing name, popularized by &lt;a href="https://www.anyscale.com/blog/continuous-batching-llm-inference"&gt;Anyscale&amp;rsquo;s 2023 blog post on 23× throughput&lt;/a&gt;), &lt;em&gt;iteration-level batching&lt;/em&gt; (Orca&amp;rsquo;s term), &lt;em&gt;in-flight batching&lt;/em&gt; (NVIDIA&amp;rsquo;s term in TensorRT-LLM). They mean the same thing. The contrast is with &lt;em&gt;static batching&lt;/em&gt;, where a batch is formed once at admission and runs to completion before the next batch starts; static batching wastes GPU cycles whenever any sequence in the batch finishes early, and it forces newly arrived requests to wait for the entire batch to drain before they can start prefilling.&lt;/p&gt;&#10;&lt;p&gt;What continuous batching &lt;em&gt;doesn&amp;rsquo;t&lt;/em&gt; solve on its own is the head-of-line blocking problem when a long prefill enters the batch. A single 50k-token prefill takes ~500ms-2s of GPU time (model- and hardware-dependent); during that forward pass, every in-flight decode sees its next token &lt;em&gt;delayed by exactly that amount&lt;/em&gt;. TPOT for users mid-conversation spikes from 30ms to 1000ms+. This is the bimodal latency mode the team in the opener was hitting. The fix is the next primitive.&lt;/p&gt;&#10;&lt;h2 id="mechanics-chunked-prefill-the-sarathi-primitive"&gt;Mechanics: chunked prefill (the Sarathi primitive)&lt;/h2&gt;&#10;&lt;p&gt;Chunked prefill splits a long prompt&amp;rsquo;s prefill into many smaller forward passes; chunks of say 512-2048 tokens; and &lt;em&gt;piggybacks&lt;/em&gt; decode steps into the same forward pass as each prefill chunk. The technique came from the 2023 &lt;a href="https://arxiv.org/abs/2308.16369"&gt;SARATHI paper&lt;/a&gt; and was extended into a production scheduler in &lt;a href="https://www.usenix.org/conference/osdi24/presentation/agrawal"&gt;Sarathi-Serve at OSDI &amp;lsquo;24&lt;/a&gt;. Each scheduled iteration carries a budget of &lt;code&gt;max_num_batched_tokens&lt;/code&gt;; say 2048; and the scheduler fills that budget greedily: in-flight decodes (one token each), then as many new prefill chunks as fit. A 50k-token prefill becomes 25 iterations of 2048 tokens each, and during all 25 iterations the in-flight decodes are &lt;em&gt;still making progress&lt;/em&gt;; one token per iteration, no stalling.&lt;/p&gt;&#10;&lt;p&gt;The trade-off is direct. Smaller &lt;code&gt;max_num_batched_tokens&lt;/code&gt; (e.g. 2048) gives better TPOT; fewer prefill tokens per iteration means decodes don&amp;rsquo;t get drowned out, and inter-token latency stays low. Larger &lt;code&gt;max_num_batched_tokens&lt;/code&gt; (e.g. 8192–16384) gives better TTFT; more prefill tokens per iteration means the prefilling sequence finishes its first token sooner. The vLLM docs frame this as: small for ITL-sensitive workloads (chat, coding assistants), large for TTFT-sensitive workloads (single-question UX, batch document processing). The default in vLLM V1 enables chunked prefill automatically; the &lt;a href="https://huggingface.co/docs/text-generation-inference/"&gt;Hugging Face TGI&lt;/a&gt; &lt;code&gt;--max-batch-prefill-tokens&lt;/code&gt; and &lt;code&gt;--waiting-served-ratio&lt;/code&gt; knobs control the same trade-off with different names. Anthropic&amp;rsquo;s June 2024 &lt;a href="https://www.databricks.com/blog/llm-inference-performance-engineering-best-practices"&gt;piggyback batching post&lt;/a&gt; (joint with Databricks Mosaic) reports 2-3× throughput improvements over un-chunked vLLM at SLO-pinned latencies on standard chat workloads.&lt;/p&gt;&#10;&lt;p&gt;A subtler benefit: chunked prefill also smooths the &lt;em&gt;service-time distribution&lt;/em&gt; in queueing-theory terms. Without it, prefill service times have a long right tail (proportional to prompt length); with it, every iteration&amp;rsquo;s service time is bounded by &lt;code&gt;max_num_batched_tokens&lt;/code&gt;, and the per-iteration GPU time becomes nearly constant. Kingman&amp;rsquo;s formula says waiting time scales with the variance of service time; bounding service-time variance shaves p99 latency even when the mean is unchanged.&lt;/p&gt;&#10;&lt;h2 id="mechanics-prefilldecode-disaggregation"&gt;Mechanics: prefill/decode disaggregation&lt;/h2&gt;&#10;&lt;p&gt;Even with chunked prefill, prefill and decode contend for the same GPU&amp;rsquo;s compute and memory. The next architectural step, introduced in the 2024 &lt;a href="https://arxiv.org/abs/2401.09670"&gt;DistServe paper&lt;/a&gt; and Microsoft&amp;rsquo;s contemporaneous &lt;a href="https://www.microsoft.com/en-us/research/blog/splitwise-improves-gpu-usage-by-splitting-llm-inference-phases/"&gt;SplitWise&lt;/a&gt;, is to run prefill on one pool of GPUs and decode on a separate pool, with the KV cache shipped from prefill-GPU to decode-GPU between phases. DistServe reports up to 7.4× higher goodput (requests per second meeting both TTFT and TPOT SLOs) and 12.6× tighter SLO bounds at the same load compared to co-located serving, by eliminating the prefill/decode interference entirely. SplitWise extends this further by using &lt;em&gt;different hardware classes&lt;/em&gt; for the two pools; compute-optimized GPUs (H100, B100) for prefill, memory-bandwidth-optimized GPUs (A100, GH200) for decode; squeezing more throughput-per-dollar from the heterogeneous hardware match. The &lt;a href="https://haoailab.com/blogs/distserve-retro/"&gt;Hao AI Lab retrospective from 18 months later&lt;/a&gt; reports that disaggregation has now become production infrastructure across NVIDIA Dynamo, llm-d, vLLM (experimental &lt;a href="https://docs.vllm.ai/en/latest/features/disagg_prefill/"&gt;disaggregated prefill&lt;/a&gt;), &lt;a href="https://github.com/sgl-project/sglang"&gt;SGLang&lt;/a&gt;, and Moonshot AI&amp;rsquo;s MoonCake; disaggregation is the production-default for high-scale serving in 2026.&lt;/p&gt;&#10;&lt;p&gt;When does disaggregation pay off? When prefill-bound workloads dominate (very long context inputs, RAG over large retrieved chunks, document-summarization batch jobs) the prefill pool can scale independently of decode; when decode-bound workloads dominate (short prompts, long outputs, agent-style streaming) the decode pool scales independently. Co-located serving makes the wrong trade-off in both directions: prefill-bound traffic starves decode, decode-bound traffic leaves prefill compute idle. For symmetric chat traffic (mixed prompt and output lengths) chunked prefill on a co-located server is usually enough; for asymmetric or long-context-heavy workloads, disaggregation is the lever.&lt;/p&gt;&#10;&lt;h2 id="code-measuring-ttft-tpot-and-end-to-end-latency-in-python"&gt;Code: measuring TTFT, TPOT, and end-to-end latency in Python&lt;/h2&gt;&#10;&lt;p&gt;The code below measures TTFT, TPOT, and end-to-end latency against a local vLLM OpenAI-compatible server. Install: &lt;code&gt;pip install openai&lt;/code&gt;. Start vLLM with &lt;code&gt;vllm serve Qwen/Qwen3-4B-Instruct --max-num-batched-tokens 2048&lt;/code&gt; (or your model and chunk size of choice). The same client also works against &lt;a href="https://huggingface.co/docs/text-generation-inference/"&gt;TGI&lt;/a&gt; or hosted endpoints with &lt;code&gt;OPENAI_BASE_URL&lt;/code&gt; overridden; the wire format is the same.&lt;/p&gt;&#10;&lt;figure class="nb-code"&gt;&#10; &lt;header class="nb-code-chrome"&gt;&#10; &lt;span class="nb-code-label"&gt;python&lt;/span&gt;&#10; &#10; &lt;/header&gt;&#10; &lt;div class="nb-code-body"&gt;&lt;div class="highlight"&gt;&lt;div class="chroma"&gt;&#10;&lt;table class="lntable"&gt;&lt;tr&gt;&lt;td class="lntd"&gt;&#10;&lt;pre tabindex="0" class="chroma"&gt;&lt;code&gt;&lt;span class="lnt"&gt; 1&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 2&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 3&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 4&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 5&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 6&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 7&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 8&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 9&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 10&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 11&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 12&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 13&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 14&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 15&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 16&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 17&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 18&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 19&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 20&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 21&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 22&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 23&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 24&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 25&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 26&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 27&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 28&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 29&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 30&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 31&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 32&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 33&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 34&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 35&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 36&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 37&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 38&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 39&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 40&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 41&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 42&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 43&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 44&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 45&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 46&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 47&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 48&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 49&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 50&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 51&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 52&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 53&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 54&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 55&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 56&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 57&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 58&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 59&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 60&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 61&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 62&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 63&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 64&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 65&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 66&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 67&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 68&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 69&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 70&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 71&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 72&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 73&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 74&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 75&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 76&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 77&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 78&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 79&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 80&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 81&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 82&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 83&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 84&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 85&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 86&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 87&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 88&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 89&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 90&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 91&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 92&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 93&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 94&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 95&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 96&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 97&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 98&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 99&#10;&lt;/span&gt;&lt;span class="lnt"&gt;100&#10;&lt;/span&gt;&lt;span class="lnt"&gt;101&#10;&lt;/span&gt;&lt;span class="lnt"&gt;102&#10;&lt;/span&gt;&lt;span class="lnt"&gt;103&#10;&lt;/span&gt;&lt;span class="lnt"&gt;104&#10;&lt;/span&gt;&lt;span class="lnt"&gt;105&#10;&lt;/span&gt;&lt;span class="lnt"&gt;106&#10;&lt;/span&gt;&lt;span class="lnt"&gt;107&#10;&lt;/span&gt;&lt;span class="lnt"&gt;108&#10;&lt;/span&gt;&lt;span class="lnt"&gt;109&#10;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/td&gt;&#10;&lt;td class="lntd"&gt;&#10;&lt;pre tabindex="0" class="chroma"&gt;&lt;code class="language-python" data-lang="python"&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="c1"&gt;# pip install openai&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="nn"&gt;os&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="nn"&gt;statistics&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="nn"&gt;time&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="kn"&gt;from&lt;/span&gt; &lt;span class="nn"&gt;dataclasses&lt;/span&gt; &lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;dataclass&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="kn"&gt;from&lt;/span&gt; &lt;span class="nn"&gt;openai&lt;/span&gt; &lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;OpenAI&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="n"&gt;client&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;OpenAI&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;base_url&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;os&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;environ&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;&amp;#34;OPENAI_BASE_URL&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;&amp;#34;http://localhost:8000/v1&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;api_key&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;#34;not-needed-for-local&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="p"&gt;)&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="nd"&gt;@dataclass&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="k"&gt;class&lt;/span&gt; &lt;span class="nc"&gt;LatencyTrace&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;ttft_ms&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;float&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;tpot_ms&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;float&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;e2e_ms&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;float&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;output_tokens&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;int&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;measure_one&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;prompt&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;str&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;max_tokens&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;int&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mi"&gt;256&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;model&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;str&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s2"&gt;&amp;#34;Qwen/Qwen3-4B-Instruct&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;-&amp;gt;&lt;/span&gt; &lt;span class="n"&gt;LatencyTrace&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="s2"&gt;&amp;#34;&amp;#34;&amp;#34;One streaming call, with per-token timing.&#10;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="s2"&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="s2"&gt; TTFT is measured from request send to the first non-empty content delta.&#10;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="s2"&gt; TPOT is the mean inter-token gap *after* the first token. e2e is wall time.&#10;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="s2"&gt; &amp;#34;&amp;#34;&amp;#34;&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;t0&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;time&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;perf_counter&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;stream&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;client&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;chat&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;completions&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;create&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;model&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;model&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;messages&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="p"&gt;[{&lt;/span&gt;&lt;span class="s2"&gt;&amp;#34;role&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;&amp;#34;user&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;&amp;#34;content&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;prompt&lt;/span&gt;&lt;span class="p"&gt;}],&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;max_tokens&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;max_tokens&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;stream&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="kc"&gt;True&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="c1"&gt;# request usage in the final chunk for accurate token counts&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;stream_options&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="s2"&gt;&amp;#34;include_usage&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kc"&gt;True&lt;/span&gt;&lt;span class="p"&gt;},&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="p"&gt;)&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;first_token_time&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="kc"&gt;None&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;token_times&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;list&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="nb"&gt;float&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;[]&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;completion_tokens&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;event&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;stream&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="c1"&gt;# Per-chunk events; last chunk carries `usage` and no content.&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;event&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;choices&lt;/span&gt; &lt;span class="ow"&gt;and&lt;/span&gt; &lt;span class="n"&gt;event&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;choices&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;delta&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;content&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;now&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;time&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;perf_counter&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;first_token_time&lt;/span&gt; &lt;span class="ow"&gt;is&lt;/span&gt; &lt;span class="kc"&gt;None&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;first_token_time&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;now&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="k"&gt;else&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;token_times&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;append&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;now&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;event&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;usage&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;completion_tokens&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;event&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;usage&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;completion_tokens&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;t_end&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;time&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;perf_counter&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;first_token_time&lt;/span&gt; &lt;span class="ow"&gt;is&lt;/span&gt; &lt;span class="kc"&gt;None&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="k"&gt;raise&lt;/span&gt; &lt;span class="ne"&gt;RuntimeError&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;&amp;#34;no tokens emitted&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;ttft_ms&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;first_token_time&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt; &lt;span class="n"&gt;t0&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="mi"&gt;1000&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="c1"&gt;# mean inter-token gap; fall back to e2e/output for single-token responses&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="nb"&gt;len&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;token_times&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;&amp;gt;=&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;gaps&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;b&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt; &lt;span class="n"&gt;a&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="mi"&gt;1000&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;a&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;b&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="nb"&gt;zip&lt;/span&gt;&lt;span class="p"&gt;([&lt;/span&gt;&lt;span class="n"&gt;first_token_time&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="n"&gt;token_times&lt;/span&gt;&lt;span class="p"&gt;[:&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt; &lt;span class="n"&gt;token_times&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="p"&gt;]&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;tpot_ms&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;statistics&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;mean&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;gaps&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="k"&gt;else&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;tpot_ms&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mf"&gt;0.0&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;e2e_ms&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;t_end&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt; &lt;span class="n"&gt;t0&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="mi"&gt;1000&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;LatencyTrace&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;ttft_ms&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;tpot_ms&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;e2e_ms&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;completion_tokens&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;percentile&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;values&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;list&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="nb"&gt;float&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt; &lt;span class="n"&gt;p&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;float&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;-&amp;gt;&lt;/span&gt; &lt;span class="nb"&gt;float&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="ow"&gt;not&lt;/span&gt; &lt;span class="n"&gt;values&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="nb"&gt;float&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;&amp;#34;nan&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;s&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nb"&gt;sorted&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;values&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;k&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nb"&gt;len&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;s&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="n"&gt;p&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt; &lt;span class="mi"&gt;100&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;f&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nb"&gt;int&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;k&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;s&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;f&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;k&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt; &lt;span class="n"&gt;f&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;s&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="nb"&gt;min&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;f&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nb"&gt;len&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;s&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;)]&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt; &lt;span class="n"&gt;s&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;f&lt;/span&gt;&lt;span class="p"&gt;])&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;run_benchmark&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;prompts&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;list&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="nb"&gt;str&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt; &lt;span class="n"&gt;n_each&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;int&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mi"&gt;20&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;-&amp;gt;&lt;/span&gt; &lt;span class="nb"&gt;dict&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="nb"&gt;str&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nb"&gt;float&lt;/span&gt;&lt;span class="p"&gt;]:&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="s2"&gt;&amp;#34;&amp;#34;&amp;#34;Sequential probe to characterize an idle server; for load testing,&#10;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="s2"&gt; drive concurrent workers (asyncio + a semaphore) and merge their traces.&amp;#34;&amp;#34;&amp;#34;&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;traces&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;list&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;LatencyTrace&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;[]&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;prompt&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;prompts&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;_&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="nb"&gt;range&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;n_each&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;traces&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;append&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;measure_one&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;prompt&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;ttfts&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;t&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;ttft_ms&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;t&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;traces&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;tpots&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;t&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;tpot_ms&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;t&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;traces&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;e2es&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;t&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;e2e_ms&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;t&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;traces&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="s2"&gt;&amp;#34;ttft_p50_ms&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;percentile&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;ttfts&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;50&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="s2"&gt;&amp;#34;ttft_p99_ms&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;percentile&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;ttfts&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;99&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="s2"&gt;&amp;#34;tpot_p50_ms&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;percentile&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;tpots&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;50&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="s2"&gt;&amp;#34;tpot_p99_ms&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;percentile&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;tpots&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;99&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="s2"&gt;&amp;#34;e2e_p50_ms&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;percentile&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;e2es&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;50&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="s2"&gt;&amp;#34;e2e_p99_ms&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;percentile&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;e2es&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;99&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="p"&gt;}&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="vm"&gt;__name__&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="s2"&gt;&amp;#34;__main__&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="c1"&gt;# Mixed prompt lengths to exercise the chunked-prefill scheduler.&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;short_prompt&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s2"&gt;&amp;#34;Write a haiku about distributed systems.&amp;#34;&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;long_prompt&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s2"&gt;&amp;#34;Summarize the key points in this passage:&lt;/span&gt;&lt;span class="se"&gt;\n\n&lt;/span&gt;&lt;span class="s2"&gt;&amp;#34;&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;&amp;#34;Continuous batching is iteration-level scheduling. &amp;#34;&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="mi"&gt;200&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;results&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;run_benchmark&lt;/span&gt;&lt;span class="p"&gt;([&lt;/span&gt;&lt;span class="n"&gt;short_prompt&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;long_prompt&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt; &lt;span class="n"&gt;n_each&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;10&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;k&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;v&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;results&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;items&lt;/span&gt;&lt;span class="p"&gt;():&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="nb"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sa"&gt;f&lt;/span&gt;&lt;span class="s2"&gt;&amp;#34;&lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;k&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s2"&gt;: &lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;v&lt;/span&gt;&lt;span class="si"&gt;:&lt;/span&gt;&lt;span class="s2"&gt;.1f&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s2"&gt;&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/td&gt;&lt;/tr&gt;&lt;/table&gt;&#10;&lt;/div&gt;&#10;&lt;/div&gt;&lt;/div&gt;&#10;&lt;/figure&gt;&#10;&lt;p&gt;the &lt;code&gt;stream_options={&amp;quot;include_usage&amp;quot;: True}&lt;/code&gt; flag is how vLLM and OpenAI return accurate output token counts in streaming mode; without it, you&amp;rsquo;d have to tokenize the response yourself. The sequential probe in &lt;code&gt;run_benchmark&lt;/code&gt; characterizes the &lt;em&gt;idle&lt;/em&gt; server; to find the bimodal-latency regime the team in the opener hit, you need a &lt;em&gt;concurrent&lt;/em&gt; load generator that holds the server at ~70-90% utilization while a single long-prompt request lands. The latency under load is what tells you whether your chunked-prefill budget is set right. A good rule of thumb: run the same harness twice, once with &lt;code&gt;--max-num-batched-tokens 2048&lt;/code&gt; and once with &lt;code&gt;--max-num-batched-tokens 16384&lt;/code&gt;; the gap between the two TTFT p99s and the gap between the two TPOT p99s reveal the trade-off frontier for your traffic.&lt;/p&gt;&#10;&lt;h2 id="tuning-the-dials-what-to-set-and-why"&gt;Tuning the dials: what to set and why&lt;/h2&gt;&#10;&lt;p&gt;The four most operationally important knobs on a vLLM-class server:&lt;/p&gt;&#10;&lt;p&gt;&lt;code&gt;max_num_batched_tokens&lt;/code&gt; (vLLM) / &lt;code&gt;--max-batch-prefill-tokens&lt;/code&gt; (TGI). The total token budget per scheduler iteration. Smaller (1024-2048) prioritizes TPOT; larger (8192-16384) prioritizes TTFT. Default in vLLM V1 is workload-tuned and varies by version; explicit setting is recommended for production deployments. Tune by running the latency harness above at two values and choosing the one that hits your SLO mix.&lt;/p&gt;&#10;&lt;p&gt;&lt;code&gt;max_num_seqs&lt;/code&gt; (vLLM) / &lt;code&gt;--max-concurrent-requests&lt;/code&gt; (TGI). The max number of in-flight sequences the scheduler will admit into a batch. Set this near the GPU&amp;rsquo;s KV-cache capacity at your typical context length; setting it too high causes preemption thrashing (sequences get evicted to make room and have to recompute prefill), setting it too low underutilizes decode bandwidth. The vLLM logs print KV-cache utilization at admission; if it sits at 95-100% during peak load you&amp;rsquo;re undersized; if it sits at &amp;lt;50% during peak you have headroom to raise concurrency.&lt;/p&gt;&#10;&lt;p&gt;Chunked prefill on/off. Default-on in vLLM V1 for most cases. Turn it off only if you&amp;rsquo;re running an exclusively prefill-dominated workload (e.g. batch document summarization) where there are no concurrent decodes to interleave with; in that case, the iteration overhead is pure cost.&lt;/p&gt;&#10;&lt;p&gt;Prefix caching on/off. vLLM V1&amp;rsquo;s &lt;a href="https://docs.vllm.ai/en/latest/features/automatic_prefix_caching.html"&gt;automatic prefix caching&lt;/a&gt; is the open-weights analogue of &lt;a href="https://jatinbansal.com/ai-engineering/prompt-caching/"&gt;Anthropic/OpenAI prompt caching&lt;/a&gt;: a content-addressed cache of K/V tensors for prompt prefixes, with &amp;lt;1% throughput overhead even at 0% hit rate (V1&amp;rsquo;s optimization). Leave it on. Then structure your prompts so prefixes are stable; system prompt and tool schemas at the top, retrieved context in the middle, user message at the bottom; to maximize cache hit rate.&lt;/p&gt;&#10;&lt;p&gt;For disaggregated serving the &lt;a href="https://docs.vllm.ai/en/latest/features/disagg_prefill/"&gt;vLLM disaggregated prefill docs&lt;/a&gt; cover the experimental V1 path; for production-scale deployments, &lt;a href="https://github.com/ai-dynamo/dynamo"&gt;NVIDIA Dynamo&lt;/a&gt; and &lt;a href="https://docs.sglang.io/advanced_features/pd_disaggregation.html"&gt;SGLang&amp;rsquo;s PD disaggregation docs&lt;/a&gt; are the maturity leaders in mid-2026.&lt;/p&gt;&#10;&lt;h2 id="further-reading-from-the-field"&gt;Further reading from the field&lt;/h2&gt;&#10;&lt;ul&gt;&#10;&lt;li&gt;&lt;a href="https://www.usenix.org/conference/osdi22/presentation/yu"&gt;Orca: A Distributed Serving System for Transformer-Based Generative Models&lt;/a&gt;; the OSDI &amp;lsquo;22 paper that introduced iteration-level (continuous) batching and selective batching. The mechanical foundation for every modern LLM serving system.&lt;/li&gt;&#10;&lt;li&gt;&lt;a href="https://arxiv.org/abs/2403.02310"&gt;Sarathi-Serve: Taming Throughput-Latency Tradeoff in LLM Inference&lt;/a&gt;; the OSDI &amp;lsquo;24 paper that productionized chunked prefill and stall-free batching, with explicit measurements of head-of-line blocking and how to eliminate it.&lt;/li&gt;&#10;&lt;li&gt;&lt;a href="https://arxiv.org/abs/2401.09670"&gt;DistServe: Disaggregating Prefill and Decoding for Goodput-optimized LLM Serving&lt;/a&gt;; the OSDI &amp;lsquo;24 paper that established the prefill/decode disaggregation pattern now adopted in production across the stack.&lt;/li&gt;&#10;&lt;li&gt;&lt;a href="https://www.anyscale.com/blog/continuous-batching-llm-inference"&gt;Anyscale: How continuous batching enables 23x throughput in LLM inference while reducing p50 latency&lt;/a&gt;; the 2023 blog post that took Orca&amp;rsquo;s idea mainstream with measured numbers on production hardware.&lt;/li&gt;&#10;&lt;/ul&gt;&#10;</content:encoded></item><item><title>Sleep-Time Compute for AI Agents</title><link>https://jatinbansal.com/ai-engineering/sleep-time-compute/</link><guid isPermaLink="true">https://jatinbansal.com/ai-engineering/sleep-time-compute/</guid><pubDate>Wed, 20 May 2026 18:30:53 +0000</pubDate><dc:creator>Jatin Bansal</dc:creator><category>AI Engineering</category><description>Background consolidation and speculative work for stateful agents with explicit consistency controls.</description><content:encoded>&lt;p&gt;Reflection, compression, deduplication, embedding refresh, and speculative answering can improve an agent’s next response. They can also make the current response unacceptably slow. Sleep-time compute moves work that is useful but not immediately required onto a background path.&lt;/p&gt;&#10;&lt;p&gt;The name comes from &lt;a href="https://arxiv.org/abs/2504.13171"&gt;Sleep-time Compute&lt;/a&gt;, which separates compute used while serving a query from compute performed between queries. The distinction is architectural, not biological: a sleep-time worker may run after an idle period, every few turns, on a schedule, or from a queue.&lt;/p&gt;&#10;&lt;p&gt;The key test is whether the current answer depends on the result. Required work remains on the hot path. Maintenance and speculative work can move off it when they have a clear consistency model and enough future reuse to justify their cost.&lt;/p&gt;&#10;&lt;h2 id="what-belongs-in-the-background"&gt;What belongs in the background&lt;/h2&gt;&#10;&lt;p&gt;Good candidates share two properties: they are expensive relative to orchestration, and their output can benefit several later turns.&lt;/p&gt;&#10;&lt;ul&gt;&#10;&lt;li&gt;Consolidating episodes into summaries or beliefs&lt;/li&gt;&#10;&lt;li&gt;Merging duplicate memories and marking stale records&lt;/li&gt;&#10;&lt;li&gt;Rebuilding embeddings after a model change&lt;/li&gt;&#10;&lt;li&gt;Refreshing indexes or materialized views&lt;/li&gt;&#10;&lt;li&gt;Evaluating likely next questions and caching expensive intermediate results&lt;/li&gt;&#10;&lt;/ul&gt;&#10;&lt;p&gt;Small writes, permission checks, and information needed to answer the active request do not belong in the queue. Deferring them may reduce displayed latency while making the answer incomplete or unsafe.&lt;/p&gt;&#10;&lt;p&gt;The database analogy is useful. Transaction processing serves current reads and writes; maintenance jobs compact logs, refresh views, and collect statistics. Those jobs improve later queries but need scheduling, resource limits, and isolation from foreground traffic.&lt;/p&gt;&#10;&lt;h2 id="the-letta-formulation"&gt;The Letta formulation&lt;/h2&gt;&#10;&lt;p&gt;The &lt;a href="https://www.letta.com/blog/sleep-time-compute"&gt;Letta write-up&lt;/a&gt; describes a foreground agent and a sleep-time agent sharing memory. The foreground path answers the user. The background agent receives a narrower tool set for memory maintenance and can use a cheaper model. This separation makes cost and authority explicit.&lt;/p&gt;&#10;&lt;p&gt;The associated &lt;a href="https://arxiv.org/abs/2504.13171"&gt;paper&lt;/a&gt; studies shifting reasoning from test time to periods before a query arrives. Its reported benefits depend on the workload and prediction quality; they should not be treated as universal multipliers. The transferable engineering pattern is the split budget: reserve interactive compute for latency-sensitive work and spend background compute where it can be amortized.&lt;/p&gt;&#10;&lt;p&gt;A single process can implement the same boundary, but separate workers are easier to rate-limit and observe. The foreground service appends events and signals activity. A worker leases a bounded batch, transforms it, and commits results with version checks.&lt;/p&gt;&#10;&lt;h2 id="a-minimal-consolidation-worker"&gt;A minimal consolidation worker&lt;/h2&gt;&#10;&lt;p&gt;This example uses a queue and an idle-or-cadence trigger. The consolidation function is deliberately separate from scheduling so it can be tested deterministically.&lt;/p&gt;&#10;&lt;figure class="nb-code"&gt;&#10; &lt;header class="nb-code-chrome"&gt;&#10; &lt;span class="nb-code-label"&gt;python&lt;/span&gt;&#10; &#10; &lt;/header&gt;&#10; &lt;div class="nb-code-body"&gt;&lt;div class="highlight"&gt;&lt;div class="chroma"&gt;&#10;&lt;table class="lntable"&gt;&lt;tr&gt;&lt;td class="lntd"&gt;&#10;&lt;pre tabindex="0" class="chroma"&gt;&lt;code&gt;&lt;span class="lnt"&gt; 1&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 2&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 3&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 4&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 5&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 6&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 7&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 8&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 9&#10;&lt;/span&gt;&lt;span class="lnt"&gt;10&#10;&lt;/span&gt;&lt;span class="lnt"&gt;11&#10;&lt;/span&gt;&lt;span class="lnt"&gt;12&#10;&lt;/span&gt;&lt;span class="lnt"&gt;13&#10;&lt;/span&gt;&lt;span class="lnt"&gt;14&#10;&lt;/span&gt;&lt;span class="lnt"&gt;15&#10;&lt;/span&gt;&lt;span class="lnt"&gt;16&#10;&lt;/span&gt;&lt;span class="lnt"&gt;17&#10;&lt;/span&gt;&lt;span class="lnt"&gt;18&#10;&lt;/span&gt;&lt;span class="lnt"&gt;19&#10;&lt;/span&gt;&lt;span class="lnt"&gt;20&#10;&lt;/span&gt;&lt;span class="lnt"&gt;21&#10;&lt;/span&gt;&lt;span class="lnt"&gt;22&#10;&lt;/span&gt;&lt;span class="lnt"&gt;23&#10;&lt;/span&gt;&lt;span class="lnt"&gt;24&#10;&lt;/span&gt;&lt;span class="lnt"&gt;25&#10;&lt;/span&gt;&lt;span class="lnt"&gt;26&#10;&lt;/span&gt;&lt;span class="lnt"&gt;27&#10;&lt;/span&gt;&lt;span class="lnt"&gt;28&#10;&lt;/span&gt;&lt;span class="lnt"&gt;29&#10;&lt;/span&gt;&lt;span class="lnt"&gt;30&#10;&lt;/span&gt;&lt;span class="lnt"&gt;31&#10;&lt;/span&gt;&lt;span class="lnt"&gt;32&#10;&lt;/span&gt;&lt;span class="lnt"&gt;33&#10;&lt;/span&gt;&lt;span class="lnt"&gt;34&#10;&lt;/span&gt;&lt;span class="lnt"&gt;35&#10;&lt;/span&gt;&lt;span class="lnt"&gt;36&#10;&lt;/span&gt;&lt;span class="lnt"&gt;37&#10;&lt;/span&gt;&lt;span class="lnt"&gt;38&#10;&lt;/span&gt;&lt;span class="lnt"&gt;39&#10;&lt;/span&gt;&lt;span class="lnt"&gt;40&#10;&lt;/span&gt;&lt;span class="lnt"&gt;41&#10;&lt;/span&gt;&lt;span class="lnt"&gt;42&#10;&lt;/span&gt;&lt;span class="lnt"&gt;43&#10;&lt;/span&gt;&lt;span class="lnt"&gt;44&#10;&lt;/span&gt;&lt;span class="lnt"&gt;45&#10;&lt;/span&gt;&lt;span class="lnt"&gt;46&#10;&lt;/span&gt;&lt;span class="lnt"&gt;47&#10;&lt;/span&gt;&lt;span class="lnt"&gt;48&#10;&lt;/span&gt;&lt;span class="lnt"&gt;49&#10;&lt;/span&gt;&lt;span class="lnt"&gt;50&#10;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/td&gt;&#10;&lt;td class="lntd"&gt;&#10;&lt;pre tabindex="0" class="chroma"&gt;&lt;code class="language-python" data-lang="python"&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="kn"&gt;from&lt;/span&gt; &lt;span class="nn"&gt;dataclasses&lt;/span&gt; &lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;dataclass&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="kn"&gt;from&lt;/span&gt; &lt;span class="nn"&gt;queue&lt;/span&gt; &lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;Queue&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;Empty&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="kn"&gt;from&lt;/span&gt; &lt;span class="nn"&gt;threading&lt;/span&gt; &lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;Event&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;Thread&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="kn"&gt;from&lt;/span&gt; &lt;span class="nn"&gt;time&lt;/span&gt; &lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;monotonic&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="nd"&gt;@dataclass&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="k"&gt;class&lt;/span&gt; &lt;span class="nc"&gt;Episode&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="nb"&gt;id&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;str&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;text&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;str&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;version&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;int&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="n"&gt;pending&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;Queue&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;Episode&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;Queue&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="n"&gt;stop&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;Event&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;consolidate&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;batch&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;list&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;Episode&lt;/span&gt;&lt;span class="p"&gt;])&lt;/span&gt; &lt;span class="o"&gt;-&amp;gt;&lt;/span&gt; &lt;span class="nb"&gt;dict&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="s2"&gt;&amp;#34;&amp;#34;&amp;#34;Call a model and return structured additions plus source IDs.&amp;#34;&amp;#34;&amp;#34;&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="s2"&gt;&amp;#34;summary&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;summarize&lt;/span&gt;&lt;span class="p"&gt;([&lt;/span&gt;&lt;span class="n"&gt;e&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;text&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;e&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;batch&lt;/span&gt;&lt;span class="p"&gt;]),&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="s2"&gt;&amp;#34;source_ids&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;e&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;id&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;e&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;batch&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="s2"&gt;&amp;#34;source_versions&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="n"&gt;e&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;id&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;e&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;version&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;e&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;batch&lt;/span&gt;&lt;span class="p"&gt;},&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="p"&gt;}&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;commit&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;result&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;dict&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;-&amp;gt;&lt;/span&gt; &lt;span class="nb"&gt;bool&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="s2"&gt;&amp;#34;&amp;#34;&amp;#34;Atomically fail if a source changed while the worker ran.&amp;#34;&amp;#34;&amp;#34;&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="k"&gt;with&lt;/span&gt; &lt;span class="n"&gt;memory_transaction&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt; &lt;span class="k"&gt;as&lt;/span&gt; &lt;span class="n"&gt;tx&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="ow"&gt;not&lt;/span&gt; &lt;span class="n"&gt;tx&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;versions_match&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;result&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;&amp;#34;source_versions&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;]):&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="kc"&gt;False&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;tx&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;insert_summary&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;result&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;&amp;#34;summary&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt; &lt;span class="n"&gt;result&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;&amp;#34;source_ids&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;])&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;tx&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;mark_consolidated&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;result&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;&amp;#34;source_ids&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;])&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="kc"&gt;True&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;worker&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;idle_seconds&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;60&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;max_batch&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;50&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;max_wait&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;900&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;batch&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;list&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;Episode&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;[]&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;last_activity&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;monotonic&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="k"&gt;while&lt;/span&gt; &lt;span class="ow"&gt;not&lt;/span&gt; &lt;span class="n"&gt;stop&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;is_set&lt;/span&gt;&lt;span class="p"&gt;():&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="k"&gt;try&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;batch&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;append&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;pending&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;timeout&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;last_activity&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;monotonic&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="k"&gt;except&lt;/span&gt; &lt;span class="n"&gt;Empty&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="k"&gt;pass&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;idle&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;monotonic&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt; &lt;span class="n"&gt;last_activity&lt;/span&gt; &lt;span class="o"&gt;&amp;gt;=&lt;/span&gt; &lt;span class="n"&gt;idle_seconds&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;overdue&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;batch&lt;/span&gt; &lt;span class="ow"&gt;and&lt;/span&gt; &lt;span class="n"&gt;monotonic&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt; &lt;span class="n"&gt;last_activity&lt;/span&gt; &lt;span class="o"&gt;&amp;gt;=&lt;/span&gt; &lt;span class="n"&gt;max_wait&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;batch&lt;/span&gt; &lt;span class="ow"&gt;and&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;idle&lt;/span&gt; &lt;span class="ow"&gt;or&lt;/span&gt; &lt;span class="n"&gt;overdue&lt;/span&gt; &lt;span class="ow"&gt;or&lt;/span&gt; &lt;span class="nb"&gt;len&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;batch&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;&amp;gt;=&lt;/span&gt; &lt;span class="n"&gt;max_batch&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;work&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;batch&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;batch&lt;/span&gt;&lt;span class="p"&gt;[:&lt;/span&gt;&lt;span class="n"&gt;max_batch&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt; &lt;span class="n"&gt;batch&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;max_batch&lt;/span&gt;&lt;span class="p"&gt;:]&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="ow"&gt;not&lt;/span&gt; &lt;span class="n"&gt;commit&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;consolidate&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;work&lt;/span&gt;&lt;span class="p"&gt;)):&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;episode&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;work&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;pending&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;put&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;episode&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="n"&gt;Thread&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;target&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;worker&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;daemon&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="kc"&gt;True&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;start&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/td&gt;&lt;/tr&gt;&lt;/table&gt;&#10;&lt;/div&gt;&#10;&lt;/div&gt;&lt;/div&gt;&#10;&lt;/figure&gt;&#10;&lt;p&gt;In a real system, &lt;code&gt;summarize&lt;/code&gt; should produce validated structured output rather than unconstrained prose. The transaction should create provenance edges from the summary to its episodes. Failed batches need bounded retries and a dead-letter queue; blindly re-enqueuing a permanently bad record creates an infinite loop.&lt;/p&gt;&#10;&lt;p&gt;Do not delete raw episodes as part of consolidation unless retention policy requires it. A summary supports broad recall, while the source episodes support exact questions and audits. Marking episodes as consolidated is enough to keep them out of later batches.&lt;/p&gt;&#10;&lt;h2 id="scheduling-and-resource-isolation"&gt;Scheduling and resource isolation&lt;/h2&gt;&#10;&lt;p&gt;An idle threshold alone fails under steady traffic: a user active every few seconds may prevent maintenance for hours. A turn-count or maximum-delay trigger provides a fallback. Conversely, a fixed short interval can run expensive jobs when there is no useful work.&lt;/p&gt;&#10;&lt;p&gt;Use several controls together:&lt;/p&gt;&#10;&lt;ul&gt;&#10;&lt;li&gt;minimum batch size or age, so tiny updates do not dominate orchestration;&lt;/li&gt;&#10;&lt;li&gt;maximum batch size, which bounds model context and transaction time;&lt;/li&gt;&#10;&lt;li&gt;rate and cost limits per hour;&lt;/li&gt;&#10;&lt;li&gt;lower worker priority or a separate compute pool;&lt;/li&gt;&#10;&lt;li&gt;back-pressure when the foreground service is under load.&lt;/li&gt;&#10;&lt;/ul&gt;&#10;&lt;p&gt;The worker’s model can often be smaller than the foreground model because the operation is narrow and schema-constrained. Verify this with task-specific evaluations. A cheap model that merges contradictory facts incorrectly is not a saving.&lt;/p&gt;&#10;&lt;h2 id="speculative-pre-computation"&gt;Speculative pre-computation&lt;/h2&gt;&#10;&lt;p&gt;Consolidation improves memory. A second use of sleep-time compute predicts likely next requests and pre-computes expensive work. A coding agent that has just diagnosed a failing test might prepare likely follow-ups such as locating related tests or drafting a focused patch.&lt;/p&gt;&#10;&lt;p&gt;Exact-answer caches are risky because even a small context change can invalidate them. Key cached work by the query representation and a version of every relevant dependency. On lookup, require both a strong query match and matching context versions. A miss should fall through to normal reasoning.&lt;/p&gt;&#10;&lt;p&gt;Prefer caching reusable intermediates over polished responses. Search results, parsed repository maps, or validated calculations are easier to reuse safely than a complete answer whose wording and assumptions may no longer fit.&lt;/p&gt;&#10;&lt;p&gt;Measure useful-hit rate rather than raw hit rate. A cache that frequently returns plausible but stale results reduces latency at the expense of correctness. Sample hits against fresh foreground computation and evict entries that do not hold up.&lt;/p&gt;&#10;&lt;h2 id="consistency-on-shared-state"&gt;Consistency on shared state&lt;/h2&gt;&#10;&lt;p&gt;The worker and foreground agent may read and write the same store. Without coordination, the worker can consolidate a record while the foreground path edits it, or publish a summary based on a mixture of versions.&lt;/p&gt;&#10;&lt;p&gt;Useful patterns include:&lt;/p&gt;&#10;&lt;ol&gt;&#10;&lt;li&gt;Single writer: the foreground appends immutable events; only the worker produces derived memory.&lt;/li&gt;&#10;&lt;li&gt;Optimistic concurrency: each record carries a version, and the worker retries when versions change before commit.&lt;/li&gt;&#10;&lt;li&gt;Snapshot processing: the worker reads from a database snapshot and publishes an output labelled with that snapshot’s revision.&lt;/li&gt;&#10;&lt;/ol&gt;&#10;&lt;p&gt;Single-writer designs are easiest to reason about but may not fit interactive memory edits. Optimistic concurrency is flexible if transforms are idempotent. Snapshot processing is useful for large periodic rebuilds.&lt;/p&gt;&#10;&lt;p&gt;Every derived record should identify its source IDs and transform version. If a source is corrected or deleted, a reverse provenance index can find summaries that require recomputation.&lt;/p&gt;&#10;&lt;h2 id="failure-modes-to-plan-for"&gt;Failure modes to plan for&lt;/h2&gt;&#10;&lt;p&gt;Over-consolidation loses detail. Keep raw evidence and test exact historical questions, not just broad recall.&lt;/p&gt;&#10;&lt;p&gt;Background cost exceeds foreground savings. Track worker tokens, model cost, CPU time, useful outputs, and later reuse. Rate-limit speculative work first.&lt;/p&gt;&#10;&lt;p&gt;A stale cache serves confidently. Version the underlying context, apply TTL or LRU limits, and fall back on any mismatch.&lt;/p&gt;&#10;&lt;p&gt;Maintenance failures become invisible. Give the worker its own traces, queue-depth metrics, retry counts, and alerts. A week-old embedding index can degrade retrieval without causing an obvious foreground exception.&lt;/p&gt;&#10;&lt;p&gt;The worker has too much authority. Restrict it to memory and cache operations. It should not edit application code, contact users, spend money, or invoke destructive tools merely because it runs outside the interactive session.&lt;/p&gt;&#10;&lt;p&gt;No idle period ever occurs. Trigger on elapsed time or turn count as well as inactivity.&lt;/p&gt;&#10;&lt;p&gt;Sleep-time compute is worthwhile when state persists, future queries reuse the work, and interactive latency matters. It is usually overhead for one-shot, open-domain requests with no shared context. Start with one measurable maintenance job, compare foreground latency and retrieval quality before and after, and expand only when the background output is demonstrably reused.&lt;/p&gt;&#10;&lt;h2 id="deployment-checklist"&gt;Deployment checklist&lt;/h2&gt;&#10;&lt;p&gt;Start with one queue and one transform. Give every job an idempotency key derived from its operation, source IDs, and transform version. If a worker crashes after committing but before acknowledging the message, a retry should observe the existing result instead of creating a duplicate summary.&lt;/p&gt;&#10;&lt;p&gt;Define queue ownership and retention. The foreground service should know whether enqueue failure blocks the request, degrades gracefully, or is retried through an outbox. Memory consolidation is normally best-effort, so an outbox written in the same transaction as the episode provides durability without adding a remote queue call to the response path.&lt;/p&gt;&#10;&lt;p&gt;Set service-level objectives for maximum queue age, successful pass rate, and cost per consolidated episode. Queue depth alone is ambiguous: a large queue can be healthy during a known import, while one old poison message can violate freshness with a nearly empty queue. Alert on age and repeated failure reason as well as depth.&lt;/p&gt;&#10;&lt;p&gt;Roll out workers with read-only or shadow mode first. Let them produce proposed summaries and compare those with existing retrieval results without publishing. Sample for contradiction, lost detail, and privacy leakage. When writes are enabled, begin with a narrow namespace and retain a kill switch that stops publication without preventing foreground memory reads.&lt;/p&gt;&#10;&lt;p&gt;Finally, rehearse rebuilding derived state. If summaries and indexes are truly derived, the team should be able to delete a test copy and regenerate it from source events with the same transform versions. A system that cannot rebuild its background products has made them primary data by accident.&lt;/p&gt;&#10;&lt;h2 id="further-reading"&gt;Further reading&lt;/h2&gt;&#10;&lt;ul&gt;&#10;&lt;li&gt;&lt;a href="https://arxiv.org/abs/2504.13171"&gt;Sleep-time Compute: Beyond Inference Scaling at Test-time&lt;/a&gt;&lt;/li&gt;&#10;&lt;li&gt;&lt;a href="https://www.letta.com/blog/sleep-time-compute"&gt;Letta: Sleep-time Compute&lt;/a&gt;&lt;/li&gt;&#10;&lt;li&gt;&lt;a href="https://www.letta.com/blog/continual-learning"&gt;Continual Learning in Token Space&lt;/a&gt;&lt;/li&gt;&#10;&lt;li&gt;&lt;a href="https://jatinbansal.com/ai-engineering/context-compression/"&gt;Summarization and Context Compression&lt;/a&gt;&lt;/li&gt;&#10;&lt;li&gt;&lt;a href="https://jatinbansal.com/ai-engineering/memory-reflection/"&gt;Reflection: From Experiences to Beliefs&lt;/a&gt;&lt;/li&gt;&#10;&lt;li&gt;&lt;a href="https://jatinbansal.com/ai-engineering/memory-write-policies/"&gt;Memory Write Policies&lt;/a&gt;&lt;/li&gt;&#10;&lt;/ul&gt;&#10;</content:encoded></item><item><title>Human-in-the-Loop Feedback Loops for LLM Systems</title><link>https://jatinbansal.com/ai-engineering/human-feedback-loops/</link><guid isPermaLink="true">https://jatinbansal.com/ai-engineering/human-feedback-loops/</guid><pubDate>Tue, 19 May 2026 14:08:00 +0000</pubDate><dc:creator>Jatin Bansal</dc:creator><category>AI Engineering</category><description>How to capture, sample, label, and route production feedback into evaluation and product changes.</description><content:encoded>&lt;p&gt;A feedback widget has little value if its records are never linked to traces, reviewed, and routed into evaluation or product changes. A useful loop preserves the request context, selects a review sample deliberately, applies a versioned rubric, and records what changed because of the label.&lt;/p&gt;&#10;&lt;h2 id="capture-signals-with-trace-context"&gt;Capture signals with trace context&lt;/h2&gt;&#10;&lt;p&gt;The signals factor into three categories. A production loop that doesn&amp;rsquo;t capture all three is at least one axis blind.&lt;/p&gt;&#10;&lt;p&gt;Explicit feedback is what the user says directly. It may be a binary verdict, a structured category, or a free-text edit. Thumbs up and down reveal intent but rarely explain the defect. A follow-up category supplies more precise labels, although only about 10–30% of users complete it. A free-text edit provides a directional correction in the original context, making it the most useful input for annotation and preference-data pipelines. Capture each form with the same trace ID.&lt;/p&gt;&#10;&lt;p&gt;&lt;strong&gt;Implicit feedback.&lt;/strong&gt; What the user does. Six common signals: &lt;em&gt;regenerate&lt;/em&gt; (the user asked for another try; high precision negative), &lt;em&gt;follow-up clarification&lt;/em&gt; (the user immediately pivoted to &amp;ldquo;no, I meant&amp;hellip;&amp;rdquo;; high precision negative), &lt;em&gt;abandonment&lt;/em&gt; (the user closed the session mid-stream; ambiguous but trend-significant), &lt;em&gt;copy-without-edit&lt;/em&gt; (the user copied the output to clipboard verbatim; high precision positive), &lt;em&gt;copy-with-edit&lt;/em&gt; (the user pasted, edited externally, returned; positive-with-correction; great supervised signal if you can capture the diff), and &lt;em&gt;dwell time&lt;/em&gt; (the user spent significantly more or less time on a response than the per-category median; ambiguous but useful in conjunction). The 2025 &lt;a href="https://arxiv.org/html/2507.23158v2"&gt;analysis of human–LLM dialogues&lt;/a&gt; found implicit signals are noisier than explicit feedback per row but vastly higher volume; the right architecture exposes them as ordered ranks over the trace store, not as standalone labels.&lt;/p&gt;&#10;&lt;p&gt;&lt;strong&gt;Reviewer annotations.&lt;/strong&gt; What internal humans say. The structural signal: a domain expert reviewing a sampled trace, scoring it against the same rubric the &lt;a href="https://jatinbansal.com/ai-engineering/llm-as-judge/"&gt;LLM judge&lt;/a&gt; uses, and either confirming the judge&amp;rsquo;s verdict (calibration data) or overriding it (new failure-mode candidate). This is the source of truth the loop calibrates against. Volume is constrained by human bandwidth; a reviewer can sustain 50–200 labels per day depending on rubric complexity; so the &lt;em&gt;sampling policy&lt;/em&gt; is the load-bearing design decision (next section).&lt;/p&gt;&#10;&lt;p&gt;A subtler signal worth mentioning: &lt;strong&gt;public-channel feedback&lt;/strong&gt;; support tickets, internal Slack threads about the assistant misbehaving, customer-success notes. This is structurally similar to explicit feedback but arrives without a trace ID, and the work of &lt;em&gt;re-linking&lt;/em&gt; a complaint back to a specific turn is non-trivial. Tools like &lt;a href="https://langfuse.com/docs/observability/features/user-feedback"&gt;Langfuse&amp;rsquo;s user-feedback API&lt;/a&gt; and &lt;a href="https://docs.smith.langchain.com/evaluation/how_to_guides/human_feedback/attach_user_feedback"&gt;LangSmith&amp;rsquo;s feedback API&lt;/a&gt; include explicit &lt;code&gt;traceId&lt;/code&gt; parameters so the application can attach feedback at the moment of capture, but support tickets and Slack threads usually don&amp;rsquo;t carry that ID, and the re-linking pass is what makes them usable. Worth doing for incidents; not worth doing at steady state.&lt;/p&gt;&#10;&lt;h2 id="sample-for-information-not-volume"&gt;Sample for information, not volume&lt;/h2&gt;&#10;&lt;p&gt;A reviewer with 60 minutes a day and a 12,000-trace-per-day stream can label maybe 80 traces. The sampler is the difference between 80 of the most informative traces and 80 of the loudest. Five policies, picked in combination by what the loop is trying to learn.&lt;/p&gt;&#10;&lt;ul&gt;&#10;&lt;li&gt;Random sampling: A uniform random subset of all production traces. The floor below which the queue should never starve, because every other policy is biased toward what the system &lt;em&gt;already knows is broken&lt;/em&gt;; only random sampling discovers &lt;em&gt;unknown unknowns&lt;/em&gt;. &lt;a href="https://eugeneyan.com/writing/eval-process/"&gt;Eugene Yan&amp;rsquo;s recommendation&lt;/a&gt; to aim for &amp;ldquo;a 50:50 split of passes and fails that spans the distribution of inputs&amp;rdquo; relies on random sampling as the recall mechanism. Allocate 10–30% of the annotation budget to random, even when the system is on fire and the targeted pulls are screaming.&lt;/li&gt;&#10;&lt;li&gt;Stratified sampling: Random within each &lt;a href="https://jatinbansal.com/ai-engineering/eval-driven-development/"&gt;error-analysis category&lt;/a&gt; so the long-tail categories aren&amp;rsquo;t drowned out by the dominant ones. The dominant category in a customer-support assistant might be &amp;ldquo;order status&amp;rdquo; at 40% of traffic; stratifying ensures the long-tail &amp;ldquo;billing dispute&amp;rdquo; or &amp;ldquo;refund processing&amp;rdquo; categories; which are usually where the failures concentrate; get proportional annotation time.&lt;/li&gt;&#10;&lt;li&gt;Negative-feedback sampling: Every trace with an explicit thumbs-down or a high-precision implicit negative (regenerate, abandon, immediate-pivot) goes into the queue with a high prior. The conversion rate from &amp;ldquo;user said it was bad&amp;rdquo; to &amp;ldquo;actually bad&amp;rdquo; is well under 100% but well above the random base rate; this is the cheapest place to find real failures. Cap the rate so a single noisy user can&amp;rsquo;t dominate the queue.&lt;/li&gt;&#10;&lt;li&gt;Uncertainty-based active learning: The &lt;a href="https://jatinbansal.com/ai-engineering/llm-as-judge/"&gt;LLM judge&amp;rsquo;s confidence&lt;/a&gt; on a trace; or the score&amp;rsquo;s distance from a threshold; is a usable uncertainty signal. Traces where the judge&amp;rsquo;s score sits at the rubric boundary (e.g. 2 on a 1–3 scale) are the rows where the model labels are most ambiguous and the human label most informative. The &lt;a href="https://arxiv.org/html/2502.11767v1"&gt;LLM-based active learning literature&lt;/a&gt; reports 50–80% reductions in annotation budget for the same downstream quality versus random sampling on this approach; the catch is that &amp;ldquo;informative for the judge&amp;rdquo; is not the same as &amp;ldquo;informative for the application,&amp;rdquo; and uncertainty sampling alone over-indexes on edge cases.&lt;/li&gt;&#10;&lt;li&gt;Targeted sampling on drift signals: When the &lt;a href="https://jatinbansal.com/ai-engineering/drift-detection/"&gt;drift detector&lt;/a&gt; flags a category or cluster as drifted, pull the next N traces from that slice into the queue ahead of the steady-state policies. This is the equivalent of an SRE on-call following the alert to the cluster that&amp;rsquo;s degrading; same shape, applied to the annotation queue.&lt;/li&gt;&#10;&lt;/ul&gt;&#10;&lt;p&gt;The RLTHF paper (&lt;a href="https://arxiv.org/abs/2502.13417"&gt;Wang et al., 2025&lt;/a&gt;) reports that combining LLM-based initial labelling with selective human review on the uncertain rows achieves &lt;em&gt;full-human-annotation&lt;/em&gt; alignment quality with only 6–7% of the human annotation effort on the HH-RLHF and TL;DR benchmarks. The result is specific to RLHF preference-pair labelling, but the principle generalises: an LLM-first pass that pre-labels the queue and routes only the uncertain rows to humans is the production pattern that scales when annotation bandwidth is the bottleneck.&lt;/p&gt;&#10;&lt;p&gt;A practical mix that holds up: 25% random, 25% stratified by category, 30% negative-feedback (capped), 10% uncertainty-based, 10% targeted on the day&amp;rsquo;s drift alerts. Re-tune the weights quarterly based on which slice&amp;rsquo;s annotations turned into the most caught failures.&lt;/p&gt;&#10;&lt;h2 id="keep-labels-consistent"&gt;Keep labels consistent&lt;/h2&gt;&#10;&lt;p&gt;The label is the metric. If the labels are inconsistent across annotators or across days, the downstream eval suite, judge calibration, and fine-tuning corpus all inherit the inconsistency. Four practices that production teams converge on.&lt;/p&gt;&#10;&lt;p&gt;&lt;strong&gt;Codebook discipline.&lt;/strong&gt; The rubric is a versioned artifact; not a wiki paragraph that drifts with each onboarded annotator. Each label category has a written definition, a positive example, a negative example, and an edge case that&amp;rsquo;s explicitly &lt;em&gt;not&lt;/em&gt; in the category. Versions are stamped on every label batch so that comparisons across time know which rubric version produced the labels. When the rubric changes, the affected categories&amp;rsquo; historical labels are quarantined or re-labelled under the new codebook; not silently re-interpreted.&lt;/p&gt;&#10;&lt;p&gt;&lt;strong&gt;Benevolent-dictator workflow.&lt;/strong&gt; &lt;a href="https://hamel.dev/blog/posts/evals-faq/index.html"&gt;Hamel Husain&amp;rsquo;s recommendation&lt;/a&gt; for most teams: appoint a single domain expert as the final judge of label quality and the rubric owner. Distributed annotation efforts without a single point of authority tend to drift into Cohen&amp;rsquo;s-kappa-below-0.5 territory within a quarter. The benevolent dictator&amp;rsquo;s job is to spot-check labels, resolve disagreements, edit the codebook, and own the inter-annotator agreement metric. When multiple annotators are necessary; for volume or for legitimately ambiguous categories; periodic alignment sessions with kappa measurement on a doubly-labelled subset keep the process honest.&lt;/p&gt;&#10;&lt;p&gt;&lt;strong&gt;Inter-annotator agreement as a process metric.&lt;/strong&gt; Double-label 5–10% of the queue (the same trace labelled independently by two reviewers), compute &lt;a href="https://mbrenndoerfer.com/writing/inter-annotator-agreement-kappa-alpha-reliability"&gt;Cohen&amp;rsquo;s kappa&lt;/a&gt; for two annotators or &lt;a href="https://search.r-project.org/CRAN/refmans/krippendorffsalpha/html/00Index.html"&gt;Krippendorff&amp;rsquo;s alpha&lt;/a&gt; for more, and track it as a dashboard line. Kappa below 0.6 on a category means the rubric is unstable; split the category, write better examples, or accept that the category is genuinely subjective and weight it differently in downstream metrics. The categories with the highest disagreement are usually the ones whose evals are most volatile and whose judge is most unreliable; treating IAA as a debugging tool catches the problem upstream of the eval suite.&lt;/p&gt;&#10;&lt;p&gt;&lt;strong&gt;Annotation latency budget.&lt;/strong&gt; A label that lands 12 weeks after the trace is logged is too late to feed back into a meaningful product cycle. The target is single-digit days from &amp;ldquo;trace generated&amp;rdquo; to &amp;ldquo;label applied,&amp;rdquo; which constrains both the queue size and the policy mix. Queues that grow past their throughput will silently age into uselessness; cap the queue, drop the lowest-priority overflow, and ship the missing labels as a known limitation rather than letting the queue rot.&lt;/p&gt;&#10;&lt;h2 id="route-labels-into-product-changes"&gt;Route labels into product changes&lt;/h2&gt;&#10;&lt;p&gt;A labelled trace is half the work. The other half is the routing decision: where does the label go, and who acts on it?&lt;/p&gt;&#10;&lt;ul&gt;&#10;&lt;li&gt;Into the eval suite: A confirmed failure that recurs in the wild is a new row on the &lt;a href="https://jatinbansal.com/ai-engineering/eval-driven-development/"&gt;golden set&lt;/a&gt;. The provenance; trace ID, date, annotator; gets stamped on the row, and the row gates merges until the underlying behaviour is fixed. This is the highest-leverage routing: it makes the failure expensive to re-introduce.&lt;/li&gt;&#10;&lt;li&gt;Into the judge calibration set: Every labelled trace is also a calibration data point for the &lt;a href="https://jatinbansal.com/ai-engineering/llm-as-judge/"&gt;LLM judge&lt;/a&gt;. Computing judge-human agreement on a rolling window catches judge drift before the eval suite numbers go bad. When agreement drops below a threshold (kappa &amp;lt; 0.6 against the human reference), the judge needs a re-tune or a model swap.&lt;/li&gt;&#10;&lt;li&gt;Into the prompt: Edits where the user rewrote the response are a direct prompt-improvement signal. The diff often clusters into a small number of recurring corrections; &amp;ldquo;use bullet points,&amp;rdquo; &amp;ldquo;cite the source,&amp;rdquo; &amp;ldquo;don&amp;rsquo;t apologise&amp;rdquo;; that are cheap to add to the system prompt as explicit rules.&lt;/li&gt;&#10;&lt;li&gt;Into the retriever: A negative trace where the right document was in the index but didn&amp;rsquo;t make it into the top-K is a &lt;a href="https://jatinbansal.com/ai-engineering/rag-evaluation/"&gt;retrieval failure&lt;/a&gt;, not a generation failure. The labelled trace becomes an example row for retriever evaluation, and over time the corpus of such rows is what drives changes to &lt;a href="https://jatinbansal.com/ai-engineering/chunking-strategies/"&gt;chunking&lt;/a&gt;, &lt;a href="https://jatinbansal.com/ai-engineering/reranking/"&gt;reranking&lt;/a&gt;, or &lt;a href="https://jatinbansal.com/ai-engineering/hybrid-search/"&gt;hybrid search weights&lt;/a&gt;.&lt;/li&gt;&#10;&lt;li&gt;Into the model: Confirmed-failure-confirmed-fix pairs are the raw material for &lt;a href="https://jatinbansal.com/ai-engineering/dpo-and-alignment/"&gt;preference-pair fine-tuning via DPO and its variants&lt;/a&gt;. Most application teams don&amp;rsquo;t fine-tune anything in 2026, but for teams that do, this is the cleanest source of training data; preference pairs grounded in real production failures, validated by a domain expert against a versioned rubric.&lt;/li&gt;&#10;&lt;/ul&gt;&#10;&lt;p&gt;The loop&amp;rsquo;s &lt;em&gt;cycle time&lt;/em&gt;; from observation to a structural fix that prevents the failure recurring; is the metric the team should be tracking, not the raw label volume. &lt;a href="https://eugeneyan.com/writing/eval-process/"&gt;Eugene Yan&amp;rsquo;s framing&lt;/a&gt; is right: whoever turns the data flywheel faster wins. A team that labels 5,000 traces a quarter and ships zero fixes is doing labelling-theatre; a team that labels 500 traces a quarter and ships 30 structural fixes is running the loop.&lt;/p&gt;&#10;&lt;h2 id="code-capturing-structured-user-feedback-against-a-langfuse-trace-in-python"&gt;Code: capturing structured user feedback against a Langfuse trace in Python&lt;/h2&gt;&#10;&lt;p&gt;The harness below captures explicit feedback (thumbs and structured reason) and implicit feedback (regenerate, dwell time, edit diff) against a &lt;a href="https://langfuse.com/"&gt;Langfuse&lt;/a&gt; trace ID, stamps each with the trace context, and exposes a simple FastAPI endpoint. Install: &lt;code&gt;pip install fastapi uvicorn langfuse pydantic&lt;/code&gt;.&lt;/p&gt;&#10;&lt;figure class="nb-code"&gt;&#10; &lt;header class="nb-code-chrome"&gt;&#10; &lt;span class="nb-code-label"&gt;python&lt;/span&gt;&#10; &#10; &lt;/header&gt;&#10; &lt;div class="nb-code-body"&gt;&lt;div class="highlight"&gt;&lt;div class="chroma"&gt;&#10;&lt;table class="lntable"&gt;&lt;tr&gt;&lt;td class="lntd"&gt;&#10;&lt;pre tabindex="0" class="chroma"&gt;&lt;code&gt;&lt;span class="lnt"&gt; 1&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 2&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 3&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 4&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 5&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 6&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 7&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 8&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 9&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 10&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 11&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 12&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 13&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 14&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 15&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 16&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 17&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 18&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 19&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 20&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 21&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 22&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 23&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 24&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 25&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 26&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 27&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 28&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 29&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 30&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 31&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 32&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 33&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 34&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 35&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 36&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 37&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 38&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 39&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 40&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 41&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 42&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 43&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 44&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 45&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 46&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 47&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 48&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 49&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 50&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 51&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 52&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 53&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 54&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 55&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 56&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 57&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 58&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 59&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 60&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 61&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 62&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 63&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 64&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 65&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 66&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 67&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 68&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 69&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 70&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 71&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 72&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 73&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 74&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 75&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 76&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 77&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 78&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 79&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 80&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 81&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 82&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 83&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 84&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 85&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 86&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 87&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 88&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 89&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 90&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 91&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 92&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 93&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 94&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 95&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 96&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 97&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 98&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 99&#10;&lt;/span&gt;&lt;span class="lnt"&gt;100&#10;&lt;/span&gt;&lt;span class="lnt"&gt;101&#10;&lt;/span&gt;&lt;span class="lnt"&gt;102&#10;&lt;/span&gt;&lt;span class="lnt"&gt;103&#10;&lt;/span&gt;&lt;span class="lnt"&gt;104&#10;&lt;/span&gt;&lt;span class="lnt"&gt;105&#10;&lt;/span&gt;&lt;span class="lnt"&gt;106&#10;&lt;/span&gt;&lt;span class="lnt"&gt;107&#10;&lt;/span&gt;&lt;span class="lnt"&gt;108&#10;&lt;/span&gt;&lt;span class="lnt"&gt;109&#10;&lt;/span&gt;&lt;span class="lnt"&gt;110&#10;&lt;/span&gt;&lt;span class="lnt"&gt;111&#10;&lt;/span&gt;&lt;span class="lnt"&gt;112&#10;&lt;/span&gt;&lt;span class="lnt"&gt;113&#10;&lt;/span&gt;&lt;span class="lnt"&gt;114&#10;&lt;/span&gt;&lt;span class="lnt"&gt;115&#10;&lt;/span&gt;&lt;span class="lnt"&gt;116&#10;&lt;/span&gt;&lt;span class="lnt"&gt;117&#10;&lt;/span&gt;&lt;span class="lnt"&gt;118&#10;&lt;/span&gt;&lt;span class="lnt"&gt;119&#10;&lt;/span&gt;&lt;span class="lnt"&gt;120&#10;&lt;/span&gt;&lt;span class="lnt"&gt;121&#10;&lt;/span&gt;&lt;span class="lnt"&gt;122&#10;&lt;/span&gt;&lt;span class="lnt"&gt;123&#10;&lt;/span&gt;&lt;span class="lnt"&gt;124&#10;&lt;/span&gt;&lt;span class="lnt"&gt;125&#10;&lt;/span&gt;&lt;span class="lnt"&gt;126&#10;&lt;/span&gt;&lt;span class="lnt"&gt;127&#10;&lt;/span&gt;&lt;span class="lnt"&gt;128&#10;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/td&gt;&#10;&lt;td class="lntd"&gt;&#10;&lt;pre tabindex="0" class="chroma"&gt;&lt;code class="language-python" data-lang="python"&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="c1"&gt;# pip install fastapi uvicorn langfuse pydantic&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="nn"&gt;os&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="kn"&gt;from&lt;/span&gt; &lt;span class="nn"&gt;datetime&lt;/span&gt; &lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;datetime&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;timezone&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="kn"&gt;from&lt;/span&gt; &lt;span class="nn"&gt;typing&lt;/span&gt; &lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;Literal&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;Optional&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="kn"&gt;from&lt;/span&gt; &lt;span class="nn"&gt;fastapi&lt;/span&gt; &lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;FastAPI&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;HTTPException&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="kn"&gt;from&lt;/span&gt; &lt;span class="nn"&gt;pydantic&lt;/span&gt; &lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;BaseModel&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;Field&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="kn"&gt;from&lt;/span&gt; &lt;span class="nn"&gt;langfuse&lt;/span&gt; &lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;get_client&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="n"&gt;app&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;FastAPI&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="n"&gt;langfuse&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;get_client&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="c1"&gt;# --- schemas ---&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="k"&gt;class&lt;/span&gt; &lt;span class="nc"&gt;ExplicitFeedback&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;BaseModel&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;trace_id&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;str&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;user_id&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;str&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;rating&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;Literal&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;&amp;#34;thumbs_up&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;&amp;#34;thumbs_down&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;reason_category&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;Optional&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;Literal&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="s2"&gt;&amp;#34;hallucination&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;&amp;#34;refused&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;&amp;#34;too_long&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;&amp;#34;too_short&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="s2"&gt;&amp;#34;missed_point&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;&amp;#34;formatting&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;&amp;#34;other&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="p"&gt;]&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="kc"&gt;None&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;free_text&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;Optional&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="nb"&gt;str&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;Field&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;default&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="kc"&gt;None&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;max_length&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;2000&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="k"&gt;class&lt;/span&gt; &lt;span class="nc"&gt;EditFeedback&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;BaseModel&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;trace_id&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;str&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;user_id&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;str&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;original_output&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;str&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;edited_output&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;str&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;edit_kind&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;Literal&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;&amp;#34;copy_and_edit&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;&amp;#34;in_place_edit&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="k"&gt;class&lt;/span&gt; &lt;span class="nc"&gt;ImplicitFeedback&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;BaseModel&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;trace_id&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;str&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;user_id&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;str&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;event&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;Literal&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="s2"&gt;&amp;#34;regenerate&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;&amp;#34;abandon_mid_stream&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;&amp;#34;copy_without_edit&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="s2"&gt;&amp;#34;follow_up_pivot&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;&amp;#34;long_dwell&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;&amp;#34;short_dwell&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="p"&gt;]&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;dwell_ms&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;Optional&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="nb"&gt;int&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="kc"&gt;None&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;follow_up_text&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;Optional&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="nb"&gt;str&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="kc"&gt;None&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="c1"&gt;# --- routes ---&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="nd"&gt;@app.post&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;&amp;#34;/feedback/explicit&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;post_explicit&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;fb&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;ExplicitFeedback&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="c1"&gt;# Score: -1 for thumbs_down, +1 for thumbs_up. Langfuse stores arbitrary&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="c1"&gt;# numeric scores against a trace_id; downstream queries filter by name.&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;langfuse&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;create_score&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;trace_id&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;fb&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;trace_id&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;name&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;#34;user_rating&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;value&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mf"&gt;1.0&lt;/span&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;fb&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;rating&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="s2"&gt;&amp;#34;thumbs_up&amp;#34;&lt;/span&gt; &lt;span class="k"&gt;else&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="mf"&gt;1.0&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;data_type&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;#34;NUMERIC&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;comment&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;fb&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;free_text&lt;/span&gt; &lt;span class="ow"&gt;or&lt;/span&gt; &lt;span class="s2"&gt;&amp;#34;&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;metadata&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="s2"&gt;&amp;#34;user_id&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;fb&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;user_id&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="s2"&gt;&amp;#34;reason_category&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;fb&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;reason_category&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="s2"&gt;&amp;#34;signal_type&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;&amp;#34;explicit&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="s2"&gt;&amp;#34;captured_at&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;datetime&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;now&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;timezone&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;utc&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;isoformat&lt;/span&gt;&lt;span class="p"&gt;(),&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="p"&gt;},&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="p"&gt;)&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="s2"&gt;&amp;#34;status&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;&amp;#34;ok&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="nd"&gt;@app.post&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;&amp;#34;/feedback/edit&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;post_edit&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;fb&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;EditFeedback&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="c1"&gt;# An edit is supervisor signal. Stamp the diff as a score-with-data so&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="c1"&gt;# the trace store keeps the original + corrected pair together.&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;langfuse&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;create_score&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;trace_id&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;fb&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;trace_id&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;name&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;#34;user_edit&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;value&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mf"&gt;1.0&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="c1"&gt;# presence of an edit; magnitude is in the metadata&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;data_type&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;#34;NUMERIC&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;comment&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;fb&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;edited_output&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;metadata&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="s2"&gt;&amp;#34;user_id&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;fb&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;user_id&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="s2"&gt;&amp;#34;original_output&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;fb&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;original_output&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="s2"&gt;&amp;#34;edit_kind&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;fb&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;edit_kind&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="s2"&gt;&amp;#34;signal_type&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;&amp;#34;explicit_correction&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="s2"&gt;&amp;#34;captured_at&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;datetime&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;now&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;timezone&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;utc&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;isoformat&lt;/span&gt;&lt;span class="p"&gt;(),&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="p"&gt;},&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="p"&gt;)&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="s2"&gt;&amp;#34;status&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;&amp;#34;ok&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="nd"&gt;@app.post&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;&amp;#34;/feedback/implicit&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;post_implicit&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;fb&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;ImplicitFeedback&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;fb&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;event&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="s2"&gt;&amp;#34;long_dwell&amp;#34;&lt;/span&gt; &lt;span class="ow"&gt;and&lt;/span&gt; &lt;span class="n"&gt;fb&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;dwell_ms&lt;/span&gt; &lt;span class="ow"&gt;is&lt;/span&gt; &lt;span class="kc"&gt;None&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="k"&gt;raise&lt;/span&gt; &lt;span class="n"&gt;HTTPException&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;400&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;&amp;#34;dwell_ms required for dwell events&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="c1"&gt;# Implicit signals carry a prior on satisfaction direction.&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;polarity&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="s2"&gt;&amp;#34;regenerate&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="mf"&gt;1.0&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="s2"&gt;&amp;#34;abandon_mid_stream&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="mf"&gt;1.0&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="s2"&gt;&amp;#34;follow_up_pivot&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="mf"&gt;1.0&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="s2"&gt;&amp;#34;short_dwell&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="mf"&gt;0.3&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="c1"&gt;# weaker negative&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="s2"&gt;&amp;#34;copy_without_edit&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mf"&gt;1.0&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="s2"&gt;&amp;#34;long_dwell&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mf"&gt;0.5&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="c1"&gt;# weaker positive&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="p"&gt;}[&lt;/span&gt;&lt;span class="n"&gt;fb&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;event&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;langfuse&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;create_score&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;trace_id&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;fb&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;trace_id&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;name&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="sa"&gt;f&lt;/span&gt;&lt;span class="s2"&gt;&amp;#34;implicit_&lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;fb&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;event&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s2"&gt;&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;value&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;polarity&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;data_type&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;#34;NUMERIC&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;metadata&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="s2"&gt;&amp;#34;user_id&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;fb&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;user_id&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="s2"&gt;&amp;#34;dwell_ms&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;fb&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;dwell_ms&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="s2"&gt;&amp;#34;follow_up_text&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;fb&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;follow_up_text&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="s2"&gt;&amp;#34;signal_type&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;&amp;#34;implicit&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="s2"&gt;&amp;#34;captured_at&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;datetime&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;now&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;timezone&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;utc&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;isoformat&lt;/span&gt;&lt;span class="p"&gt;(),&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="p"&gt;},&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="p"&gt;)&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="s2"&gt;&amp;#34;status&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;&amp;#34;ok&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="nd"&gt;@app.get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;&amp;#34;/&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;root&lt;/span&gt;&lt;span class="p"&gt;():&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="s2"&gt;&amp;#34;ok&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kc"&gt;True&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="vm"&gt;__name__&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="s2"&gt;&amp;#34;__main__&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="nn"&gt;uvicorn&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;uvicorn&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;run&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;app&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;host&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;#34;0.0.0.0&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;port&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;8080&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/td&gt;&lt;/tr&gt;&lt;/table&gt;&#10;&lt;/div&gt;&#10;&lt;/div&gt;&lt;/div&gt;&#10;&lt;/figure&gt;&#10;&lt;p&gt;&lt;em&gt;the trace ID is required&lt;/em&gt;, not optional; feedback without a trace ID is feedback that can&amp;rsquo;t be investigated, and the API should reject it. Explicit, edit, and implicit signals land under different score names* (&lt;code&gt;user_rating&lt;/code&gt;, &lt;code&gt;user_edit&lt;/code&gt;, &lt;code&gt;implicit_*&lt;/code&gt;) so downstream queries can filter cleanly; mixing them under one name will make the dashboards lie. The polarity weights for implicit signals are starting points*, not laws; recalibrate them quarterly against labelled traces (high-precision implicit signals should correlate strongly with confirmed-failure annotations; if they don&amp;rsquo;t, the weight is wrong). The metadata is structured*, not free-text; &lt;code&gt;reason_category&lt;/code&gt; and &lt;code&gt;signal_type&lt;/code&gt; are enums in the schema, which is what makes the trace store queryable a year from now. Free-text fields are useful for support escalation but can&amp;rsquo;t power dashboards.&lt;/p&gt;&#10;&lt;h2 id="further-reading"&gt;Further reading&lt;/h2&gt;&#10;&lt;ul&gt;&#10;&lt;li&gt;&lt;a href="https://hamel.dev/blog/posts/evals-faq/"&gt;Hamel Husain &amp;amp; Shreya Shankar; LLM Evals FAQ&lt;/a&gt;; the long-form Q&amp;amp;A that is the field&amp;rsquo;s reference for error analysis, annotation workflow, and turning production traces into evals. The &amp;ldquo;single benevolent dictator&amp;rdquo; recommendation and the cost-benefit framework for which failures warrant a built evaluator both live here. Read this in full if you read nothing else.&lt;/li&gt;&#10;&lt;li&gt;&lt;a href="https://hamel.dev/blog/posts/field-guide/"&gt;Hamel Husain; A Field Guide to Rapidly Improving AI Products&lt;/a&gt;; the practitioner&amp;rsquo;s tour of building a custom annotation surface, one-click feedback, and the loop from observation to fix. Pairs naturally with the FAQ as the implementation companion.&lt;/li&gt;&#10;&lt;li&gt;&lt;a href="https://eugeneyan.com/writing/eval-process/"&gt;Eugene Yan; An LLM-as-Judge Won&amp;rsquo;t Save The Product - Fixing Your Process Will&lt;/a&gt;; the case for process-over-tools, the 50:50 pass/fail sampling recommendation, and the data-flywheel framing that the cycle time is the metric. The clearest single piece on why teams that buy tools without changing process get nothing for the spend.&lt;/li&gt;&#10;&lt;li&gt;&lt;a href="https://arxiv.org/abs/2502.13417"&gt;Wang et al.; RLTHF: Targeted Human Feedback for LLM Alignment&lt;/a&gt;; the academic case for LLM-pre-labelling plus human correction on the uncertain rows, with the 6–7% annotation-budget reduction result on HH-RLHF and TL;DR. The right reference for sizing the active-learning fraction of the queue.&lt;/li&gt;&#10;&lt;li&gt;&lt;a href="https://arxiv.org/abs/2510.06674"&gt;Liu et al.; Agent-in-the-Loop: A Data Flywheel for Continuous Improvement in LLM-based Customer Support&lt;/a&gt;; the production case study from a real customer-support deployment with four annotation types integrated into live operations and the months-to-weeks compression on the retraining cycle. The closest thing to an industrial-scale reference architecture for the human-feedback loop.&lt;/li&gt;&#10;&lt;/ul&gt;&#10;</content:encoded></item><item><title>Drift Detection and Regression Testing for LLM Systems</title><link>https://jatinbansal.com/ai-engineering/drift-detection/</link><guid isPermaLink="true">https://jatinbansal.com/ai-engineering/drift-detection/</guid><pubDate>Tue, 19 May 2026 13:48:00 +0000</pubDate><dc:creator>Jatin Bansal</dc:creator><category>AI Engineering</category><description>How to detect distribution shifts and test model upgrades with paired evaluation, shadow runs, and canaries.</description><content:encoded>&lt;p&gt;An LLM application can deteriorate while its code and aggregate score remain unchanged. The user mix may shift toward a weak category, a provider may change a model, or the evaluator may move. Drift monitoring compares current production distributions with a trusted reference; regression testing compares system versions on matched inputs.&lt;/p&gt;&#10;&lt;h2 id="three-kinds-of-drift"&gt;Three kinds of drift&lt;/h2&gt;&#10;&lt;p&gt;The signals factor into three axes. Detecting any one without the others is half-blind.&lt;/p&gt;&#10;&lt;p&gt;&lt;strong&gt;Input drift.&lt;/strong&gt; What users are sending. The cheap shadow is &lt;em&gt;category mix&lt;/em&gt;: every user message is classified into an error-analysis category at ingest, the per-category count is logged, and a &lt;a href="https://en.wikipedia.org/wiki/Population_stability_index"&gt;Population Stability Index (PSI)&lt;/a&gt; between the current 24-hour window and the reference 7-day window is computed once an hour. PSI above 0.2 flags material drift in the literature, above 0.1 is worth a glance. The expensive shadow is &lt;em&gt;embedding centroid distance&lt;/em&gt;: the &lt;a href="https://jatinbansal.com/ai-engineering/text-embeddings/"&gt;text-embedding&lt;/a&gt; of every user message goes into the trace store, the current-window centroid is the mean of those embeddings, and the Euclidean (or cosine) distance to the reference-window centroid is the drift signal. &lt;a href="https://arize.com/resource/understanding-umap-and-hdbscan/"&gt;Arize Phoenix&amp;rsquo;s embedding-drift module&lt;/a&gt; is the canonical implementation; it clusters with HDBSCAN, projects with UMAP, and exposes the centroid distance as a per-cluster trend line. The dual signal; category PSI plus centroid distance; catches both the &amp;ldquo;users are asking new things&amp;rdquo; failure mode (PSI fires; centroid moves) and the &amp;ldquo;users are asking similar things differently&amp;rdquo; mode (PSI flat; centroid moves) that a single signal would miss.&lt;/p&gt;&#10;&lt;p&gt;&lt;strong&gt;Output drift.&lt;/strong&gt; What the model is returning. The shadow at this layer is per-output features that don&amp;rsquo;t require judging; output length distribution, refusal rate, tool-call frequency per turn, &lt;a href="https://jatinbansal.com/ai-engineering/structured-output/"&gt;structured-output&lt;/a&gt; validity rate, &lt;a href="https://jatinbansal.com/ai-engineering/prompt-caching/"&gt;cache hit rate&lt;/a&gt;. These are cheap to compute over every span, and the &lt;a href="https://en.wikipedia.org/wiki/Kolmogorov%E2%80%93Smirnov_test"&gt;Kolmogorov-Smirnov test&lt;/a&gt; on the per-window distribution against the reference distribution is the alerting signal. A 10-point drop in tool-call frequency is the kind of failure that doesn&amp;rsquo;t show in any quality metric until support escalates it, but shows immediately in the KS test on tool-use rate. Output drift is the easiest axis to instrument and the easiest to miss because none of these signals are quality measurements; they&amp;rsquo;re correlates of quality, and a system that monitors only the judged quality score will see the regression two days after the cheap output-feature signal would have fired.&lt;/p&gt;&#10;&lt;p&gt;&lt;strong&gt;Concept drift.&lt;/strong&gt; Whether the &lt;em&gt;relationship&lt;/em&gt; between input and output is what it used to be; the actual quality measurement. The shadow at this layer is the judged score on a fixed evaluation slice, run continuously over a rolling sample of production traces. Sample 1% of traces per category per hour, run the &lt;a href="https://jatinbansal.com/ai-engineering/llm-as-judge/"&gt;pinned judge&lt;/a&gt; against them, compute the per-category mean score and bootstrapped confidence interval, alert when the current-window CI lower bound falls below the reference-window CI upper bound. This is the most expensive signal; every alert costs you the judge-call price for the sampled traces, times the categories, times the windows; but it&amp;rsquo;s the only one that measures the thing you care about. The cheaper signals on the other two axes are correlates; this one is the target.&lt;/p&gt;&#10;&lt;p&gt;The discipline is to &lt;em&gt;gate the expensive signal on the cheap ones&lt;/em&gt;. The hourly judge-replay runs against full samples; the category-PSI and centroid-distance checks run continuously and, when they fire, &lt;em&gt;trigger&lt;/em&gt; an immediate judge re-score on the drifted slice. The combination; continuous cheap monitoring plus on-demand expensive judging; keeps the bill bounded while keeping the latency to alert under an hour for the drift modes that matter.&lt;/p&gt;&#10;&lt;h2 id="statistical-tests-and-embedding-shift"&gt;Statistical tests and embedding shift&lt;/h2&gt;&#10;&lt;p&gt;A handful of statistical tools cover the LLM drift surface; pick by data shape.&lt;/p&gt;&#10;&lt;ul&gt;&#10;&lt;li&gt;&lt;strong&gt;&lt;a href="https://en.wikipedia.org/wiki/Population_stability_index"&gt;Population Stability Index (PSI)&lt;/a&gt;&lt;/strong&gt; for categorical distributions; error-analysis category mix, tool-call distribution, &lt;a href="https://jatinbansal.com/ai-engineering/llm-observability/"&gt;finish-reason&lt;/a&gt; distribution. PSI is &lt;code&gt;Σ (current_pct - reference_pct) * ln(current_pct / reference_pct)&lt;/code&gt; over the bins. Material drift threshold ~0.2, alert threshold ~0.1, fine-grained inspection above 0.05. Cheap; runs on aggregates.&lt;/li&gt;&#10;&lt;li&gt;&lt;strong&gt;&lt;a href="https://en.wikipedia.org/wiki/Kolmogorov%E2%80%93Smirnov_test"&gt;Kolmogorov-Smirnov (KS) test&lt;/a&gt;&lt;/strong&gt; for continuous distributions; output length, latency, per-span cost, judge score. KS reports the max absolute difference between the two empirical CDFs and a p-value; the p-value is the alert signal but the effect size (the D statistic) is what matters at scale. Cheap; runs on samples.&lt;/li&gt;&#10;&lt;li&gt;&lt;strong&gt;&lt;a href="https://en.wikipedia.org/wiki/Kullback%E2%80%93Leibler_divergence"&gt;Kullback-Leibler (KL) divergence&lt;/a&gt;&lt;/strong&gt; for probability distributions where you have explicit probabilities; token-level distributions if you have logprobs, soft-label outputs from a classifier head. KL is &lt;code&gt;Σ p(i) * log(p(i) / q(i))&lt;/code&gt;; it&amp;rsquo;s asymmetric, and the symmetric variant (Jensen-Shannon) is the more useful drift metric in practice. Cheaper than KS; needs probabilities.&lt;/li&gt;&#10;&lt;li&gt;Embedding centroid distance: Euclidean or cosine distance between current-window and reference-window mean embeddings. The bare metric is the cheap version; cluster-aware variants (compute per-cluster centroids after HDBSCAN, alert when any cluster&amp;rsquo;s centroid moves or a new cluster appears with mass above a threshold) are the production version. &lt;a href="https://arize.com/docs/ax/machine-learning/computer-vision/how-to-cv/embedding-drift"&gt;Arize&amp;rsquo;s documented implementation&lt;/a&gt; is the cleanest reference.&lt;/li&gt;&#10;&lt;li&gt;&lt;strong&gt;Paired-bootstrap confidence intervals&lt;/strong&gt; for regression testing; the right tool when you&amp;rsquo;re comparing two systems on the same fixed inputs. Sample with replacement from the golden-set rows, recompute the per-system mean score on each bootstrap sample, take the 2.5th and 97.5th percentiles of the difference distribution as the 95% CI on the delta. A non-zero-containing CI is the significance gate. Bootstrap because the per-row scores aren&amp;rsquo;t normally distributed and t-tests overstate confidence.&lt;/li&gt;&#10;&lt;li&gt;Domain-classifier drift: &lt;a href="https://www.evidentlyai.com/blog/tutorial-detecting-drift-in-text-data"&gt;Evidently&amp;rsquo;s approach for raw text&lt;/a&gt;: train a binary classifier to distinguish reference-window text from current-window text; if the classifier hits ROC AUC &amp;gt; ~0.7, the distributions are separable and drift is happening. Useful when the input doesn&amp;rsquo;t reduce cleanly to categories or embeddings (e.g. raw markdown documents, long-form code).&lt;/li&gt;&#10;&lt;/ul&gt;&#10;&lt;p&gt;Pick the tool by the data shape, not by what&amp;rsquo;s fashionable. Don&amp;rsquo;t reach for KL divergence on a categorical mix where PSI is the bog-standard answer, and don&amp;rsquo;t run a KS test on a 7-bin distribution where PSI is more interpretable.&lt;/p&gt;&#10;&lt;h2 id="test-a-model-upgrade"&gt;Test a model upgrade&lt;/h2&gt;&#10;&lt;p&gt;A model upgrade; Sonnet 4.6 to Opus 4.7, Haiku 4.5 to whatever Anthropic ships next; is the most common shipping event that breaks an LLM app, because the provider&amp;rsquo;s pre-release evals tell you about &lt;em&gt;their&lt;/em&gt; benchmarks, not yours. Anthropic&amp;rsquo;s &lt;a href="https://platform.claude.com/docs/en/about-claude/model-deprecations"&gt;recent deprecation cadence&lt;/a&gt;; Sonnet 4 and Opus 4 retired June 15, 2026; Sonnet 4.5 cutover by May 18; means most production teams will run this protocol every few months for the lifetime of the application. The protocol that holds up has four phases.&lt;/p&gt;&#10;&lt;p&gt;&lt;strong&gt;Phase 1: offline regression on the golden set.&lt;/strong&gt; Run &lt;a href="https://jatinbansal.com/ai-engineering/eval-driven-development/"&gt;the golden set&lt;/a&gt; against both the baseline model and the candidate. Score every row with the &lt;em&gt;same pinned judge&lt;/em&gt; (the &lt;a href="https://jatinbansal.com/ai-engineering/llm-as-judge/"&gt;LLM-as-judge article&lt;/a&gt; is load-bearing here; judge calibration changes invalidate the comparison). Compute paired-bootstrap CIs on the per-category quality delta. The gate: every category&amp;rsquo;s delta CI either contains zero (no regression) or is strictly above zero (improvement); any category whose CI is strictly negative is a regression and the candidate fails the gate. Aggregate-only gating misses category-level regressions; gate per category.&lt;/p&gt;&#10;&lt;p&gt;&lt;strong&gt;Phase 2: shadow run on historical production traffic.&lt;/strong&gt; Pull the last 7 days of production traces, replay the inputs through the candidate without serving the outputs, and run the same judge against both baseline outputs (already scored, in the trace store) and candidate outputs. The signal here is two-fold: the same paired-bootstrap CI on per-category deltas (now over real production distribution, not the curated golden set), and a &lt;em&gt;behavioral diff&lt;/em&gt;; output length, tool-call frequency, refusal rate, structured-output validity; that catches behavioral regressions the judge isn&amp;rsquo;t sensitive to. A model that emits 40% more tokens to say the same thing is a cost regression even if the quality is identical, and the behavioral diff is what surfaces it.&lt;/p&gt;&#10;&lt;p&gt;&lt;strong&gt;Phase 3: canary deploy.&lt;/strong&gt; Route 1% of live traffic to the candidate, monitor for an hour, judge a 10% sample, gate on the same per-category CI test. If it passes, step to 5%, monitor for an hour, judge. Continue to 20%, 50%, 100% with the same checkpoint at each stop. The increments and dwell times are the &lt;a href="https://martinfowler.com/articles/feature-toggles.html"&gt;progressive-delivery primitives&lt;/a&gt; borrowed verbatim from microservices; the only LLM-specific adjustment is that the per-window judged sample is the gate, not request-level quality. A clean canary fails into rollback by flipping a single config flag, which is why the routing has to be a config knob, not a deploy step.&lt;/p&gt;&#10;&lt;p&gt;&lt;strong&gt;Phase 4: dual-run window and decommission.&lt;/strong&gt; After cutover, keep the baseline warm for at least a week; same prompt, same routing infrastructure, just not the default route. The dual-run window catches the slower drift modes that the canary missed: user behavior changing in response to the new model&amp;rsquo;s verbosity, downstream consumers depending on a specific output shape, cache patterns that the canary&amp;rsquo;s short window didn&amp;rsquo;t exercise. After the dual-run window passes clean, decommission the baseline and reclaim the resources; the &lt;a href="https://platform.claude.com/docs/en/about-claude/model-deprecations"&gt;provider&amp;rsquo;s pricing&lt;/a&gt; often makes the dual-run window a small cost relative to the rollback insurance it buys.&lt;/p&gt;&#10;&lt;p&gt;A subtler point. The protocol works for &lt;em&gt;model upgrades&lt;/em&gt; and &lt;em&gt;prompt edits&lt;/em&gt; with no modification; both are changes to the per-call surface and both fail in the same way. It does &lt;em&gt;not&lt;/em&gt; directly cover &lt;em&gt;retrieval changes&lt;/em&gt; (a new chunking strategy, a re-embedded vector store, a reranker swap) because the change affects the &lt;em&gt;context&lt;/em&gt; the model sees and the offline golden set&amp;rsquo;s recall metrics matter more than the judged output. &lt;a href="https://jatinbansal.com/ai-engineering/rag-evaluation/"&gt;The RAG evaluation article&lt;/a&gt; is the right specialization there; the protocol structure is the same, the metrics are different. Similarly, &lt;a href="https://jatinbansal.com/ai-engineering/memory-conflict-and-forgetting/"&gt;embedding-model upgrades&lt;/a&gt; need the dual-index migration pattern; the per-call regression protocol catches the symptom but not the cause.&lt;/p&gt;&#10;&lt;h2 id="code-an-embedding-drift-detector-over-a-trace-store-in-python"&gt;Code: an embedding-drift detector over a trace store in Python&lt;/h2&gt;&#10;&lt;p&gt;The detector below pulls user messages from the &lt;a href="https://github.com/langfuse/langfuse"&gt;Langfuse&lt;/a&gt; trace store, embeds them, computes PSI over the categorical mix, and centroid-distance over the embeddings, and emits a structured drift report. Install: &lt;code&gt;pip install langfuse openai numpy scipy&lt;/code&gt;.&lt;/p&gt;&#10;&lt;figure class="nb-code"&gt;&#10; &lt;header class="nb-code-chrome"&gt;&#10; &lt;span class="nb-code-label"&gt;python&lt;/span&gt;&#10; &#10; &lt;/header&gt;&#10; &lt;div class="nb-code-body"&gt;&lt;div class="highlight"&gt;&lt;div class="chroma"&gt;&#10;&lt;table class="lntable"&gt;&lt;tr&gt;&lt;td class="lntd"&gt;&#10;&lt;pre tabindex="0" class="chroma"&gt;&lt;code&gt;&lt;span class="lnt"&gt; 1&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 2&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 3&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 4&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 5&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 6&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 7&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 8&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 9&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 10&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 11&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 12&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 13&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 14&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 15&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 16&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 17&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 18&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 19&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 20&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 21&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 22&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 23&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 24&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 25&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 26&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 27&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 28&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 29&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 30&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 31&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 32&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 33&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 34&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 35&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 36&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 37&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 38&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 39&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 40&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 41&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 42&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 43&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 44&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 45&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 46&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 47&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 48&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 49&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 50&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 51&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 52&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 53&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 54&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 55&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 56&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 57&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 58&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 59&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 60&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 61&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 62&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 63&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 64&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 65&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 66&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 67&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 68&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 69&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 70&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 71&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 72&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 73&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 74&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 75&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 76&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 77&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 78&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 79&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 80&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 81&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 82&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 83&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 84&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 85&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 86&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 87&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 88&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 89&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 90&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 91&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 92&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 93&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 94&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 95&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 96&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 97&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 98&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 99&#10;&lt;/span&gt;&lt;span class="lnt"&gt;100&#10;&lt;/span&gt;&lt;span class="lnt"&gt;101&#10;&lt;/span&gt;&lt;span class="lnt"&gt;102&#10;&lt;/span&gt;&lt;span class="lnt"&gt;103&#10;&lt;/span&gt;&lt;span class="lnt"&gt;104&#10;&lt;/span&gt;&lt;span class="lnt"&gt;105&#10;&lt;/span&gt;&lt;span class="lnt"&gt;106&#10;&lt;/span&gt;&lt;span class="lnt"&gt;107&#10;&lt;/span&gt;&lt;span class="lnt"&gt;108&#10;&lt;/span&gt;&lt;span class="lnt"&gt;109&#10;&lt;/span&gt;&lt;span class="lnt"&gt;110&#10;&lt;/span&gt;&lt;span class="lnt"&gt;111&#10;&lt;/span&gt;&lt;span class="lnt"&gt;112&#10;&lt;/span&gt;&lt;span class="lnt"&gt;113&#10;&lt;/span&gt;&lt;span class="lnt"&gt;114&#10;&lt;/span&gt;&lt;span class="lnt"&gt;115&#10;&lt;/span&gt;&lt;span class="lnt"&gt;116&#10;&lt;/span&gt;&lt;span class="lnt"&gt;117&#10;&lt;/span&gt;&lt;span class="lnt"&gt;118&#10;&lt;/span&gt;&lt;span class="lnt"&gt;119&#10;&lt;/span&gt;&lt;span class="lnt"&gt;120&#10;&lt;/span&gt;&lt;span class="lnt"&gt;121&#10;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/td&gt;&#10;&lt;td class="lntd"&gt;&#10;&lt;pre tabindex="0" class="chroma"&gt;&lt;code class="language-python" data-lang="python"&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="c1"&gt;# pip install langfuse openai numpy scipy&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="nn"&gt;os&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="kn"&gt;from&lt;/span&gt; &lt;span class="nn"&gt;datetime&lt;/span&gt; &lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;datetime&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;timedelta&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;timezone&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="kn"&gt;from&lt;/span&gt; &lt;span class="nn"&gt;collections&lt;/span&gt; &lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;Counter&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="nn"&gt;numpy&lt;/span&gt; &lt;span class="k"&gt;as&lt;/span&gt; &lt;span class="nn"&gt;np&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="kn"&gt;from&lt;/span&gt; &lt;span class="nn"&gt;langfuse&lt;/span&gt; &lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;get_client&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="kn"&gt;from&lt;/span&gt; &lt;span class="nn"&gt;openai&lt;/span&gt; &lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;OpenAI&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="kn"&gt;from&lt;/span&gt; &lt;span class="nn"&gt;scipy.stats&lt;/span&gt; &lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;ks_2samp&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="n"&gt;langfuse&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;get_client&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="n"&gt;openai_client&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;OpenAI&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="n"&gt;EMBED_MODEL&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s2"&gt;&amp;#34;text-embedding-3-large&amp;#34;&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;fetch_user_messages&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;start&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;datetime&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;end&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;datetime&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;limit&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;int&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mi"&gt;5000&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="s2"&gt;&amp;#34;&amp;#34;&amp;#34;Pull user-turn inputs from the trace store as (text, category, output_len) tuples.&amp;#34;&amp;#34;&amp;#34;&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;traces&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;langfuse&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;api&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;trace&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;list&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;from_timestamp&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;start&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;to_timestamp&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;end&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;limit&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;limit&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;data&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;rows&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;[]&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;t&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;traces&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="ow"&gt;not&lt;/span&gt; &lt;span class="n"&gt;t&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;input&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="k"&gt;continue&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;text&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nb"&gt;str&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;t&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;input&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;&amp;#34;user_msg&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;&amp;#34;&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;category&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;t&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;metadata&lt;/span&gt; &lt;span class="ow"&gt;or&lt;/span&gt; &lt;span class="p"&gt;{})&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;&amp;#34;category&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;&amp;#34;unknown&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;output_len&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nb"&gt;len&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nb"&gt;str&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;t&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;output&lt;/span&gt; &lt;span class="ow"&gt;or&lt;/span&gt; &lt;span class="s2"&gt;&amp;#34;&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;rows&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;append&lt;/span&gt;&lt;span class="p"&gt;((&lt;/span&gt;&lt;span class="n"&gt;text&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;category&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;output_len&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;rows&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;psi&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;reference&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;list&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;current&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;list&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;bins&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;int&lt;/span&gt; &lt;span class="o"&gt;|&lt;/span&gt; &lt;span class="kc"&gt;None&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="kc"&gt;None&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;-&amp;gt;&lt;/span&gt; &lt;span class="nb"&gt;float&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="s2"&gt;&amp;#34;&amp;#34;&amp;#34;Population Stability Index over categorical mixes.&#10;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="s2"&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="s2"&gt; For a categorical input (list of labels), bins is the union of labels.&#10;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="s2"&gt; For a continuous input, pass bins as an integer and we histogram.&#10;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="s2"&gt; &amp;#34;&amp;#34;&amp;#34;&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;bins&lt;/span&gt; &lt;span class="ow"&gt;is&lt;/span&gt; &lt;span class="kc"&gt;None&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="c1"&gt;# Categorical: bins are the union of unique values.&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;labels&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nb"&gt;sorted&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nb"&gt;set&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;reference&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;|&lt;/span&gt; &lt;span class="nb"&gt;set&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;current&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;ref_counts&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;Counter&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;reference&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;cur_counts&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;Counter&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;current&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;ref_total&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nb"&gt;max&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nb"&gt;len&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;reference&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;cur_total&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nb"&gt;max&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nb"&gt;len&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;current&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;psi_total&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mf"&gt;0.0&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;label&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;labels&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;ref_pct&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;ref_counts&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;label&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="mf"&gt;1e-6&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt; &lt;span class="n"&gt;ref_total&lt;/span&gt; &lt;span class="c1"&gt;# smoothing&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;cur_pct&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;cur_counts&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;label&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="mf"&gt;1e-6&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt; &lt;span class="n"&gt;cur_total&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;psi_total&lt;/span&gt; &lt;span class="o"&gt;+=&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;cur_pct&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt; &lt;span class="n"&gt;ref_pct&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="n"&gt;np&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;log&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;cur_pct&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt; &lt;span class="n"&gt;ref_pct&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="nb"&gt;float&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;psi_total&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="c1"&gt;# Continuous: equal-width bins over the reference range.&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;edges&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;np&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;linspace&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nb"&gt;min&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;reference&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt; &lt;span class="nb"&gt;max&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;reference&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt; &lt;span class="n"&gt;bins&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;ref_hist&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;_&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;np&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;histogram&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;reference&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;bins&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;edges&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;cur_hist&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;_&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;np&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;histogram&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;current&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;bins&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;edges&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;ref_pct&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;ref_hist&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="mf"&gt;1e-6&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt; &lt;span class="nb"&gt;max&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;ref_hist&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;sum&lt;/span&gt;&lt;span class="p"&gt;())&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;cur_pct&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;cur_hist&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="mf"&gt;1e-6&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt; &lt;span class="nb"&gt;max&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;cur_hist&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;sum&lt;/span&gt;&lt;span class="p"&gt;())&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="nb"&gt;float&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;np&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;sum&lt;/span&gt;&lt;span class="p"&gt;((&lt;/span&gt;&lt;span class="n"&gt;cur_pct&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt; &lt;span class="n"&gt;ref_pct&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="n"&gt;np&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;log&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;cur_pct&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt; &lt;span class="n"&gt;ref_pct&lt;/span&gt;&lt;span class="p"&gt;)))&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;embed_batch&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;texts&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;list&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="nb"&gt;str&lt;/span&gt;&lt;span class="p"&gt;])&lt;/span&gt; &lt;span class="o"&gt;-&amp;gt;&lt;/span&gt; &lt;span class="n"&gt;np&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;ndarray&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="s2"&gt;&amp;#34;&amp;#34;&amp;#34;Batch-embed texts; production should retry/backoff.&amp;#34;&amp;#34;&amp;#34;&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;resp&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;openai_client&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;embeddings&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;create&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;model&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;EMBED_MODEL&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nb"&gt;input&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;texts&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;np&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;array&lt;/span&gt;&lt;span class="p"&gt;([&lt;/span&gt;&lt;span class="n"&gt;e&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;embedding&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;e&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;resp&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;data&lt;/span&gt;&lt;span class="p"&gt;])&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;centroid_distance&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;ref_emb&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;np&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;ndarray&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;cur_emb&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;np&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;ndarray&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;-&amp;gt;&lt;/span&gt; &lt;span class="nb"&gt;float&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="s2"&gt;&amp;#34;&amp;#34;&amp;#34;Cosine distance between the two centroids.&amp;#34;&amp;#34;&amp;#34;&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;ref_c&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;ref_emb&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;mean&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;axis&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;cur_c&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;cur_emb&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;mean&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;axis&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;cos&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;np&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;dot&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;ref_c&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;cur_c&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;np&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;linalg&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;norm&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;ref_c&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="n"&gt;np&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;linalg&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;norm&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;cur_c&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="nb"&gt;float&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mf"&gt;1.0&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt; &lt;span class="n"&gt;cos&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;run_drift_check&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;reference_days&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;int&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mi"&gt;7&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;current_hours&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;int&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mi"&gt;24&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;-&amp;gt;&lt;/span&gt; &lt;span class="nb"&gt;dict&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;end&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;datetime&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;now&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;timezone&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;utc&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;ref_start&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;end&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt; &lt;span class="n"&gt;timedelta&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;days&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;reference_days&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt; &lt;span class="n"&gt;timedelta&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;hours&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;current_hours&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;ref_end&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;end&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt; &lt;span class="n"&gt;timedelta&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;hours&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;current_hours&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;ref&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;fetch_user_messages&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;ref_start&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;ref_end&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;cur&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;fetch_user_messages&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;ref_end&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;end&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="ow"&gt;not&lt;/span&gt; &lt;span class="n"&gt;ref&lt;/span&gt; &lt;span class="ow"&gt;or&lt;/span&gt; &lt;span class="ow"&gt;not&lt;/span&gt; &lt;span class="n"&gt;cur&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="s2"&gt;&amp;#34;status&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;&amp;#34;insufficient_data&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;&amp;#34;ref_n&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;len&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;ref&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt; &lt;span class="s2"&gt;&amp;#34;cur_n&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;len&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;cur&lt;/span&gt;&lt;span class="p"&gt;)}&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="c1"&gt;# Category PSI&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;cat_psi&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;psi&lt;/span&gt;&lt;span class="p"&gt;([&lt;/span&gt;&lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;r&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;ref&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;c&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;c&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;cur&lt;/span&gt;&lt;span class="p"&gt;])&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="c1"&gt;# Output-length KS&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;ks_stat&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;ks_p&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;ks_2samp&lt;/span&gt;&lt;span class="p"&gt;([&lt;/span&gt;&lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;2&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;r&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;ref&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;c&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;2&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;c&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;cur&lt;/span&gt;&lt;span class="p"&gt;])&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="c1"&gt;# Embedding centroid distance — sample to bound cost&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;ref_sample&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;r&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;ref&lt;/span&gt;&lt;span class="p"&gt;[:&lt;/span&gt; &lt;span class="nb"&gt;min&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;500&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nb"&gt;len&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;ref&lt;/span&gt;&lt;span class="p"&gt;))]]&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;cur_sample&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;c&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;c&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;cur&lt;/span&gt;&lt;span class="p"&gt;[:&lt;/span&gt; &lt;span class="nb"&gt;min&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;500&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nb"&gt;len&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;cur&lt;/span&gt;&lt;span class="p"&gt;))]]&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;ref_emb&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;embed_batch&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;ref_sample&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;cur_emb&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;embed_batch&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;cur_sample&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;cent_dist&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;centroid_distance&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;ref_emb&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;cur_emb&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="c1"&gt;# Decision: any single signal can fire, but the action is &amp;#34;investigate the slice&amp;#34;.&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;alerts&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;[]&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;cat_psi&lt;/span&gt; &lt;span class="o"&gt;&amp;gt;&lt;/span&gt; &lt;span class="mf"&gt;0.2&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;alerts&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;append&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sa"&gt;f&lt;/span&gt;&lt;span class="s2"&gt;&amp;#34;category_psi=&lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;cat_psi&lt;/span&gt;&lt;span class="si"&gt;:&lt;/span&gt;&lt;span class="s2"&gt;.3f&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s2"&gt; (material)&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="k"&gt;elif&lt;/span&gt; &lt;span class="n"&gt;cat_psi&lt;/span&gt; &lt;span class="o"&gt;&amp;gt;&lt;/span&gt; &lt;span class="mf"&gt;0.1&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;alerts&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;append&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sa"&gt;f&lt;/span&gt;&lt;span class="s2"&gt;&amp;#34;category_psi=&lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;cat_psi&lt;/span&gt;&lt;span class="si"&gt;:&lt;/span&gt;&lt;span class="s2"&gt;.3f&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s2"&gt; (worth_glance)&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;ks_p&lt;/span&gt; &lt;span class="o"&gt;&amp;lt;&lt;/span&gt; &lt;span class="mf"&gt;0.01&lt;/span&gt; &lt;span class="ow"&gt;and&lt;/span&gt; &lt;span class="n"&gt;ks_stat&lt;/span&gt; &lt;span class="o"&gt;&amp;gt;&lt;/span&gt; &lt;span class="mf"&gt;0.1&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;alerts&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;append&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sa"&gt;f&lt;/span&gt;&lt;span class="s2"&gt;&amp;#34;output_length_ks_D=&lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;ks_stat&lt;/span&gt;&lt;span class="si"&gt;:&lt;/span&gt;&lt;span class="s2"&gt;.3f&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s2"&gt;, p=&lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;ks_p&lt;/span&gt;&lt;span class="si"&gt;:&lt;/span&gt;&lt;span class="s2"&gt;.4f&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s2"&gt;&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;cent_dist&lt;/span&gt; &lt;span class="o"&gt;&amp;gt;&lt;/span&gt; &lt;span class="mf"&gt;0.05&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;alerts&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;append&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sa"&gt;f&lt;/span&gt;&lt;span class="s2"&gt;&amp;#34;embedding_centroid_distance=&lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;cent_dist&lt;/span&gt;&lt;span class="si"&gt;:&lt;/span&gt;&lt;span class="s2"&gt;.4f&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s2"&gt;&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="s2"&gt;&amp;#34;status&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;&amp;#34;drift&amp;#34;&lt;/span&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;alerts&lt;/span&gt; &lt;span class="k"&gt;else&lt;/span&gt; &lt;span class="s2"&gt;&amp;#34;stable&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="s2"&gt;&amp;#34;alerts&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;alerts&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="s2"&gt;&amp;#34;category_psi&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;cat_psi&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="s2"&gt;&amp;#34;output_length_ks_stat&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;float&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;ks_stat&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="s2"&gt;&amp;#34;output_length_ks_p&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;float&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;ks_p&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="s2"&gt;&amp;#34;embedding_centroid_distance&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;cent_dist&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="s2"&gt;&amp;#34;n_reference&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;len&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;ref&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="s2"&gt;&amp;#34;n_current&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;len&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;cur&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="p"&gt;}&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="vm"&gt;__name__&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="s2"&gt;&amp;#34;__main__&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="nn"&gt;json&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="nb"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;json&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;dumps&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;run_drift_check&lt;/span&gt;&lt;span class="p"&gt;(),&lt;/span&gt; &lt;span class="n"&gt;indent&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;2&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/td&gt;&lt;/tr&gt;&lt;/table&gt;&#10;&lt;/div&gt;&#10;&lt;/div&gt;&lt;/div&gt;&#10;&lt;/figure&gt;&#10;&lt;p&gt;The three signals; category PSI, output-length KS, embedding-centroid distance; are &lt;em&gt;independent&lt;/em&gt;. Any one firing is enough to investigate; the conjunction is the strong signal. The thresholds are starting points (PSI 0.2 / 0.1 from the &lt;a href="https://en.wikipedia.org/wiki/Population_stability_index"&gt;scorecard literature&lt;/a&gt;, KS p&amp;lt;0.01 and D&amp;gt;0.1 to filter weak effects); recalibrate against your own false-positive rate after a month. The embedding step is sampled to 500 messages per window; at $0.13 per million tokens for &lt;code&gt;text-embedding-3-large&lt;/code&gt; and ~80 tokens per message, an hourly run costs around $0.50 a day, which is the cheap shadow signal. The expensive judged signal sits behind this and runs only when an alert fires.&lt;/p&gt;&#10;&lt;h2 id="further-reading"&gt;Further reading&lt;/h2&gt;&#10;&lt;ul&gt;&#10;&lt;li&gt;&lt;a href="https://huyenchip.com/2022/02/07/data-distribution-shifts-and-monitoring.html"&gt;Chip Huyen; Data Distribution Shifts and Monitoring&lt;/a&gt;; the ML-systems reference for the conceptual frame: covariate shift, concept drift, label shift, and the monitoring pipeline that catches each. Pre-LLM but the taxonomy transfers cleanly, and the discussion of &lt;em&gt;when&lt;/em&gt; to recompute baselines and &lt;em&gt;how&lt;/em&gt; to gate retraining is the load-bearing piece for the LLM application.&lt;/li&gt;&#10;&lt;li&gt;&lt;a href="https://www.evidentlyai.com/blog/tutorial-detecting-drift-in-text-data"&gt;Evidently; Monitoring NLP models in production: detecting drift in text data&lt;/a&gt;; the practitioner&amp;rsquo;s tutorial on text-data drift, including the domain-classifier approach, descriptor monitoring, and the embedding-drift comparison. The cleanest single resource for the input-distribution-drift problem on raw text.&lt;/li&gt;&#10;&lt;li&gt;&lt;a href="https://arize.com/docs/ax/machine-learning/computer-vision/how-to-cv/embedding-drift"&gt;Arize; Embedding Drift documentation&lt;/a&gt;; Arize&amp;rsquo;s canonical writeup of centroid-distance-based drift on embedding inputs, with the HDBSCAN clustering pipeline and the UMAP projection that the Phoenix UI exposes. Pairs well with &lt;a href="https://medium.com/data-science/measuring-embedding-drift-aa9b7ddb84ae"&gt;Aparna Dhinakaran&amp;rsquo;s measuring embedding drift&lt;/a&gt; for the data-science framing.&lt;/li&gt;&#10;&lt;li&gt;&lt;a href="https://eugeneyan.com/writing/llm-patterns/"&gt;Eugene Yan; Patterns for Building LLM-Based Systems&lt;/a&gt;; the cross-pillar pattern catalogue, with the &lt;em&gt;guardrails&lt;/em&gt; and &lt;em&gt;evals&lt;/em&gt; sections covering the steady-state monitoring discipline this article instantiates. The connective tissue piece for evals-plus-observability-plus-drift as one stack.&lt;/li&gt;&#10;&lt;li&gt;&lt;a href="https://arxiv.org/abs/2311.11123"&gt;Ma et al.; (Why) Is My Prompt Getting Worse? Rethinking Regression Testing for Evolving LLM APIs&lt;/a&gt;; the academic case for &lt;em&gt;prompt-level&lt;/em&gt; regression testing across silent model updates, with measurements of how much the same prompt drifts across provider versions and the case for paired, fixed-input regression suites. Maps directly to the protocol in this article.&lt;/li&gt;&#10;&lt;/ul&gt;&#10;</content:encoded></item><item><title>Production Tracing and Observability for LLM Systems</title><link>https://jatinbansal.com/ai-engineering/llm-observability/</link><guid isPermaLink="true">https://jatinbansal.com/ai-engineering/llm-observability/</guid><pubDate>Tue, 19 May 2026 13:32:00 +0000</pubDate><dc:creator>Jatin Bansal</dc:creator><category>AI Engineering</category><description>How to trace model calls, tools, retrieval, cost, cache usage, and evaluation in LLM applications.</description><content:encoded>&lt;p&gt;Flat model-call logs cannot show which loop step produced a bad answer, what the retriever returned, or whether a cache miss caused a latency spike. An LLM trace connects the user turn, generations, retrievals, tools, and evaluators so an engineer can reconstruct the path to the final response.&lt;/p&gt;&#10;&lt;h2 id="trace-the-application-path"&gt;Trace the application path&lt;/h2&gt;&#10;&lt;p&gt;LLM observability captures structured telemetry for every model call, tool invocation, retrieval, and evaluator pass. One user turn produces a tree of spans linked by trace and parent IDs. Typed attributes hold prompts, completions, tool arguments, retrieved chunks, token counts, cache state, latency, and cost. The &lt;a href="https://opentelemetry.io/docs/specs/semconv/gen-ai/"&gt;OpenTelemetry GenAI conventions&lt;/a&gt; and &lt;a href="https://github.com/Arize-ai/openinference"&gt;OpenInference&lt;/a&gt; provide portable attribute names.&lt;/p&gt;&#10;&lt;h2 id="define-the-span-tree"&gt;Define the span tree&lt;/h2&gt;&#10;&lt;p&gt;A working LLM trace for an agent turn has six span types. The relationships between them are what turn flat logs into a debuggable surface.&lt;/p&gt;&#10;&lt;ol&gt;&#10;&lt;li&gt;The session span: One per user conversation. Holds tenant ID, user ID, session start/end timestamps, total cost and tokens rolled up from children. The query unit when support escalates a thread.&lt;/li&gt;&#10;&lt;li&gt;The turn span (parent): One per user → assistant exchange. Children are everything that ran inside that exchange. Carries the user message, the final assistant message, the cumulative cost, the cumulative latency, and signal flags (&lt;code&gt;error&lt;/code&gt;, &lt;code&gt;budget_breach&lt;/code&gt;, &lt;code&gt;compaction_triggered&lt;/code&gt;). Parent span of every other span in the turn.&lt;/li&gt;&#10;&lt;li&gt;The generation span (LLM call): The OTel &lt;code&gt;gen_ai.client&lt;/code&gt; span. One per &lt;code&gt;messages.create&lt;/code&gt; call to a provider. Attributes: &lt;code&gt;gen_ai.request.model&lt;/code&gt;, &lt;code&gt;gen_ai.system&lt;/code&gt; (anthropic/openai/&amp;hellip;), &lt;code&gt;gen_ai.usage.input_tokens&lt;/code&gt;, &lt;code&gt;gen_ai.usage.output_tokens&lt;/code&gt;, &lt;code&gt;gen_ai.usage.cache_read_input_tokens&lt;/code&gt;, &lt;code&gt;gen_ai.usage.cache_creation_input_tokens&lt;/code&gt;, &lt;code&gt;gen_ai.response.finish_reasons&lt;/code&gt;, latency, cost. Payload as &lt;code&gt;gen_ai.prompt.{N}.content&lt;/code&gt; and &lt;code&gt;gen_ai.completion.{N}.content&lt;/code&gt; events (or, in the &lt;a href="https://github.com/Arize-ai/openinference/blob/main/spec/semantic_conventions.md"&gt;OpenInference convention&lt;/a&gt;, &lt;code&gt;llm.input_messages&lt;/code&gt; and &lt;code&gt;llm.output_messages&lt;/code&gt; as JSON-encoded attributes).&lt;/li&gt;&#10;&lt;li&gt;The tool call span: One per &lt;code&gt;tool_use&lt;/code&gt; block dispatched. Attributes: &lt;code&gt;tool.name&lt;/code&gt;, &lt;code&gt;tool.arguments&lt;/code&gt; (JSON), &lt;code&gt;tool.result&lt;/code&gt; (JSON), &lt;code&gt;tool.error&lt;/code&gt; (if any), latency, retry count. The child of the generation span that emitted the &lt;code&gt;tool_use&lt;/code&gt;, which is in turn the parent of the &lt;em&gt;next&lt;/em&gt; generation span that consumes the &lt;code&gt;tool_result&lt;/code&gt;. This is the loop body of &lt;a href="https://jatinbansal.com/ai-engineering/agent-loop/"&gt;the agent loop article&lt;/a&gt; made visible.&lt;/li&gt;&#10;&lt;li&gt;The retrieval span: One per &lt;a href="https://jatinbansal.com/ai-engineering/vector-databases-ann/"&gt;retriever&lt;/a&gt; call. Attributes: &lt;code&gt;retrieval.query&lt;/code&gt;, &lt;code&gt;retrieval.top_k&lt;/code&gt;, &lt;code&gt;retrieval.documents&lt;/code&gt; (list of &lt;code&gt;{id, score, snippet}&lt;/code&gt; records), latency. Child of the tool-call span when retrieval is exposed as a tool; child of the turn span directly when retrieval runs in a &lt;a href="https://jatinbansal.com/ai-engineering/context-engineering-jit-vs-aot/"&gt;JIT context loop&lt;/a&gt; outside the model&amp;rsquo;s tool surface.&lt;/li&gt;&#10;&lt;li&gt;The evaluator/judge span: One per LLM-judge call. Attributes: &lt;code&gt;eval.rubric&lt;/code&gt;, &lt;code&gt;eval.score&lt;/code&gt;, &lt;code&gt;eval.reasoning&lt;/code&gt;, judge model name. Linked to the turn span by the same trace ID but typically run &lt;em&gt;after&lt;/em&gt; the turn closes; sampled traces get re-scored offline, and the eval spans append to the existing trace rather than living in a separate store. This is the bridge between observability and the &lt;a href="https://jatinbansal.com/ai-engineering/eval-driven-development/"&gt;eval pyramid&lt;/a&gt;: the judge is offline, but the &lt;em&gt;trace it scored&lt;/em&gt; is the production artifact.&lt;/li&gt;&#10;&lt;/ol&gt;&#10;&lt;p&gt;The non-obvious requirement is the &lt;em&gt;parent linkage discipline&lt;/em&gt;. A turn span with three generation spans as siblings (no causal ordering) is much less debuggable than a turn span with a chain of generation → tool-call → generation spans, where each generation&amp;rsquo;s parent is the tool-call whose result it consumed. The chain encodes the ReAct loop&amp;rsquo;s iteration count, and lets you ask &amp;ldquo;show me turns where the second generation emitted an error&amp;rdquo;; the kind of query that catches the failure mode where the loop driver is hiding an exception three frames deep.&lt;/p&gt;&#10;&lt;h2 id="choose-semantic-conventions"&gt;Choose semantic conventions&lt;/h2&gt;&#10;&lt;p&gt;Two semantic conventions compete in 2026. Choose deliberately; they are converging but not identical.&lt;/p&gt;&#10;&lt;p&gt;&lt;strong&gt;&lt;a href="https://opentelemetry.io/docs/specs/semconv/gen-ai/"&gt;OpenTelemetry GenAI semantic conventions&lt;/a&gt;&lt;/strong&gt; are the OTel SIG&amp;rsquo;s standardization, developed since April 2024. The &lt;code&gt;gen_ai.client&lt;/code&gt; span is stable as of late 2025; the &lt;code&gt;gen_ai.agent&lt;/code&gt; span and a number of attribute names are still experimental in mid-2026. The convention uses log-event attachments for prompt/completion content (so payloads can be sampled separately from span attributes), which fits OTel&amp;rsquo;s existing log pipeline shape but makes naive trace UIs awkward; the message content is in events, not attributes. Vendor support: &lt;a href="https://www.datadoghq.com/blog/llm-otel-semantic-convention/"&gt;Datadog&lt;/a&gt;, &lt;a href="https://www.honeycomb.io/use-cases/ai-llm-observability"&gt;Honeycomb&lt;/a&gt;, &lt;a href="https://langfuse.com/integrations/native/opentelemetry"&gt;Langfuse&lt;/a&gt;, and increasingly Phoenix.&lt;/p&gt;&#10;&lt;p&gt;&lt;strong&gt;&lt;a href="https://github.com/Arize-ai/openinference"&gt;OpenInference&lt;/a&gt;&lt;/strong&gt; is Arize&amp;rsquo;s parallel convention, originally designed for &lt;a href="https://github.com/Arize-ai/phoenix"&gt;Arize Phoenix&lt;/a&gt; and shipped as the default instrumentation for LangChain, LlamaIndex, and many SDKs. OpenInference uses span attributes (&lt;code&gt;llm.input_messages&lt;/code&gt;, &lt;code&gt;llm.output_messages&lt;/code&gt;, &lt;code&gt;llm.token_count.*&lt;/code&gt;, &lt;code&gt;tool.name&lt;/code&gt;, &lt;code&gt;tool.parameters&lt;/code&gt;) rather than events, which is friendlier to most trace viewers but bloats individual spans. The convention covers retrieval (&lt;code&gt;retrieval.documents&lt;/code&gt;), agents (&lt;code&gt;agent.span.kind&lt;/code&gt;), and embeddings (&lt;code&gt;embedding.embeddings&lt;/code&gt;) in addition to the basic LLM call.&lt;/p&gt;&#10;&lt;p&gt;In practice, the two are close enough that a translation layer between them is straightforward, and &lt;a href="https://github.com/Arize-ai/openinference/blob/main/spec/semantic_conventions.md"&gt;OpenInference&amp;rsquo;s roadmap&lt;/a&gt; tracks the OTel work explicitly. The pragmatic rule: if your target backend is Phoenix or Arize AX, instrument against OpenInference; if your target is anything else (Langfuse, Datadog, LangSmith, Honeycomb, a vanilla OTel collector), instrument against OTel GenAI conventions. Auto-instrumentation libraries (the OpenInference SDKs, the OTel auto-instrumentations) handle most of the per-framework wiring; the choice is mostly about which attribute names land in your queries.&lt;/p&gt;&#10;&lt;h2 id="sample-traces-and-scrub-pii"&gt;Sample traces and scrub PII&lt;/h2&gt;&#10;&lt;p&gt;Two policies determine whether the trace pipeline is sustainable or a cost runaway.&lt;/p&gt;&#10;&lt;p&gt;&lt;strong&gt;Sampling.&lt;/strong&gt; A naive pipeline that stores every prompt and completion runs into payload-cost economics fast. The pattern that survives: &lt;strong&gt;head-based sampling for cheap aggregates, tail-based sampling for forensic traces, plus 100% retention of error/anomaly traces.&lt;/strong&gt; Concretely:&lt;/p&gt;&#10;&lt;ul&gt;&#10;&lt;li&gt;Every span is summarized into a metric row (no payload, just attributes) and shipped to the metrics store. Cheap, sub-millisecond, every turn.&lt;/li&gt;&#10;&lt;li&gt;A configurable fraction of spans; 1-10% in production, 100% in staging; have their &lt;em&gt;payloads&lt;/em&gt; shipped to the trace store. Random selection by trace ID so all spans in a sampled trace are kept together.&lt;/li&gt;&#10;&lt;li&gt;Any trace with an &lt;code&gt;error&lt;/code&gt; flag, a &lt;code&gt;budget_breach&lt;/code&gt;, a &lt;code&gt;judge.score&lt;/code&gt; below a threshold, or a &lt;a href="https://jatinbansal.com/ai-engineering/human-feedback-loops/"&gt;user-submitted thumbs-down&lt;/a&gt; is force-sampled at 100% regardless of the random sample. Forensic capture for the cases you actually need to debug.&lt;/li&gt;&#10;&lt;li&gt;The metric store is queried for trends; the trace store is queried for investigation. A bug report arrives with a session ID; the trace store has the session&amp;rsquo;s spans; the metric store doesn&amp;rsquo;t.&lt;/li&gt;&#10;&lt;/ul&gt;&#10;&lt;p&gt;The &lt;a href="https://opentelemetry.io/docs/concepts/sampling/"&gt;tail-based variant&lt;/a&gt; requires a buffer at the collector that holds spans until the trace closes, then samples based on aggregate criteria; useful when &amp;ldquo;errors&amp;rdquo; aren&amp;rsquo;t known at span start. Production teams typically combine: head-based for the baseline sample, tail-based with the OTel collector&amp;rsquo;s &lt;code&gt;tail_sampling_processor&lt;/code&gt; for the error/anomaly augmentation.&lt;/p&gt;&#10;&lt;p&gt;&lt;strong&gt;PII.&lt;/strong&gt; Prompts contain user data; completions contain model-generated data that sometimes reflects the user data verbatim. The minimum-viable policy: a scrubbing layer between instrumentation and export, with named-entity recognition (regex first for emails, phone numbers, credit-card patterns; an NER pass for names and addresses if the workload warrants it), a deterministic hash for fields that need to be queryable across spans without being identifying (e.g. user IDs), and a versioned scrubbing manifest checked into the repo so changes go through review. The &lt;a href="https://jatinbansal.com/ai-engineering/pii-and-privacy/"&gt;PII detection and data privacy article&lt;/a&gt; walks the detection cascade and the deletion pipeline for the privacy layer the trace store is also subject to. The &lt;a href="https://jatinbansal.com/ai-engineering/memory-privacy-multi-tenancy/"&gt;GDPR right-to-be-forgotten&lt;/a&gt; story applies here too; a trace containing a deleted user&amp;rsquo;s data is a deletion-target the same way the memory store is.&lt;/p&gt;&#10;&lt;p&gt;A subtler PII issue: judged eval spans append after the fact to a &lt;em&gt;production&lt;/em&gt; trace, and the judge&amp;rsquo;s reasoning may quote the user&amp;rsquo;s content. If your scrubbing runs at instrumentation time but the judge runs at eval time, the judge&amp;rsquo;s output bypasses the scrubber. Add a separate scrub pass on judge-emitted attributes, or run the judge against the already-scrubbed payload and accept the (usually small) quality hit.&lt;/p&gt;&#10;&lt;h2 id="attribute-cost-and-cache-usage"&gt;Attribute cost and cache usage&lt;/h2&gt;&#10;&lt;p&gt;Two attributes that production teams find themselves wishing they had captured from day one: &lt;strong&gt;cache state&lt;/strong&gt; and &lt;strong&gt;cost attribution&lt;/strong&gt;.&lt;/p&gt;&#10;&lt;p&gt;&lt;strong&gt;Cache state.&lt;/strong&gt; The &lt;a href="https://jatinbansal.com/ai-engineering/prompt-caching/"&gt;prompt-caching article&lt;/a&gt; made the case for &lt;code&gt;cache_creation_input_tokens&lt;/code&gt; vs &lt;code&gt;cache_read_input_tokens&lt;/code&gt; being first-class. The corresponding observability requirement: every generation span tags those two counts as separate attributes, and a derived &lt;code&gt;cache_hit_rate = cache_read / (cache_read + cache_creation + cache_miss_input)&lt;/code&gt; metric lands on the per-session dashboard. The &lt;a href="https://jatinbansal.com/ai-engineering/agent-harness-anatomy/"&gt;agent harness anatomy article&lt;/a&gt; flagged that a harness without cache telemetry runs at 5-10× the cost it should be; invisible until the invoice arrives. The trace is the only place to see &lt;em&gt;which turn&lt;/em&gt; killed the cache, which is the only way to fix the assembly bug at its source.&lt;/p&gt;&#10;&lt;p&gt;&lt;strong&gt;Cost.&lt;/strong&gt; Per-span cost in dollars (computed from token counts and a pricing table) lets the trace store answer &amp;ldquo;show me the top 10 most expensive turns this week,&amp;rdquo; &amp;ldquo;what&amp;rsquo;s the cost distribution by tool,&amp;rdquo; &amp;ldquo;which tenants are heavy users.&amp;rdquo; Tagging the session span with &lt;code&gt;tenant_id&lt;/code&gt; and the generation span with &lt;code&gt;model.id&lt;/code&gt; and &lt;code&gt;model.tier&lt;/code&gt; enables per-tenant per-model rollups that finance asks for once a quarter. Pricing tables change quarterly across providers; pin the pricing-table version into the trace store at the span attribute level (&lt;code&gt;cost.pricing_version&lt;/code&gt;) so historical comparisons stay coherent across rate changes.&lt;/p&gt;&#10;&lt;h2 id="code-instrumented-python-harness-with-langfuse"&gt;Code: instrumented Python harness with Langfuse&lt;/h2&gt;&#10;&lt;p&gt;&lt;a href="https://github.com/langfuse/langfuse"&gt;Langfuse&amp;rsquo;s Python SDK&lt;/a&gt; is the cleanest entry point for a self-hostable, OTel-native trace pipeline. The harness below wraps an Anthropic agent with the loop body from the &lt;a href="https://jatinbansal.com/ai-engineering/agent-loop/"&gt;agent loop article&lt;/a&gt;, emits the span shape above, and ships to a Langfuse backend. Install: &lt;code&gt;pip install langfuse anthropic&lt;/code&gt;.&lt;/p&gt;&#10;&lt;figure class="nb-code"&gt;&#10; &lt;header class="nb-code-chrome"&gt;&#10; &lt;span class="nb-code-label"&gt;python&lt;/span&gt;&#10; &#10; &lt;/header&gt;&#10; &lt;div class="nb-code-body"&gt;&lt;div class="highlight"&gt;&lt;div class="chroma"&gt;&#10;&lt;table class="lntable"&gt;&lt;tr&gt;&lt;td class="lntd"&gt;&#10;&lt;pre tabindex="0" class="chroma"&gt;&lt;code&gt;&lt;span class="lnt"&gt; 1&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 2&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 3&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 4&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 5&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 6&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 7&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 8&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 9&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 10&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 11&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 12&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 13&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 14&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 15&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 16&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 17&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 18&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 19&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 20&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 21&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 22&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 23&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 24&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 25&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 26&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 27&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 28&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 29&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 30&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 31&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 32&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 33&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 34&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 35&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 36&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 37&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 38&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 39&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 40&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 41&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 42&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 43&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 44&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 45&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 46&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 47&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 48&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 49&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 50&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 51&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 52&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 53&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 54&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 55&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 56&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 57&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 58&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 59&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 60&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 61&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 62&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 63&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 64&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 65&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 66&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 67&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 68&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 69&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 70&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 71&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 72&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 73&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 74&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 75&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 76&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 77&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 78&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 79&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 80&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 81&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 82&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 83&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 84&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 85&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 86&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 87&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 88&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 89&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 90&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 91&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 92&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 93&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 94&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 95&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 96&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 97&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 98&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 99&#10;&lt;/span&gt;&lt;span class="lnt"&gt;100&#10;&lt;/span&gt;&lt;span class="lnt"&gt;101&#10;&lt;/span&gt;&lt;span class="lnt"&gt;102&#10;&lt;/span&gt;&lt;span class="lnt"&gt;103&#10;&lt;/span&gt;&lt;span class="lnt"&gt;104&#10;&lt;/span&gt;&lt;span class="lnt"&gt;105&#10;&lt;/span&gt;&lt;span class="lnt"&gt;106&#10;&lt;/span&gt;&lt;span class="lnt"&gt;107&#10;&lt;/span&gt;&lt;span class="lnt"&gt;108&#10;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/td&gt;&#10;&lt;td class="lntd"&gt;&#10;&lt;pre tabindex="0" class="chroma"&gt;&lt;code class="language-python" data-lang="python"&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="c1"&gt;# pip install langfuse anthropic&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="nn"&gt;os&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt; &lt;span class="nn"&gt;time&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt; &lt;span class="nn"&gt;hashlib&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="kn"&gt;from&lt;/span&gt; &lt;span class="nn"&gt;anthropic&lt;/span&gt; &lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;Anthropic&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="kn"&gt;from&lt;/span&gt; &lt;span class="nn"&gt;langfuse&lt;/span&gt; &lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;get_client&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="c1"&gt;# Langfuse client picks up LANGFUSE_PUBLIC_KEY / LANGFUSE_SECRET_KEY / LANGFUSE_HOST env vars.&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="n"&gt;langfuse&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;get_client&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="n"&gt;anthropic_client&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;Anthropic&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="n"&gt;MODEL&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s2"&gt;&amp;#34;claude-sonnet-4-6&amp;#34;&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="n"&gt;PRICING&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="s2"&gt;&amp;#34;claude-sonnet-4-6&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mf"&gt;3.00&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mf"&gt;15.00&lt;/span&gt;&lt;span class="p"&gt;)}&lt;/span&gt; &lt;span class="c1"&gt;# $/MTok input, output&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;scrub&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;text&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;str&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;-&amp;gt;&lt;/span&gt; &lt;span class="nb"&gt;str&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="s2"&gt;&amp;#34;&amp;#34;&amp;#34;Naive PII scrubbing; production uses a real NER pipeline.&amp;#34;&amp;#34;&amp;#34;&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="c1"&gt;# Hash emails for queryability without identification.&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="nn"&gt;re&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;re&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;sub&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="sa"&gt;r&lt;/span&gt;&lt;span class="s2"&gt;&amp;#34;[\w.+-]+@[\w-]+\.[\w.-]+&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="k"&gt;lambda&lt;/span&gt; &lt;span class="n"&gt;m&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="sa"&gt;f&lt;/span&gt;&lt;span class="s2"&gt;&amp;#34;&amp;lt;email:&lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;hashlib&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;sha256&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;m&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;group&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;encode&lt;/span&gt;&lt;span class="p"&gt;())&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;hexdigest&lt;/span&gt;&lt;span class="p"&gt;()[:&lt;/span&gt;&lt;span class="mi"&gt;8&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s2"&gt;&amp;gt;&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;text&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="p"&gt;)&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;run_turn&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;session_id&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;str&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;tenant_id&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;str&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;user_msg&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;str&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;tools&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;list&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;dispatch&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="c1"&gt;# Turn span is the parent of every other span in this iteration.&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="k"&gt;with&lt;/span&gt; &lt;span class="n"&gt;langfuse&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;start_as_current_observation&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;as_type&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;#34;span&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;name&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;#34;agent.turn&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="nb"&gt;input&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="s2"&gt;&amp;#34;user_msg&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;scrub&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;user_msg&lt;/span&gt;&lt;span class="p"&gt;)},&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;metadata&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="s2"&gt;&amp;#34;session_id&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;session_id&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;&amp;#34;tenant_id&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;tenant_id&lt;/span&gt;&lt;span class="p"&gt;},&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;as&lt;/span&gt; &lt;span class="n"&gt;turn&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;messages&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;[{&lt;/span&gt;&lt;span class="s2"&gt;&amp;#34;role&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;&amp;#34;user&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;&amp;#34;content&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;user_msg&lt;/span&gt;&lt;span class="p"&gt;}]&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;total_cost&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mf"&gt;0.0&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;step&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="nb"&gt;range&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;10&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt; &lt;span class="c1"&gt;# step cap; see agent-loop article&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;t0&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;time&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;monotonic&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="c1"&gt;# Generation span — OTel-style gen_ai.client attributes.&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="k"&gt;with&lt;/span&gt; &lt;span class="n"&gt;langfuse&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;start_as_current_observation&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;as_type&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;#34;generation&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;name&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;#34;anthropic.messages.create&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;model&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;MODEL&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="nb"&gt;input&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="p"&gt;[{&lt;/span&gt;&lt;span class="s2"&gt;&amp;#34;role&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;m&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;&amp;#34;role&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt; &lt;span class="s2"&gt;&amp;#34;content&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;scrub&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nb"&gt;str&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;m&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;&amp;#34;content&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;]))}&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;m&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;messages&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;metadata&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="s2"&gt;&amp;#34;step&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;step&lt;/span&gt;&lt;span class="p"&gt;},&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;as&lt;/span&gt; &lt;span class="n"&gt;gen&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;resp&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;anthropic_client&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;messages&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;create&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;model&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;MODEL&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;max_tokens&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;1024&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;tools&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;tools&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;messages&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;messages&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="p"&gt;)&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;latency_ms&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nb"&gt;int&lt;/span&gt;&lt;span class="p"&gt;((&lt;/span&gt;&lt;span class="n"&gt;time&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;monotonic&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt; &lt;span class="n"&gt;t0&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="mi"&gt;1000&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="c1"&gt;# Token + cache attribution.&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;in_tok&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;resp&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;usage&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;input_tokens&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;out_tok&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;resp&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;usage&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;output_tokens&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;cache_read&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nb"&gt;getattr&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;resp&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;usage&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;&amp;#34;cache_read_input_tokens&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="ow"&gt;or&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;cache_create&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nb"&gt;getattr&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;resp&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;usage&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;&amp;#34;cache_creation_input_tokens&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="ow"&gt;or&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;cost&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;in_tok&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt; &lt;span class="mf"&gt;1e6&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="n"&gt;PRICING&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;MODEL&lt;/span&gt;&lt;span class="p"&gt;][&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;out_tok&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt; &lt;span class="mf"&gt;1e6&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="n"&gt;PRICING&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;MODEL&lt;/span&gt;&lt;span class="p"&gt;][&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;total_cost&lt;/span&gt; &lt;span class="o"&gt;+=&lt;/span&gt; &lt;span class="n"&gt;cost&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;gen&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;update&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;output&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;scrub&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nb"&gt;str&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;resp&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;content&lt;/span&gt;&lt;span class="p"&gt;)),&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;usage_details&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="s2"&gt;&amp;#34;input&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;in_tok&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;&amp;#34;output&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;out_tok&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="s2"&gt;&amp;#34;cache_read_input_tokens&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;cache_read&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="s2"&gt;&amp;#34;cache_creation_input_tokens&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;cache_create&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="p"&gt;},&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;cost_details&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="s2"&gt;&amp;#34;total&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;cost&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;&amp;#34;pricing_version&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;&amp;#34;2026-05&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;},&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;metadata&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="s2"&gt;&amp;#34;latency_ms&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;latency_ms&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;&amp;#34;finish_reason&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;resp&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;stop_reason&lt;/span&gt;&lt;span class="p"&gt;},&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="p"&gt;)&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="c1"&gt;# Append assistant turn.&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;messages&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;append&lt;/span&gt;&lt;span class="p"&gt;({&lt;/span&gt;&lt;span class="s2"&gt;&amp;#34;role&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;&amp;#34;assistant&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;&amp;#34;content&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;resp&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;content&lt;/span&gt;&lt;span class="p"&gt;})&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;resp&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;stop_reason&lt;/span&gt; &lt;span class="o"&gt;!=&lt;/span&gt; &lt;span class="s2"&gt;&amp;#34;tool_use&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;turn&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;update&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;output&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;scrub&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nb"&gt;str&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;resp&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;content&lt;/span&gt;&lt;span class="p"&gt;)),&lt;/span&gt; &lt;span class="n"&gt;metadata&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="s2"&gt;&amp;#34;total_cost_usd&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;total_cost&lt;/span&gt;&lt;span class="p"&gt;})&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;resp&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="c1"&gt;# Dispatch each tool_use block as its own span.&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;tool_results&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;[]&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;block&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;resp&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;content&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;block&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;type&lt;/span&gt; &lt;span class="o"&gt;!=&lt;/span&gt; &lt;span class="s2"&gt;&amp;#34;tool_use&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="k"&gt;continue&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="k"&gt;with&lt;/span&gt; &lt;span class="n"&gt;langfuse&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;start_as_current_observation&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;as_type&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;#34;span&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;name&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="sa"&gt;f&lt;/span&gt;&lt;span class="s2"&gt;&amp;#34;tool.&lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;block&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;name&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s2"&gt;&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="nb"&gt;input&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;block&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;input&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;as&lt;/span&gt; &lt;span class="n"&gt;tool_span&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;t1&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;time&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;monotonic&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="k"&gt;try&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;result&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;dispatch&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;block&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;name&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;block&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;input&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;is_error&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="kc"&gt;False&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="k"&gt;except&lt;/span&gt; &lt;span class="ne"&gt;Exception&lt;/span&gt; &lt;span class="k"&gt;as&lt;/span&gt; &lt;span class="n"&gt;e&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;result&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nb"&gt;str&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;e&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;is_error&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="kc"&gt;True&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;tool_span&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;update&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;output&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;result&lt;/span&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="ow"&gt;not&lt;/span&gt; &lt;span class="n"&gt;is_error&lt;/span&gt; &lt;span class="k"&gt;else&lt;/span&gt; &lt;span class="kc"&gt;None&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;metadata&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="s2"&gt;&amp;#34;latency_ms&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;int&lt;/span&gt;&lt;span class="p"&gt;((&lt;/span&gt;&lt;span class="n"&gt;time&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;monotonic&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt; &lt;span class="n"&gt;t1&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="mi"&gt;1000&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="s2"&gt;&amp;#34;is_error&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;is_error&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="p"&gt;},&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;level&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;#34;ERROR&amp;#34;&lt;/span&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;is_error&lt;/span&gt; &lt;span class="k"&gt;else&lt;/span&gt; &lt;span class="s2"&gt;&amp;#34;DEFAULT&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="p"&gt;)&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;tool_results&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;append&lt;/span&gt;&lt;span class="p"&gt;({&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="s2"&gt;&amp;#34;type&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;&amp;#34;tool_result&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;&amp;#34;tool_use_id&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;block&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;id&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="s2"&gt;&amp;#34;content&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;str&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;result&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt; &lt;span class="s2"&gt;&amp;#34;is_error&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;is_error&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="p"&gt;})&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;messages&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;append&lt;/span&gt;&lt;span class="p"&gt;({&lt;/span&gt;&lt;span class="s2"&gt;&amp;#34;role&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;&amp;#34;user&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;&amp;#34;content&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;tool_results&lt;/span&gt;&lt;span class="p"&gt;})&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;turn&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;update&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;metadata&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="s2"&gt;&amp;#34;budget_breach&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kc"&gt;True&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;&amp;#34;reason&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;&amp;#34;step_cap&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;},&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;level&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;#34;WARNING&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="p"&gt;)&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="c1"&gt;# Flush at process exit for short-lived applications.&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="nn"&gt;atexit&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt; &lt;span class="n"&gt;atexit&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;register&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;langfuse&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;flush&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/td&gt;&lt;/tr&gt;&lt;/table&gt;&#10;&lt;/div&gt;&#10;&lt;/div&gt;&lt;/div&gt;&#10;&lt;/figure&gt;&#10;&lt;p&gt;every span is opened with &lt;code&gt;start_as_current_observation&lt;/code&gt;, which uses Python&amp;rsquo;s context-manager protocol to ensure parents are set correctly even on exception paths; a custom span manager that forgets this drops the parent linkage on error, which is exactly when you need the trace. &lt;code&gt;usage_details&lt;/code&gt; carries the cache attribution as separate fields so the cache-hit-rate metric is computable per span without re-parsing the payload. &lt;code&gt;cost.pricing_version&lt;/code&gt; is pinned at the span level; when the provider changes pricing, historical traces still reconstruct the cost they incurred at the time, not the cost they would incur today.&lt;/p&gt;&#10;&lt;h2 id="further-reading"&gt;Further reading&lt;/h2&gt;&#10;&lt;ul&gt;&#10;&lt;li&gt;&lt;a href="https://opentelemetry.io/blog/2026/genai-observability/"&gt;OpenTelemetry; GenAI Observability blog&lt;/a&gt;; the official OTel GenAI SIG writeup of the semantic conventions, with the rationale for log-event-based payload attachment and the &lt;code&gt;gen_ai.client&lt;/code&gt; / &lt;code&gt;gen_ai.agent&lt;/code&gt; span types.&lt;/li&gt;&#10;&lt;li&gt;&lt;a href="https://opentelemetry.io/docs/specs/semconv/gen-ai/"&gt;OpenTelemetry; Generative AI Semantic Conventions spec&lt;/a&gt;; the canonical attribute reference. Stable vs experimental flags marked per attribute; the document any platform-agnostic instrumentation should be written against.&lt;/li&gt;&#10;&lt;li&gt;&lt;a href="https://github.com/Arize-ai/openinference/blob/main/spec/semantic_conventions.md"&gt;Arize; OpenInference Semantic Conventions&lt;/a&gt;; the parallel convention with span attributes for &lt;code&gt;llm.input_messages&lt;/code&gt;, &lt;code&gt;tool.parameters&lt;/code&gt;, &lt;code&gt;retrieval.documents&lt;/code&gt;, and the agent-specific extensions Phoenix renders natively.&lt;/li&gt;&#10;&lt;li&gt;&lt;a href="https://langfuse.com/integrations/native/opentelemetry"&gt;Langfuse; OpenTelemetry integration docs&lt;/a&gt;; the cleanest worked example of an OTel pipeline shipping to a real trace backend, with the Langfuse-specific extensions (cost attribution, dataset linkage) layered on top of the standard spans.&lt;/li&gt;&#10;&lt;li&gt;&lt;a href="https://hamel.dev/blog/posts/field-guide/"&gt;Hamel Husain; A Field Guide to Rapidly Improving AI Products&lt;/a&gt;; the practitioner&amp;rsquo;s tour of observability-first product work, with the case that &amp;ldquo;you cannot improve what you don&amp;rsquo;t observe&amp;rdquo; applies to LLM systems with even more force than it applies to web apps.&lt;/li&gt;&#10;&lt;li&gt;&lt;a href="https://eugeneyan.com/writing/llm-evaluators/"&gt;Eugene Yan; Evaluating the Effectiveness of LLM-Evaluators&lt;/a&gt;; the metric-side companion to traces. Eugene&amp;rsquo;s evaluators survey covers the judge-bias modes that make trace-attached judged scores trustworthy or not, and is the cleanest writeup of the cross-pillar discipline that ties metrics, logs, and traces together for LLM systems.&lt;/li&gt;&#10;&lt;/ul&gt;&#10;</content:encoded></item><item><title>LLM-as-Judge: Pointwise and Pairwise</title><link>https://jatinbansal.com/ai-engineering/llm-as-judge/</link><guid isPermaLink="true">https://jatinbansal.com/ai-engineering/llm-as-judge/</guid><pubDate>Tue, 19 May 2026 13:25:00 +0000</pubDate><dc:creator>Jatin Bansal</dc:creator><category>AI Engineering</category><description>How to design and calibrate pointwise and pairwise LLM judges against human labels.</description><content:encoded>&lt;p&gt;A team ships a rubric-based judge that grades helpfulness 1–5. The first week, every PR&amp;rsquo;s mean score lands between 3.6 and 3.9. The suite looks healthy, the gate is green, the model upgrade ships. Two weeks later support escalates a wave of complaints about the assistant being curt and unhelpful, and a closer look at the traces shows the upgraded model emits half the tokens for the same questions. The judge had been counting tokens as effort and effort as quality; the verbosity bias was scoring length, not helpfulness, and the new model lost that bias dividend. Nobody on the team had calibrated the judge against a human reviewer in three months. The judge had drifted into vanity, and the eval gate was meaningless before anyone noticed.&lt;/p&gt;&#10;&lt;h2 id="the-evaluation-contract"&gt;The evaluation contract&lt;/h2&gt;&#10;&lt;p&gt;LLM-as-judge uses one language model to score, rank, or critique another model&amp;rsquo;s output against a versioned rubric. Reliable use requires a fixed evaluation set, calibration against human labels, and explicit tests for position, verbosity, self-preference, and formatting bias.&lt;/p&gt;&#10;&lt;p&gt;The technique was canonized by &lt;a href="https://arxiv.org/abs/2306.05685"&gt;Zheng et al.&amp;rsquo;s &amp;ldquo;Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena&amp;rdquo;&lt;/a&gt;, which established that a strong judge model reaches ~85% agreement with human experts on MT-Bench using pairwise comparison; at parity with the ~81% human–human agreement on the same task. The paper also catalogued the bias modes that the rest of the field has been mitigating since. The complementary lineage is &lt;a href="https://arxiv.org/abs/2303.16634"&gt;G-Eval&lt;/a&gt;, which formalized chain-of-thought rubric grading with token-probability-weighted continuous scores and is the template most modern pointwise harnesses descend from.&lt;/p&gt;&#10;&lt;h2 id="pointwise-scoring"&gt;Pointwise scoring&lt;/h2&gt;&#10;&lt;p&gt;A pointwise judge scores one input/output pair against a rubric. Its prompt contains the rubric, input, candidate output, and a structured verdict format. Rubric design and calibration are harder than the call itself; broad properties such as &amp;ldquo;helpfulness&amp;rdquo; should be split into narrower criteria.&lt;/p&gt;&#10;&lt;p&gt;The discipline the field has converged on, traced back to &lt;a href="https://hamel.dev/blog/posts/llm-judge/"&gt;Hamel Husain&amp;rsquo;s work&lt;/a&gt; and the &lt;a href="https://arxiv.org/abs/2303.16634"&gt;G-Eval paper&lt;/a&gt;, looks like this:&lt;/p&gt;&#10;&lt;ol&gt;&#10;&lt;li&gt;Criterion-separated. One judge prompt per property; faithfulness, relevance, instruction-following, tone; not one prompt that scores all four. Each prompt&amp;rsquo;s rubric is two or three sentences naming the property and the failure modes.&lt;/li&gt;&#10;&lt;li&gt;Binary or low-cardinality: Pass/fail or 1–3 beats 1–5, which beats 1–10. A binary judge is easier to calibrate against a human reviewer because the confusion matrix has only two cells. Drawing on work across more than 30 companies, Hamel reports that domain-expert pass/fail labels correlate better with actual quality than granular scores and are harder to game with verbosity. Pass/fail loses ordinal information; use a 1–3 scale with anchored examples when near-threshold trends matter.&lt;/li&gt;&#10;&lt;li&gt;Anchored examples. The rubric includes one or two example outputs at each score level, drawn from your own traces. This is the equivalent of rater training in human evaluation: the judge needs to see what a &amp;ldquo;3&amp;rdquo; looks like in your domain, not in the abstract.&lt;/li&gt;&#10;&lt;li&gt;Chain-of-thought first, verdict last. The judge writes its reasoning before its verdict. G-Eval&amp;rsquo;s central trick is to ask the judge to enumerate the evaluation steps in CoT form before emitting the score; the score-after-reasoning ordering improves human correlation significantly compared to score-then-justify. Modern pointwise harnesses adopt this universally.&lt;/li&gt;&#10;&lt;li&gt;Token-probability scoring (optional). When the judge emits a single token score (e.g. &amp;ldquo;3&amp;rdquo;), the logits over the score tokens can be re-weighted to produce a continuous score: &lt;code&gt;score = Σ p(k) · k&lt;/code&gt; for k in the score range. G-Eval shows this reduces score quantization noise and improves correlation with human ratings. This requires logprobs access; supported in the OpenAI SDK directly and &lt;a href="https://github.com/anthropics/anthropic-sdk-python"&gt;now exposed as &lt;code&gt;logprobs=True&lt;/code&gt; in the Anthropic Messages API&lt;/a&gt;, and is most useful for trend-tracking, less so for hard gates.&lt;/li&gt;&#10;&lt;/ol&gt;&#10;&lt;p&gt;The pointwise judge&amp;rsquo;s failure mode is &lt;em&gt;scale drift&lt;/em&gt;: the same property scored across two judge upgrades is not on the same scale, and historical comparisons silently lie unless you pin the judge or recalibrate. The cheap mitigation is to pin the judge model in the eval config (&lt;code&gt;claude-haiku-4-5&lt;/code&gt; is fine for most rubrics) and treat a judge upgrade as a metric reset.&lt;/p&gt;&#10;&lt;h2 id="pairwise-comparison"&gt;Pairwise comparison&lt;/h2&gt;&#10;&lt;p&gt;The pairwise judge takes two candidate outputs A and B for the same input and emits &amp;ldquo;A wins,&amp;rdquo; &amp;ldquo;B wins,&amp;rdquo; or &amp;ldquo;tie.&amp;rdquo; The protocol comes directly from &lt;a href="https://arena.ai/"&gt;Chatbot Arena&lt;/a&gt;, where pairwise crowd preferences feed Elo ratings, and from &lt;a href="https://arxiv.org/abs/2306.05685"&gt;MT-Bench&lt;/a&gt;, where pairwise judge calls fed the agreement studies above. Pairwise is the right shape when you want to compare two systems against each other; model A vs model B, prompt v1 vs prompt v2, and when the property is too ambiguous for a stable absolute rubric.&lt;/p&gt;&#10;&lt;p&gt;Pairwise&amp;rsquo;s reliability advantage is real. The judge does not have to map an output onto an absolute scale; it only has to decide which of two outputs is better given the same context. That cancels out a lot of rubric-interpretation noise, which is why pairwise GPT-4 hits 85% human agreement on MT-Bench. But pairwise has two costs: &lt;em&gt;quadratic call count&lt;/em&gt;; comparing N systems requires N·(N-1)/2 pairs per query, and &lt;em&gt;position bias&lt;/em&gt;, which dominates the noise budget if untreated.&lt;/p&gt;&#10;&lt;p&gt;Position bias is the judge&amp;rsquo;s tendency to prefer the first option (or, depending on the model, the last) regardless of content. &lt;a href="https://arxiv.org/abs/2306.05685"&gt;Zheng et al.&lt;/a&gt; reported Claude-v1 with 70% one-sided preference and GPT-3.5 with 50% positional bias on MT-Bench; recent frontier-model studies show the effect persists, with &lt;a href="https://openreview.net/forum?id=y3jJmrKWQ4"&gt;position bias remaining a 40% inconsistency rate for GPT-4-class judges&lt;/a&gt; on rubric-based pairwise tasks. The mitigation is non-optional and adds a 2x cost: run each comparison twice with positions swapped, and only count a &amp;ldquo;win&amp;rdquo; when both orderings agree. Disagreements become ties. This is the canonical &amp;ldquo;both-orderings&amp;rdquo; protocol and is what every production pairwise pipeline implements.&lt;/p&gt;&#10;&lt;p&gt;The second pairwise gotcha is calibration of partial wins. Two orderings that agree on &amp;ldquo;A wins&amp;rdquo; are a clean win for A. Two orderings that flip; A wins in (A, B) and B wins in (B, A); are not really a tie; they&amp;rsquo;re a judge whose verdict was determined by position, not content. The instrumented protocol counts these separately as &amp;ldquo;position-flipped&amp;rdquo; cases and gates the eval&amp;rsquo;s reliability on the fraction below ~10%. If you see 30% position-flipped cases, the judge is too noisy for pairwise on this rubric and you either fall back to pointwise or switch judge models.&lt;/p&gt;&#10;&lt;h2 id="judge-biases"&gt;Judge biases&lt;/h2&gt;&#10;&lt;p&gt;Four biases appear in every untreated judge pipeline, and the discipline is to treat each one with an explicit mitigation rather than trust the judge&amp;rsquo;s calibration:&lt;/p&gt;&#10;&lt;ul&gt;&#10;&lt;li&gt;Position bias. Treated by both-orderings, above.&lt;/li&gt;&#10;&lt;li&gt;Verbosity bias. The judge prefers longer outputs even when the shorter one is more correct: length reads as effort, effort reads as quality. &lt;a href="https://eugeneyan.com/writing/llm-evaluators/"&gt;Eugene Yan&amp;rsquo;s survey&lt;/a&gt; reports Claude-v1 and GPT-3.5 preferring longer responses over 90% of the time on the verbosity probe. The mitigation is a rubric that explicitly rewards conciseness when the shorter response covers the same ground, plus length-normalized scoring on rubrics where length and quality are orthogonal. Don&amp;rsquo;t trust the judge to detect verbose padding on its own.&lt;/li&gt;&#10;&lt;li&gt;Self-preference bias. A judge from a model family scores outputs from its own family higher than outputs from other families. Zheng et al. measured GPT-4 with 10% self-enhancement and Claude-v1 with 25% self-preference. The mitigation is cross-family judging: use a judge from a different family than the systems being compared, or use a &lt;a href="https://arxiv.org/abs/2404.18796"&gt;panel of judges&lt;/a&gt; (more below).&lt;/li&gt;&#10;&lt;li&gt;Length / authority / format biases. Catch-all category: the judge prefers outputs that look authoritative (citations, hedged claims), follow a familiar format (bullet lists over prose, even when prose is better), or contain technical jargon. These biases are domain-specific; the mitigation is to detect them via the calibration loop: if the judge and a domain expert disagree systematically on a category of output, that&amp;rsquo;s a bias signal, and the rubric needs an explicit constraint to counteract it.&lt;/li&gt;&#10;&lt;/ul&gt;&#10;&lt;p&gt;These four are not a complete list; the &lt;a href="https://arxiv.org/html/2411.15594v6"&gt;Survey on LLM-as-a-Judge&lt;/a&gt; catalogues a dozen more. But the four above are the critical ones in practice; treat them and the rest is small noise on top.&lt;/p&gt;&#10;&lt;h2 id="panels-of-judges"&gt;Panels of judges&lt;/h2&gt;&#10;&lt;p&gt;A panel of smaller judges from disjoint model families outperforms a single large judge on most tasks, costs less, and has lower intra-model bias. &lt;a href="https://arxiv.org/abs/2404.18796"&gt;Verga et al. (&amp;ldquo;Replacing Judges with Juries&amp;rdquo;)&lt;/a&gt; measured a three-judge panel of smaller models (Command-R, Haiku, GPT-3.5) beating GPT-4 as a single judge on six datasets while costing 7x less. The mechanism is two-fold: cross-family panels neutralize each model&amp;rsquo;s self-preference, and averaging over independent estimators reduces the per-call variance the same way ensemble methods do everywhere else.&lt;/p&gt;&#10;&lt;p&gt;The trade-off is operational: more judge clients to manage, more credentials, more API surface to fail on. Production teams ship panels for the expensive nightly judged sweep where the variance reduction matters; the cheap per-PR layer often runs with a single pinned judge for simplicity. The cleanest panel is three judges from three families; pick one Anthropic, one OpenAI, one open-weights (e.g. a &lt;a href="https://www.llama.com/"&gt;Llama&lt;/a&gt; or &lt;a href="https://qwen.ai/"&gt;Qwen&lt;/a&gt; variant). Aggregate by majority vote for binary verdicts and by mean for continuous scores.&lt;/p&gt;&#10;&lt;h2 id="calibration-against-humans"&gt;Calibration against humans&lt;/h2&gt;&#10;&lt;p&gt;The judge has no meaning until you&amp;rsquo;ve measured its agreement with a domain expert on a held-out calibration set. The protocol is:&lt;/p&gt;&#10;&lt;ol&gt;&#10;&lt;li&gt;Sample 100–200 traces stratified by category from production or from the existing eval set.&lt;/li&gt;&#10;&lt;li&gt;Have one domain expert; not three, not a committee; score every sample on the rubric, using the same prompt the judge will see. The single-expert constraint is from Hamel&amp;rsquo;s work: averaging over multiple humans introduces variance that swamps the signal at small sample sizes.&lt;/li&gt;&#10;&lt;li&gt;Score the same samples with the candidate judge.&lt;/li&gt;&#10;&lt;li&gt;Compute Cohen&amp;rsquo;s kappa between expert and judge for categorical verdicts (or Spearman/Kendall correlation for continuous scores). Plain percentage agreement overstates: an 80% percentage agreement can correspond to a 0.62 kappa, which is fair but not strong (Eugene Yan reports exactly this gap on TriviaQA with Llama-3-8b as judge).&lt;/li&gt;&#10;&lt;li&gt;Iterate on the rubric until kappa exceeds ~0.7 on the calibration set, or; if the underlying property is genuinely ambiguous; accept that the judge has a floor and switch to a different property.&lt;/li&gt;&#10;&lt;/ol&gt;&#10;&lt;p&gt;Re-run the calibration on a fresh sample monthly, or after any judge-model upgrade, prompt change, or rubric edit. The judge&amp;rsquo;s agreement number is the metric that says the metric is trustworthy.&lt;/p&gt;&#10;&lt;h2 id="pointwise-judge-in-python"&gt;Pointwise judge in Python&lt;/h2&gt;&#10;&lt;p&gt;The Python harness below pins the judge model, runs the rubric in chain-of-thought mode, parses a structured verdict via &lt;a href="https://pydantic.dev/docs/"&gt;Pydantic&lt;/a&gt;, and exposes a pairwise shim that wraps two pointwise calls. Install: &lt;code&gt;pip install anthropic pydantic&lt;/code&gt;.&lt;/p&gt;&#10;&lt;figure class="nb-code"&gt;&#10; &lt;header class="nb-code-chrome"&gt;&#10; &lt;span class="nb-code-label"&gt;python&lt;/span&gt;&#10; &#10; &lt;/header&gt;&#10; &lt;div class="nb-code-body"&gt;&lt;div class="highlight"&gt;&lt;div class="chroma"&gt;&#10;&lt;table class="lntable"&gt;&lt;tr&gt;&lt;td class="lntd"&gt;&#10;&lt;pre tabindex="0" class="chroma"&gt;&lt;code&gt;&lt;span class="lnt"&gt; 1&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 2&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 3&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 4&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 5&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 6&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 7&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 8&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 9&#10;&lt;/span&gt;&lt;span class="lnt"&gt;10&#10;&lt;/span&gt;&lt;span class="lnt"&gt;11&#10;&lt;/span&gt;&lt;span class="lnt"&gt;12&#10;&lt;/span&gt;&lt;span class="lnt"&gt;13&#10;&lt;/span&gt;&lt;span class="lnt"&gt;14&#10;&lt;/span&gt;&lt;span class="lnt"&gt;15&#10;&lt;/span&gt;&lt;span class="lnt"&gt;16&#10;&lt;/span&gt;&lt;span class="lnt"&gt;17&#10;&lt;/span&gt;&lt;span class="lnt"&gt;18&#10;&lt;/span&gt;&lt;span class="lnt"&gt;19&#10;&lt;/span&gt;&lt;span class="lnt"&gt;20&#10;&lt;/span&gt;&lt;span class="lnt"&gt;21&#10;&lt;/span&gt;&lt;span class="lnt"&gt;22&#10;&lt;/span&gt;&lt;span class="lnt"&gt;23&#10;&lt;/span&gt;&lt;span class="lnt"&gt;24&#10;&lt;/span&gt;&lt;span class="lnt"&gt;25&#10;&lt;/span&gt;&lt;span class="lnt"&gt;26&#10;&lt;/span&gt;&lt;span class="lnt"&gt;27&#10;&lt;/span&gt;&lt;span class="lnt"&gt;28&#10;&lt;/span&gt;&lt;span class="lnt"&gt;29&#10;&lt;/span&gt;&lt;span class="lnt"&gt;30&#10;&lt;/span&gt;&lt;span class="lnt"&gt;31&#10;&lt;/span&gt;&lt;span class="lnt"&gt;32&#10;&lt;/span&gt;&lt;span class="lnt"&gt;33&#10;&lt;/span&gt;&lt;span class="lnt"&gt;34&#10;&lt;/span&gt;&lt;span class="lnt"&gt;35&#10;&lt;/span&gt;&lt;span class="lnt"&gt;36&#10;&lt;/span&gt;&lt;span class="lnt"&gt;37&#10;&lt;/span&gt;&lt;span class="lnt"&gt;38&#10;&lt;/span&gt;&lt;span class="lnt"&gt;39&#10;&lt;/span&gt;&lt;span class="lnt"&gt;40&#10;&lt;/span&gt;&lt;span class="lnt"&gt;41&#10;&lt;/span&gt;&lt;span class="lnt"&gt;42&#10;&lt;/span&gt;&lt;span class="lnt"&gt;43&#10;&lt;/span&gt;&lt;span class="lnt"&gt;44&#10;&lt;/span&gt;&lt;span class="lnt"&gt;45&#10;&lt;/span&gt;&lt;span class="lnt"&gt;46&#10;&lt;/span&gt;&lt;span class="lnt"&gt;47&#10;&lt;/span&gt;&lt;span class="lnt"&gt;48&#10;&lt;/span&gt;&lt;span class="lnt"&gt;49&#10;&lt;/span&gt;&lt;span class="lnt"&gt;50&#10;&lt;/span&gt;&lt;span class="lnt"&gt;51&#10;&lt;/span&gt;&lt;span class="lnt"&gt;52&#10;&lt;/span&gt;&lt;span class="lnt"&gt;53&#10;&lt;/span&gt;&lt;span class="lnt"&gt;54&#10;&lt;/span&gt;&lt;span class="lnt"&gt;55&#10;&lt;/span&gt;&lt;span class="lnt"&gt;56&#10;&lt;/span&gt;&lt;span class="lnt"&gt;57&#10;&lt;/span&gt;&lt;span class="lnt"&gt;58&#10;&lt;/span&gt;&lt;span class="lnt"&gt;59&#10;&lt;/span&gt;&lt;span class="lnt"&gt;60&#10;&lt;/span&gt;&lt;span class="lnt"&gt;61&#10;&lt;/span&gt;&lt;span class="lnt"&gt;62&#10;&lt;/span&gt;&lt;span class="lnt"&gt;63&#10;&lt;/span&gt;&lt;span class="lnt"&gt;64&#10;&lt;/span&gt;&lt;span class="lnt"&gt;65&#10;&lt;/span&gt;&lt;span class="lnt"&gt;66&#10;&lt;/span&gt;&lt;span class="lnt"&gt;67&#10;&lt;/span&gt;&lt;span class="lnt"&gt;68&#10;&lt;/span&gt;&lt;span class="lnt"&gt;69&#10;&lt;/span&gt;&lt;span class="lnt"&gt;70&#10;&lt;/span&gt;&lt;span class="lnt"&gt;71&#10;&lt;/span&gt;&lt;span class="lnt"&gt;72&#10;&lt;/span&gt;&lt;span class="lnt"&gt;73&#10;&lt;/span&gt;&lt;span class="lnt"&gt;74&#10;&lt;/span&gt;&lt;span class="lnt"&gt;75&#10;&lt;/span&gt;&lt;span class="lnt"&gt;76&#10;&lt;/span&gt;&lt;span class="lnt"&gt;77&#10;&lt;/span&gt;&lt;span class="lnt"&gt;78&#10;&lt;/span&gt;&lt;span class="lnt"&gt;79&#10;&lt;/span&gt;&lt;span class="lnt"&gt;80&#10;&lt;/span&gt;&lt;span class="lnt"&gt;81&#10;&lt;/span&gt;&lt;span class="lnt"&gt;82&#10;&lt;/span&gt;&lt;span class="lnt"&gt;83&#10;&lt;/span&gt;&lt;span class="lnt"&gt;84&#10;&lt;/span&gt;&lt;span class="lnt"&gt;85&#10;&lt;/span&gt;&lt;span class="lnt"&gt;86&#10;&lt;/span&gt;&lt;span class="lnt"&gt;87&#10;&lt;/span&gt;&lt;span class="lnt"&gt;88&#10;&lt;/span&gt;&lt;span class="lnt"&gt;89&#10;&lt;/span&gt;&lt;span class="lnt"&gt;90&#10;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/td&gt;&#10;&lt;td class="lntd"&gt;&#10;&lt;pre tabindex="0" class="chroma"&gt;&lt;code class="language-python" data-lang="python"&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="c1"&gt;# pip install anthropic pydantic&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="kn"&gt;from&lt;/span&gt; &lt;span class="nn"&gt;anthropic&lt;/span&gt; &lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;Anthropic&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="kn"&gt;from&lt;/span&gt; &lt;span class="nn"&gt;pydantic&lt;/span&gt; &lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;BaseModel&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;Field&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="nn"&gt;json&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="n"&gt;JUDGE_MODEL&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s2"&gt;&amp;#34;claude-haiku-4-5&amp;#34;&lt;/span&gt; &lt;span class="c1"&gt;# pinned; treat upgrade as metric reset&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="n"&gt;client&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;Anthropic&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="k"&gt;class&lt;/span&gt; &lt;span class="nc"&gt;Verdict&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;BaseModel&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;reasoning&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;str&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;Field&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="o"&gt;...&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;description&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;#34;Step-by-step rubric reasoning before the score.&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;score&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;int&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;Field&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="o"&gt;...&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;ge&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;le&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;3&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;description&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;#34;1=fail, 2=partial, 3=pass&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="n"&gt;POINTWISE_PROMPT&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s2"&gt;&amp;#34;&amp;#34;&amp;#34;&lt;/span&gt;&lt;span class="se"&gt;\&#10;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="s2"&gt;You are an evaluator. Score the response against the rubric below.&#10;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="s2"&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="s2"&gt;Rubric (faithfulness): the response makes only claims that are explicitly supported by the provided context. Score:&#10;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="s2"&gt;- 3 (pass): every claim supported, no extrapolation&#10;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="s2"&gt;- 2 (partial): one claim under-supported but the rest are clean&#10;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="s2"&gt;- 1 (fail): hallucinated or contradicted claims&#10;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="s2"&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="s2"&gt;Anchored examples:&#10;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="s2"&gt;- A 3 looks like: &amp;lt;one trace from your domain, redacted&amp;gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="s2"&gt;- A 1 looks like: &amp;lt;one trace, redacted&amp;gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="s2"&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="s2"&gt;Reason step by step first, then emit a JSON object: {&amp;#34;reasoning&amp;#34;: &amp;#34;...&amp;#34;, &amp;#34;score&amp;#34;: N}.&#10;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="s2"&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="s2"&gt;Context:&#10;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="si"&gt;{context}&lt;/span&gt;&lt;span class="s2"&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="s2"&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="s2"&gt;Response:&#10;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="si"&gt;{response}&lt;/span&gt;&lt;span class="s2"&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="s2"&gt;&amp;#34;&amp;#34;&amp;#34;&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;pointwise_judge&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;context&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;str&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;response&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;str&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;-&amp;gt;&lt;/span&gt; &lt;span class="n"&gt;Verdict&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;msg&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;client&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;messages&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;create&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;model&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;JUDGE_MODEL&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;max_tokens&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;512&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;messages&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="p"&gt;[{&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="s2"&gt;&amp;#34;role&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;&amp;#34;user&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="s2"&gt;&amp;#34;content&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;POINTWISE_PROMPT&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;format&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;context&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;context&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;response&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;response&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="p"&gt;}],&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="p"&gt;)&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;text&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;msg&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;content&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;text&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="c1"&gt;# Robust JSON extraction: find the last {...} block in the output.&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;start&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;text&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;rfind&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;&amp;#34;{&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;end&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;text&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;rfind&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;&amp;#34;}&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;Verdict&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;model_validate_json&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;text&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;start&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="n"&gt;end&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;])&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="c1"&gt;# Pairwise via two pointwise calls plus the both-orderings adapter.&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;pairwise_judge&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;context&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;str&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;response_a&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;str&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;response_b&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;str&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;-&amp;gt;&lt;/span&gt; &lt;span class="nb"&gt;str&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="s2"&gt;&amp;#34;&amp;#34;&amp;#34;Returns &amp;#39;A&amp;#39;, &amp;#39;B&amp;#39;, &amp;#39;tie&amp;#39;, or &amp;#39;position-flipped&amp;#39;.&amp;#34;&amp;#34;&amp;#34;&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;a_first&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;_compare&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;context&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;response_a&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;response_b&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="c1"&gt;# asks: which is better?&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;b_first&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;_compare&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;context&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;response_b&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;response_a&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;a_first&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="s2"&gt;&amp;#34;first&amp;#34;&lt;/span&gt; &lt;span class="ow"&gt;and&lt;/span&gt; &lt;span class="n"&gt;b_first&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="s2"&gt;&amp;#34;second&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="s2"&gt;&amp;#34;A&amp;#34;&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;a_first&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="s2"&gt;&amp;#34;second&amp;#34;&lt;/span&gt; &lt;span class="ow"&gt;and&lt;/span&gt; &lt;span class="n"&gt;b_first&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="s2"&gt;&amp;#34;first&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="s2"&gt;&amp;#34;B&amp;#34;&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;a_first&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="n"&gt;b_first&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="s2"&gt;&amp;#34;position-flipped&amp;#34;&lt;/span&gt; &lt;span class="c1"&gt;# judge prefers position, not content&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="s2"&gt;&amp;#34;tie&amp;#34;&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="n"&gt;PAIRWISE_PROMPT&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s2"&gt;&amp;#34;&amp;#34;&amp;#34;&lt;/span&gt;&lt;span class="se"&gt;\&#10;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="s2"&gt;You are an evaluator. Two responses to the same query are below. Decide which&#10;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="s2"&gt;better follows the rubric (faithfulness as above), or whether they&amp;#39;re equivalent.&#10;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="s2"&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="s2"&gt;Reason step by step first, then emit a JSON object: {&amp;#34;reasoning&amp;#34;: &amp;#34;...&amp;#34;, &amp;#34;verdict&amp;#34;: &amp;#34;first&amp;#34;|&amp;#34;second&amp;#34;|&amp;#34;tie&amp;#34;}.&#10;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="s2"&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="s2"&gt;Context:&#10;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="si"&gt;{context}&lt;/span&gt;&lt;span class="s2"&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="s2"&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="s2"&gt;Response 1:&#10;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="si"&gt;{first}&lt;/span&gt;&lt;span class="s2"&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="s2"&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="s2"&gt;Response 2:&#10;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="si"&gt;{second}&lt;/span&gt;&lt;span class="s2"&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="s2"&gt;&amp;#34;&amp;#34;&amp;#34;&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;_compare&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;context&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;str&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;first&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;str&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;second&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;str&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;-&amp;gt;&lt;/span&gt; &lt;span class="nb"&gt;str&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;msg&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;client&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;messages&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;create&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;model&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;JUDGE_MODEL&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;max_tokens&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;512&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;messages&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="p"&gt;[{&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="s2"&gt;&amp;#34;role&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;&amp;#34;user&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="s2"&gt;&amp;#34;content&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;PAIRWISE_PROMPT&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;format&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;context&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;context&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;first&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;first&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;second&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;second&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="p"&gt;}],&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="p"&gt;)&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;text&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;msg&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;content&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;text&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;start&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;text&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;rfind&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;&amp;#34;{&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;end&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;text&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;rfind&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;&amp;#34;}&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;json&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;loads&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;text&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;start&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="n"&gt;end&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;])[&lt;/span&gt;&lt;span class="s2"&gt;&amp;#34;verdict&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/td&gt;&lt;/tr&gt;&lt;/table&gt;&#10;&lt;/div&gt;&#10;&lt;/div&gt;&lt;/div&gt;&#10;&lt;/figure&gt;&#10;&lt;p&gt;The pointwise rubric evaluates one property per prompt, so four properties require four calls per row. The pairwise implementation records position-flipped verdicts separately from ties because they indicate judge noise. JSON parsing is permissive because judges can prepend prose; strict structured output is another valid implementation.&lt;/p&gt;&#10;&lt;h2 id="further-reading"&gt;Further reading&lt;/h2&gt;&#10;&lt;ul&gt;&#10;&lt;li&gt;&lt;a href="https://arxiv.org/abs/2306.05685"&gt;Zheng et al.: Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena&lt;/a&gt;. The foundational paper. 85% human–judge agreement, the position/verbosity/self-enhancement bias taxonomy, and the methodology every modern judge implementation traces back to.&lt;/li&gt;&#10;&lt;li&gt;&lt;a href="https://arxiv.org/abs/2303.16634"&gt;Liu et al.: G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment&lt;/a&gt;. The chain-of-thought rubric template plus the token-probability-weighted continuous scoring trick. The template under most modern pointwise harnesses.&lt;/li&gt;&#10;&lt;li&gt;&lt;a href="https://arxiv.org/abs/2404.18796"&gt;Verga et al.: Replacing Judges with Juries&lt;/a&gt;. The panel-of-judges paper. Three small cross-family judges beat one large judge at 7x lower cost, with lower self-preference bias.&lt;/li&gt;&#10;&lt;li&gt;&lt;a href="https://hamel.dev/blog/posts/llm-judge/"&gt;Hamel Husain: Using LLM-as-a-Judge For Evaluation&lt;/a&gt;. The practitioner&amp;rsquo;s guide. Binary verdicts over Likert scales, critique-shadowing, the calibration loop against a single principal domain expert.&lt;/li&gt;&#10;&lt;li&gt;&lt;a href="https://eugeneyan.com/writing/llm-evaluators/"&gt;Eugene Yan: Evaluating the Effectiveness of LLM-Evaluators&lt;/a&gt;; survey of two dozen papers on judge biases, finetuned evaluators (JudgeLM, Prometheus, PandaLM), and the Cohen&amp;rsquo;s kappa vs percentage agreement gap.&lt;/li&gt;&#10;&lt;li&gt;&lt;a href="https://arxiv.org/html/2411.15594v6"&gt;Gu et al.: A Survey on LLM-as-a-Judge&lt;/a&gt;; comprehensive 2024–2025 survey with the long catalogue of bias modes and mitigation strategies referenced above.&lt;/li&gt;&#10;&lt;/ul&gt;&#10;</content:encoded></item><item><title>Eval-Driven Development for LLM Systems</title><link>https://jatinbansal.com/ai-engineering/eval-driven-development/</link><guid isPermaLink="true">https://jatinbansal.com/ai-engineering/eval-driven-development/</guid><pubDate>Tue, 19 May 2026 13:11:00 +0000</pubDate><dc:creator>Jatin Bansal</dc:creator><category>AI Engineering</category><description>Evaluation-driven development with error analysis, golden sets, layered checks, and CI gates.</description><content:encoded>&lt;p&gt;LLM application regressions often appear as distribution shifts rather than deterministic failures. A model upgrade may alter tool calls, a prompt edit may break one query class, and a retriever change may shift the supplied context. A versioned eval suite makes these changes visible before promotion.&lt;/p&gt;&#10;&lt;h2 id="treat-evals-as-the-release-contract"&gt;Treat evals as the release contract&lt;/h2&gt;&#10;&lt;p&gt;Eval-driven development treats a versioned eval suite as the release contract for an LLM application. The suite grows from failures found in real traces because the input distribution is too broad to specify in advance. It measures quality together with cost and latency, so an expensive quality gain can still count as a regression. A deterministic slice runs on every pull request, while slower LLM-judged checks run nightly or before release. Both enforce explicit promotion thresholds.&lt;/p&gt;&#10;&lt;p&gt;A note on the name. &amp;ldquo;Eval-&lt;em&gt;driven&lt;/em&gt; development&amp;rdquo; gestures at TDD, and that gesture is partly misleading. &lt;a href="https://hamel.dev/blog/posts/evals-faq/"&gt;Hamel Husain and Shreya Shankar argue strongly&lt;/a&gt; that pure write-evals-first does not work for LLM systems, because the surface area is unbounded and you cannot enumerate failure modes from a spec sheet. The right loop is &lt;em&gt;error analysis first, evals second&lt;/em&gt;; look at traces, categorize what is actually breaking, then write an eval that catches the category. The TDD shape (write the failing test, fix the bug, commit both together) survives; the &amp;ldquo;imagine all the failures&amp;rdquo; shape does not. Call it eval-driven development as long as you remember which half of the analogy holds.&lt;/p&gt;&#10;&lt;h2 id="start-with-error-analysis"&gt;Start with error analysis&lt;/h2&gt;&#10;&lt;p&gt;The loop that produces a useful eval suite is concrete enough to write down. Five steps, repeated weekly during early development, monthly once the system stabilizes:&lt;/p&gt;&#10;&lt;ol&gt;&#10;&lt;li&gt;Capture traces: Log every model call with its inputs, retrieved context, output, latency, token counts, and cost. Sample randomly or stratify by query type. The &lt;a href="https://jatinbansal.com/ai-engineering/llm-observability/"&gt;production tracing layer&lt;/a&gt; covers span shape, OTel GenAI semantic conventions, sampling, and platform choices. Evals only require that the traces are queryable.&lt;/li&gt;&#10;&lt;li&gt;Open-coded review. A single domain expert (not three) reads 50–100 traces back-to-back and writes free-text notes about every problem they see. No category schema yet. This is the &amp;ldquo;journaling&amp;rdquo; pass &lt;a href="https://hamel.dev/blog/posts/evals-faq/why-is-error-analysis-so-important-in-llm-evals-and-how-is-it-performed.html"&gt;Hamel and Shreya borrow from qualitative research&lt;/a&gt;; the goal is to surface failure modes you didn&amp;rsquo;t know existed.&lt;/li&gt;&#10;&lt;li&gt;Axial coding. Cluster the open-coded notes into categories: &amp;ldquo;wrong product name,&amp;rdquo; &amp;ldquo;ignored the prior turn,&amp;rdquo; &amp;ldquo;output JSON missed a required field,&amp;rdquo; &amp;ldquo;took the long path through the tool call graph.&amp;rdquo; Most categories will be small; a few will dominate the volume. The categories are now your taxonomy.&lt;/li&gt;&#10;&lt;li&gt;Write the eval row. For each category, pick the cheapest assertion that catches it. Schema validation for the JSON case. Substring match for the product-name case. Tool-call-sequence assertion for the wrong-path case. LLM-as-judge with a specific rubric for the &amp;ldquo;ignored prior turn&amp;rdquo; case where no cheaper assertion exists. Add a row to the suite for each.&lt;/li&gt;&#10;&lt;li&gt;Wire to CI and watch the trend. The deterministic rows gate the next merge. The judged rows trend on the dashboard. When a new failure shows up in production, return to step 1 with that trace.&lt;/li&gt;&#10;&lt;/ol&gt;&#10;&lt;p&gt;The trap that kills internal evals is going straight from &amp;ldquo;we want quality&amp;rdquo; to &amp;ldquo;let&amp;rsquo;s write a faithfulness judge&amp;rdquo; without the open-coded pass. You end up with a metric that measures a generic property nobody on the team can characterize, scores in the 0.7–0.85 band on every change, and has no relationship to the bugs your users actually report. Error analysis grounds the eval in the system&amp;rsquo;s &lt;em&gt;actual&lt;/em&gt; failure distribution.&lt;/p&gt;&#10;&lt;h2 id="combine-golden-sets-with-live-evaluation"&gt;Combine golden sets with live evaluation&lt;/h2&gt;&#10;&lt;p&gt;The eval inputs come from two places, and both are necessary. The golden set is a frozen, curated list of inputs; typically 50–500; checked into version control, never modified except to add new rows or deprecate old ones. It&amp;rsquo;s the regression unit. Every change to the system is compared against the suite&amp;rsquo;s previous score on exactly the same inputs. Without this, you cannot tell whether a metric movement is system change or input change. The golden set is small enough to be hand-curated and stable enough to be a contract.&lt;/p&gt;&#10;&lt;p&gt;Live evals sample real production traffic, redact PII, and score the sample asynchronously. They catch &lt;a href="https://jatinbansal.com/ai-engineering/drift-detection/"&gt;distribution drift&lt;/a&gt;; the kinds of queries users started sending after Wednesday&amp;rsquo;s marketing push; that the golden set, frozen six months ago, doesn&amp;rsquo;t represent. The dashboard shows both: golden score (changes only when the system changes) and live score (changes when the system &lt;em&gt;or&lt;/em&gt; the input distribution changes). The gap between them is the distribution drift signal.&lt;/p&gt;&#10;&lt;p&gt;A practical rule for golden-set construction: every row should map to a specific category from your error analysis. A row that exists &amp;ldquo;for coverage&amp;rdquo; of some imagined edge case is a row you can&amp;rsquo;t interpret when it regresses. The NurtureBoss case study in &lt;a href="https://hamel.dev/blog/posts/evals/"&gt;Hamel&amp;rsquo;s evals piece&lt;/a&gt; is the canonical worked example; they grew their golden set by mining the cases their date-handling logic had previously failed on, not by brainstorming dates a user might type.&lt;/p&gt;&#10;&lt;h2 id="measure-cost-and-latency-with-quality"&gt;Measure cost and latency with quality&lt;/h2&gt;&#10;&lt;p&gt;A common failure mode is to track quality alone, leaving cost and latency in a separate dashboard nobody opens. The day you ship a model upgrade that improves faithfulness by three points and triples per-call cost, the quality-only dashboard cheers and the finance team panics. Treat each eval row as emitting three numbers; quality score, p50 token cost, p95 latency; and require the suite&amp;rsquo;s release-gating threshold to include all three. The right framing is the Pareto frontier: a change that moves quality up and cost down is unambiguous; a change that moves only one and worsens the other is a judgment call that has to happen on the PR, not after deploy. This is the same idea the &lt;a href="https://jatinbansal.com/ai-engineering/prompt-caching/"&gt;prompt-caching article&lt;/a&gt; develops for the inference layer and the &lt;a href="https://jatinbansal.com/ai-engineering/rag-evaluation/"&gt;RAG-evaluation article&lt;/a&gt; develops for retrieval pipelines.&lt;/p&gt;&#10;&lt;h2 id="code-a-minimal-python-eval-harness"&gt;Code: a minimal Python eval harness&lt;/h2&gt;&#10;&lt;p&gt;The simplest working harness is a couple hundred lines of Python that ingests a golden set, runs the system under test, evaluates each row with a mix of deterministic and judged checks, and writes a result row per commit SHA. Install: &lt;code&gt;pip install anthropic pydantic&lt;/code&gt; (and your test runner; pytest works fine).&lt;/p&gt;&#10;&lt;figure class="nb-code"&gt;&#10; &lt;header class="nb-code-chrome"&gt;&#10; &lt;span class="nb-code-label"&gt;python&lt;/span&gt;&#10; &#10; &lt;/header&gt;&#10; &lt;div class="nb-code-body"&gt;&lt;div class="highlight"&gt;&lt;div class="chroma"&gt;&#10;&lt;table class="lntable"&gt;&lt;tr&gt;&lt;td class="lntd"&gt;&#10;&lt;pre tabindex="0" class="chroma"&gt;&lt;code&gt;&lt;span class="lnt"&gt; 1&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 2&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 3&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 4&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 5&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 6&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 7&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 8&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 9&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 10&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 11&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 12&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 13&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 14&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 15&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 16&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 17&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 18&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 19&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 20&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 21&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 22&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 23&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 24&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 25&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 26&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 27&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 28&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 29&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 30&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 31&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 32&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 33&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 34&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 35&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 36&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 37&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 38&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 39&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 40&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 41&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 42&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 43&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 44&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 45&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 46&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 47&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 48&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 49&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 50&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 51&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 52&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 53&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 54&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 55&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 56&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 57&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 58&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 59&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 60&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 61&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 62&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 63&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 64&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 65&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 66&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 67&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 68&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 69&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 70&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 71&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 72&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 73&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 74&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 75&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 76&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 77&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 78&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 79&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 80&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 81&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 82&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 83&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 84&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 85&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 86&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 87&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 88&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 89&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 90&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 91&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 92&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 93&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 94&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 95&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 96&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 97&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 98&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 99&#10;&lt;/span&gt;&lt;span class="lnt"&gt;100&#10;&lt;/span&gt;&lt;span class="lnt"&gt;101&#10;&lt;/span&gt;&lt;span class="lnt"&gt;102&#10;&lt;/span&gt;&lt;span class="lnt"&gt;103&#10;&lt;/span&gt;&lt;span class="lnt"&gt;104&#10;&lt;/span&gt;&lt;span class="lnt"&gt;105&#10;&lt;/span&gt;&lt;span class="lnt"&gt;106&#10;&lt;/span&gt;&lt;span class="lnt"&gt;107&#10;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/td&gt;&#10;&lt;td class="lntd"&gt;&#10;&lt;pre tabindex="0" class="chroma"&gt;&lt;code class="language-python" data-lang="python"&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="c1"&gt;# pip install anthropic pydantic&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="nn"&gt;json&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="nn"&gt;time&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="kn"&gt;from&lt;/span&gt; &lt;span class="nn"&gt;dataclasses&lt;/span&gt; &lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;dataclass&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="kn"&gt;from&lt;/span&gt; &lt;span class="nn"&gt;typing&lt;/span&gt; &lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;Callable&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="kn"&gt;from&lt;/span&gt; &lt;span class="nn"&gt;anthropic&lt;/span&gt; &lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;Anthropic&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="kn"&gt;from&lt;/span&gt; &lt;span class="nn"&gt;pydantic&lt;/span&gt; &lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;BaseModel&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="n"&gt;client&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;Anthropic&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="nd"&gt;@dataclass&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="k"&gt;class&lt;/span&gt; &lt;span class="nc"&gt;EvalRow&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="nb"&gt;id&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;str&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;category&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;str&lt;/span&gt; &lt;span class="c1"&gt;# from error analysis&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="nb"&gt;input&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;dict&lt;/span&gt; &lt;span class="c1"&gt;# whatever shape the system takes&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;must_contain&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;list&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="nb"&gt;str&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="kc"&gt;None&lt;/span&gt; &lt;span class="c1"&gt;# deterministic substring check&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;must_not_contain&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;list&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="nb"&gt;str&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="kc"&gt;None&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;schema&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;type&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;BaseModel&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;|&lt;/span&gt; &lt;span class="kc"&gt;None&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="kc"&gt;None&lt;/span&gt; &lt;span class="c1"&gt;# deterministic schema check&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;judge_rubric&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;str&lt;/span&gt; &lt;span class="o"&gt;|&lt;/span&gt; &lt;span class="kc"&gt;None&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="kc"&gt;None&lt;/span&gt; &lt;span class="c1"&gt;# LLM-judged check&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;max_latency_ms&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;int&lt;/span&gt; &lt;span class="o"&gt;|&lt;/span&gt; &lt;span class="kc"&gt;None&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="kc"&gt;None&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;max_cost_usd&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;float&lt;/span&gt; &lt;span class="o"&gt;|&lt;/span&gt; &lt;span class="kc"&gt;None&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="kc"&gt;None&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="nd"&gt;@dataclass&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="k"&gt;class&lt;/span&gt; &lt;span class="nc"&gt;EvalResult&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;row_id&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;str&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;category&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;str&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;passed_deterministic&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;bool&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;judge_score&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;float&lt;/span&gt; &lt;span class="o"&gt;|&lt;/span&gt; &lt;span class="kc"&gt;None&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;latency_ms&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;int&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;cost_usd&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;float&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;error&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;str&lt;/span&gt; &lt;span class="o"&gt;|&lt;/span&gt; &lt;span class="kc"&gt;None&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="kc"&gt;None&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;llm_judge&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;output&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;str&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;rubric&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;str&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;-&amp;gt;&lt;/span&gt; &lt;span class="nb"&gt;float&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="s2"&gt;&amp;#34;&amp;#34;&amp;#34;1-5 score from a small judge model against a rubric.&amp;#34;&amp;#34;&amp;#34;&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;prompt&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="sa"&gt;f&lt;/span&gt;&lt;span class="s2"&gt;&amp;#34;Rubric:&lt;/span&gt;&lt;span class="se"&gt;\n&lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;rubric&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="se"&gt;\n\n&lt;/span&gt;&lt;span class="s2"&gt;&amp;#34;&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="sa"&gt;f&lt;/span&gt;&lt;span class="s2"&gt;&amp;#34;Output:&lt;/span&gt;&lt;span class="se"&gt;\n&lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;output&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="se"&gt;\n\n&lt;/span&gt;&lt;span class="s2"&gt;&amp;#34;&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="sa"&gt;f&lt;/span&gt;&lt;span class="s2"&gt;&amp;#34;Score this 1-5 against the rubric. Reply with only the integer.&amp;#34;&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="p"&gt;)&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;msg&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;client&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;messages&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;create&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;model&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;#34;claude-haiku-4-5&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;max_tokens&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;4&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;messages&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="p"&gt;[{&lt;/span&gt;&lt;span class="s2"&gt;&amp;#34;role&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;&amp;#34;user&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;&amp;#34;content&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;prompt&lt;/span&gt;&lt;span class="p"&gt;}],&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="p"&gt;)&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="k"&gt;try&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="nb"&gt;int&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;msg&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;content&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;text&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;strip&lt;/span&gt;&lt;span class="p"&gt;())&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt; &lt;span class="mf"&gt;5.0&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="k"&gt;except&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="ne"&gt;ValueError&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="ne"&gt;IndexError&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="nb"&gt;float&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;&amp;#34;nan&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;run_row&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;row&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;EvalRow&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;system_under_test&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;Callable&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;-&amp;gt;&lt;/span&gt; &lt;span class="n"&gt;EvalResult&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;t0&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;time&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;monotonic&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="k"&gt;try&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;out&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;system_under_test&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;row&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;input&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="k"&gt;except&lt;/span&gt; &lt;span class="ne"&gt;Exception&lt;/span&gt; &lt;span class="k"&gt;as&lt;/span&gt; &lt;span class="n"&gt;e&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;EvalResult&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;row&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;id&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;row&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;category&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="kc"&gt;False&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="kc"&gt;None&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mf"&gt;0.0&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nb"&gt;str&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;e&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;latency_ms&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nb"&gt;int&lt;/span&gt;&lt;span class="p"&gt;((&lt;/span&gt;&lt;span class="n"&gt;time&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;monotonic&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt; &lt;span class="n"&gt;t0&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="mi"&gt;1000&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="c1"&gt;# Deterministic checks&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;passed&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="kc"&gt;True&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;row&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;must_contain&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;passed&lt;/span&gt; &lt;span class="o"&gt;&amp;amp;=&lt;/span&gt; &lt;span class="nb"&gt;all&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;s&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;out&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;&amp;#34;text&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;s&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;row&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;must_contain&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;row&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;must_not_contain&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;passed&lt;/span&gt; &lt;span class="o"&gt;&amp;amp;=&lt;/span&gt; &lt;span class="nb"&gt;all&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;s&lt;/span&gt; &lt;span class="ow"&gt;not&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;out&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;&amp;#34;text&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;s&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;row&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;must_not_contain&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;row&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;schema&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="k"&gt;try&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;row&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;schema&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;model_validate_json&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;out&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;&amp;#34;text&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;])&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="k"&gt;except&lt;/span&gt; &lt;span class="ne"&gt;Exception&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;passed&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="kc"&gt;False&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;row&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;max_latency_ms&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;passed&lt;/span&gt; &lt;span class="o"&gt;&amp;amp;=&lt;/span&gt; &lt;span class="n"&gt;latency_ms&lt;/span&gt; &lt;span class="o"&gt;&amp;lt;=&lt;/span&gt; &lt;span class="n"&gt;row&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;max_latency_ms&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="c1"&gt;# Judged check (only if deterministic passed; cheap-first ladder)&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;judge_score&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="kc"&gt;None&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;passed&lt;/span&gt; &lt;span class="ow"&gt;and&lt;/span&gt; &lt;span class="n"&gt;row&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;judge_rubric&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;judge_score&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;llm_judge&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;out&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;&amp;#34;text&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt; &lt;span class="n"&gt;row&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;judge_rubric&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;EvalResult&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;row_id&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;row&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;id&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;category&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;row&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;category&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;passed_deterministic&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;passed&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;judge_score&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;judge_score&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;latency_ms&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;latency_ms&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;cost_usd&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;out&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;&amp;#34;cost_usd&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mf"&gt;0.0&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="p"&gt;)&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;run_suite&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;rows&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;list&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;EvalRow&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt; &lt;span class="n"&gt;system_under_test&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;Callable&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;-&amp;gt;&lt;/span&gt; &lt;span class="nb"&gt;dict&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;results&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;run_row&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;system_under_test&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;r&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;rows&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;by_cat&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;{}&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;r&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;results&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;by_cat&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;setdefault&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;category&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="p"&gt;[])&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;append&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="s2"&gt;&amp;#34;pass_rate&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;sum&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;passed_deterministic&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;r&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;results&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt; &lt;span class="nb"&gt;len&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;results&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="s2"&gt;&amp;#34;judge_mean&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="nb"&gt;sum&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;judge_score&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;r&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;results&lt;/span&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;judge_score&lt;/span&gt; &lt;span class="ow"&gt;is&lt;/span&gt; &lt;span class="ow"&gt;not&lt;/span&gt; &lt;span class="kc"&gt;None&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="o"&gt;/&lt;/span&gt; &lt;span class="nb"&gt;max&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nb"&gt;sum&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;r&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;results&lt;/span&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;judge_score&lt;/span&gt; &lt;span class="ow"&gt;is&lt;/span&gt; &lt;span class="ow"&gt;not&lt;/span&gt; &lt;span class="kc"&gt;None&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="p"&gt;),&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="s2"&gt;&amp;#34;p95_latency_ms&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;sorted&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;latency_ms&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;r&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;results&lt;/span&gt;&lt;span class="p"&gt;)[&lt;/span&gt;&lt;span class="nb"&gt;int&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nb"&gt;len&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;results&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="mf"&gt;0.95&lt;/span&gt;&lt;span class="p"&gt;)],&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="s2"&gt;&amp;#34;total_cost_usd&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;sum&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;cost_usd&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;r&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;results&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="s2"&gt;&amp;#34;by_category&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;cat&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="s2"&gt;&amp;#34;pass_rate&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;sum&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;passed_deterministic&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;r&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;rs&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt; &lt;span class="nb"&gt;len&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;rs&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="s2"&gt;&amp;#34;n&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;len&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;rs&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="p"&gt;}&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;cat&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;rs&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;by_cat&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;items&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="p"&gt;},&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="s2"&gt;&amp;#34;results&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;results&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="p"&gt;}&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/td&gt;&lt;/tr&gt;&lt;/table&gt;&#10;&lt;/div&gt;&#10;&lt;/div&gt;&lt;/div&gt;&#10;&lt;/figure&gt;&#10;&lt;p&gt;The harness is deliberately minimal so the shape stays visible. Three things to flag. The cheap-first ladder; judged checks only run when deterministic checks pass; a row that fails schema validation doesn&amp;rsquo;t pay for a judge call. per-category breakdown; the aggregate is for the dashboard, the per-category numbers are for action. The cost and latency rollups are first-class outputs, not afterthoughts. A CI script wraps this with a pass/fail decision: &lt;code&gt;result[&amp;quot;pass_rate&amp;quot;] &amp;gt;= 0.95 and result[&amp;quot;judge_mean&amp;quot;] &amp;gt;= 0.80 and result[&amp;quot;p95_latency_ms&amp;quot;] &amp;lt;= 3000&lt;/code&gt;.&lt;/p&gt;&#10;&lt;p&gt;A worked row that exercises every assertion type:&lt;/p&gt;&#10;&lt;figure class="nb-code"&gt;&#10; &lt;header class="nb-code-chrome"&gt;&#10; &lt;span class="nb-code-label"&gt;python&lt;/span&gt;&#10; &#10; &lt;/header&gt;&#10; &lt;div class="nb-code-body"&gt;&lt;div class="highlight"&gt;&lt;div class="chroma"&gt;&#10;&lt;table class="lntable"&gt;&lt;tr&gt;&lt;td class="lntd"&gt;&#10;&lt;pre tabindex="0" class="chroma"&gt;&lt;code&gt;&lt;span class="lnt"&gt; 1&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 2&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 3&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 4&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 5&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 6&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 7&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 8&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 9&#10;&lt;/span&gt;&lt;span class="lnt"&gt;10&#10;&lt;/span&gt;&lt;span class="lnt"&gt;11&#10;&lt;/span&gt;&lt;span class="lnt"&gt;12&#10;&lt;/span&gt;&lt;span class="lnt"&gt;13&#10;&lt;/span&gt;&lt;span class="lnt"&gt;14&#10;&lt;/span&gt;&lt;span class="lnt"&gt;15&#10;&lt;/span&gt;&lt;span class="lnt"&gt;16&#10;&lt;/span&gt;&lt;span class="lnt"&gt;17&#10;&lt;/span&gt;&lt;span class="lnt"&gt;18&#10;&lt;/span&gt;&lt;span class="lnt"&gt;19&#10;&lt;/span&gt;&lt;span class="lnt"&gt;20&#10;&lt;/span&gt;&lt;span class="lnt"&gt;21&#10;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/td&gt;&#10;&lt;td class="lntd"&gt;&#10;&lt;pre tabindex="0" class="chroma"&gt;&lt;code class="language-python" data-lang="python"&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="k"&gt;class&lt;/span&gt; &lt;span class="nc"&gt;OrderResponse&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;BaseModel&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;order_id&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;str&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;status&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;str&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;items&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;list&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="nb"&gt;dict&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="n"&gt;ROWS&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;EvalRow&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="nb"&gt;id&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;#34;order_status_basic&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;category&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;#34;order_status_intent&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="nb"&gt;input&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="s2"&gt;&amp;#34;prompt&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;&amp;#34;What&amp;#39;s the status of order 12345?&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;},&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;must_contain&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;&amp;#34;12345&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;must_not_contain&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;&amp;#34;I don&amp;#39;t know&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;&amp;#34;as an AI&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;schema&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;OrderResponse&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;judge_rubric&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="s2"&gt;&amp;#34;The response answers the user&amp;#39;s order status question, &amp;#34;&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="s2"&gt;&amp;#34;stays factual to the tool output, and is under 2 sentences.&amp;#34;&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="p"&gt;),&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;max_latency_ms&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;4000&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="p"&gt;),&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="c1"&gt;# ...49 more rows, one per category × variant from error analysis&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/td&gt;&lt;/tr&gt;&lt;/table&gt;&#10;&lt;/div&gt;&#10;&lt;/div&gt;&lt;/div&gt;&#10;&lt;/figure&gt;&#10;&lt;h2 id="further-reading"&gt;Further reading&lt;/h2&gt;&#10;&lt;ul&gt;&#10;&lt;li&gt;&lt;a href="https://hamel.dev/blog/posts/evals/"&gt;Hamel Husain, Your AI Product Needs Evals&lt;/a&gt;; an error-analysis-first approach with the NurtureBoss case study and LLM-as-judge guidance.&lt;/li&gt;&#10;&lt;li&gt;&lt;a href="https://hamel.dev/blog/posts/evals-faq/"&gt;Hamel Husain &amp;amp; Shreya Shankar; LLM Evals FAQ&lt;/a&gt;; the long-form Q&amp;amp;A version, updated through 2026, with extended treatment of error analysis, judge alignment, and why &amp;ldquo;write evals first&amp;rdquo; is the wrong instinct.&lt;/li&gt;&#10;&lt;li&gt;&lt;a href="https://eugeneyan.com/writing/evals/"&gt;Eugene Yan; Task-Specific LLM Evals That Do (and Don&amp;rsquo;t) Work&lt;/a&gt;; the practitioner&amp;rsquo;s take on which evals are worth building and which are vanity. Pairs well with Eugene&amp;rsquo;s broader &lt;a href="https://eugeneyan.com/writing/llm-patterns/"&gt;LLM patterns piece&lt;/a&gt; on where evals fit in the larger stack.&lt;/li&gt;&#10;&lt;li&gt;&lt;a href="https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents"&gt;Anthropic; Demystifying Evals for AI Agents&lt;/a&gt;; Anthropic&amp;rsquo;s engineering blog walks through eval design for agent systems specifically, with worked examples from their own product surface.&lt;/li&gt;&#10;&lt;li&gt;&lt;a href="https://arxiv.org/abs/2404.12272"&gt;Shankar et al.; Who Validates the Validators?&lt;/a&gt;; the academic paper that introduced &lt;em&gt;evaluation criteria drift&lt;/em&gt; and the iterative-rubric methodology that underpins much of the current practice.&lt;/li&gt;&#10;&lt;/ul&gt;&#10;</content:encoded></item><item><title>Production Memory Frameworks: MemGPT/Letta, mem0, Zep, Graphiti</title><link>https://jatinbansal.com/ai-engineering/production-memory-frameworks/</link><guid isPermaLink="true">https://jatinbansal.com/ai-engineering/production-memory-frameworks/</guid><pubDate>Tue, 19 May 2026 03:27:25 +0000</pubDate><dc:creator>Jatin Bansal</dc:creator><category>AI Engineering</category><description>How Letta, mem0, Zep, and Graphiti differ in storage, retrieval, and operational cost.</description><content:encoded>&lt;p&gt;Production memory requires more than a vector store: write gates, episode formats, retrieval policy, tenant isolation, and maintenance jobs all need an owner. Letta, mem0, Zep, and Graphiti package different subsets of that work and impose different storage and retrieval contracts.&lt;/p&gt;&#10;&lt;h2 id="what-a-memory-framework-owns"&gt;What a memory framework owns&lt;/h2&gt;&#10;&lt;p&gt;A production memory framework is a runtime that bundles the write pipeline, storage substrate(s), read pipeline, multi-tenancy primitives, and maintenance passes into a single SDK. A &lt;em&gt;substrate&lt;/em&gt; (pgvector, Qdrant) is unopinionated; a framework picks an episode shape, a write gate, a retrieval blend, a tier policy, and a tenant model, then exposes them as a coherent &lt;code&gt;add&lt;/code&gt;/&lt;code&gt;search&lt;/code&gt;/&lt;code&gt;update&lt;/code&gt;/&lt;code&gt;delete&lt;/code&gt; API. Adopting a framework is buying its opinions.&lt;/p&gt;&#10;&lt;p&gt;Four define the field in 2026. &lt;a href="https://docs.letta.com/concepts/letta/"&gt;MemGPT/Letta&lt;/a&gt; is the productized version of the &lt;a href="https://arxiv.org/abs/2310.08560"&gt;original MemGPT paper&lt;/a&gt;; three-tier hierarchical memory (core/recall/archival) with the agent self-managing tier promotion via tool calls. &lt;a href="https://docs.mem0.ai/"&gt;mem0&lt;/a&gt; is the distill-at-write vector layer with an optional &lt;a href="https://docs.mem0.ai/open-source/graph_memory/overview"&gt;graph extension (Mem0g)&lt;/a&gt;; the LLM-gated fact-extraction pipeline runs on every &lt;code&gt;add&lt;/code&gt;. &lt;a href="https://www.getzep.com/"&gt;Zep&lt;/a&gt; is the graph-first hybrid: a bi-temporal knowledge graph wraps vector and BM25 indexes, all retrieval fused, no LLM in the read path. &lt;a href="https://help.getzep.com/graphiti/getting-started/welcome"&gt;Graphiti&lt;/a&gt; is Zep&amp;rsquo;s open-source temporal-graph engine, usable standalone when you want the bi-temporal substrate without the cloud product on top.&lt;/p&gt;&#10;&lt;h2 id="compare-the-operating-models"&gt;Compare the operating models&lt;/h2&gt;&#10;&lt;table&gt;&#10;&#9;&lt;thead&gt;&#10;&#9;&#9;&#9;&lt;tr&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;th&gt;Dimension&lt;/th&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;th&gt;Letta (MemGPT)&lt;/th&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;th&gt;mem0&lt;/th&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;th&gt;Zep&lt;/th&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;th&gt;Graphiti&lt;/th&gt;&#10;&#9;&#9;&#9;&lt;/tr&gt;&#10;&#9;&lt;/thead&gt;&#10;&#9;&lt;tbody&gt;&#10;&#9;&#9;&#9;&lt;tr&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;td&gt;Primary substrate&lt;/td&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;td&gt;Hierarchical (core/recall/archival)&lt;/td&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;td&gt;Vector + optional graph&lt;/td&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;td&gt;Graph + vector + BM25&lt;/td&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;td&gt;Bi-temporal graph&lt;/td&gt;&#10;&#9;&#9;&#9;&lt;/tr&gt;&#10;&#9;&#9;&#9;&lt;tr&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;td&gt;Write path cost&lt;/td&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;td&gt;Low (DB write + tool call)&lt;/td&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;td&gt;High (LLM fact extraction per turn)&lt;/td&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;td&gt;Very high (entity + relation extraction + bi-temporal stamping)&lt;/td&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;td&gt;Very high (same as Zep)&lt;/td&gt;&#10;&#9;&#9;&#9;&lt;/tr&gt;&#10;&#9;&#9;&#9;&lt;tr&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;td&gt;Read path cost&lt;/td&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;td&gt;Low (tool calls, no LLM in core)&lt;/td&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;td&gt;Low (vector search + optional graph traversal)&lt;/td&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;td&gt;Low (pure traversal + RRF, no LLM)&lt;/td&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;td&gt;Low (pure traversal + RRF)&lt;/td&gt;&#10;&#9;&#9;&#9;&lt;/tr&gt;&#10;&#9;&#9;&#9;&lt;tr&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;td&gt;Bi-temporal&lt;/td&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;td&gt;No (single transaction clock)&lt;/td&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;td&gt;No&lt;/td&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;td&gt;Yes (valid + transaction time)&lt;/td&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;td&gt;Yes&lt;/td&gt;&#10;&#9;&#9;&#9;&lt;/tr&gt;&#10;&#9;&#9;&#9;&lt;tr&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;td&gt;Self-managed by agent&lt;/td&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;td&gt;Yes (agent calls tier-promotion tools)&lt;/td&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;td&gt;No (harness-driven)&lt;/td&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;td&gt;No (harness-driven)&lt;/td&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;td&gt;No&lt;/td&gt;&#10;&#9;&#9;&#9;&lt;/tr&gt;&#10;&#9;&#9;&#9;&lt;tr&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;td&gt;Multi-tenancy&lt;/td&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;td&gt;Per-agent state (built-in)&lt;/td&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;td&gt;&lt;code&gt;user_id&lt;/code&gt; required parameter&lt;/td&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;td&gt;&lt;code&gt;user_id&lt;/code&gt; / &lt;code&gt;session_id&lt;/code&gt; (built-in)&lt;/td&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;td&gt;&lt;code&gt;group_id&lt;/code&gt; namespace&lt;/td&gt;&#10;&#9;&#9;&#9;&lt;/tr&gt;&#10;&#9;&#9;&#9;&lt;tr&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;td&gt;Hosted option&lt;/td&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;td&gt;Letta Cloud + self-host&lt;/td&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;td&gt;mem0 Cloud + open-source&lt;/td&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;td&gt;Zep Cloud + open-source community edition&lt;/td&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;td&gt;Self-host only&lt;/td&gt;&#10;&#9;&#9;&#9;&lt;/tr&gt;&#10;&#9;&#9;&#9;&lt;tr&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;td&gt;2026 benchmark anchor&lt;/td&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;td&gt;~83% LongMemEval (community report)&lt;/td&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;td&gt;94.4% LongMemEval (token-efficient algorithm, &lt;a href="https://mem0.ai/research"&gt;mem0.ai/research&lt;/a&gt;)&lt;/td&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;td&gt;71.2% LongMemEval (&lt;a href="https://arxiv.org/abs/2501.13956"&gt;Zep paper&lt;/a&gt;)&lt;/td&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;td&gt;Same substrate as Zep&lt;/td&gt;&#10;&#9;&#9;&#9;&lt;/tr&gt;&#10;&#9;&#9;&#9;&lt;tr&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;td&gt;Best-fit workload&lt;/td&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;td&gt;Long-running stateful agents where the agent itself manages context&lt;/td&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;td&gt;Chatbots and assistants with high user-fact density&lt;/td&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;td&gt;CRM, compliance, healthcare; relational + temporal queries dominate&lt;/td&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;td&gt;Greenfield graph-first builds, self-hosted&lt;/td&gt;&#10;&#9;&#9;&#9;&lt;/tr&gt;&#10;&#9;&#9;&#9;&lt;tr&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;td&gt;Worst-fit workload&lt;/td&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;td&gt;Stateless or short-lived agents&lt;/td&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;td&gt;Workloads where raw episodes matter more than distilled facts&lt;/td&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;td&gt;Workloads with no relational structure&lt;/td&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;td&gt;Same as Zep, plus teams who want a managed product&lt;/td&gt;&#10;&#9;&#9;&#9;&lt;/tr&gt;&#10;&#9;&lt;/tbody&gt;&#10;&lt;/table&gt;&#10;&lt;p&gt;The benchmark numbers in that row are the most volatile entry in the table. Mem0&amp;rsquo;s LoCoMo score went from 66.9% in 2025 to &lt;a href="https://mem0.ai/blog/state-of-ai-agent-memory-2026"&gt;92.5% in 2026&lt;/a&gt;; partly genuine algorithm improvement, partly protocol stabilization, partly hill-climbing. Read protocols (judge model, ingest pipeline, top-K) before comparing across rows. The &lt;a href="https://jatinbansal.com/ai-engineering/memory-evaluation/"&gt;memory evaluation article&lt;/a&gt; is the detailed treatment on why direct cross-framework comparison is harder than it looks.&lt;/p&gt;&#10;&lt;h2 id="decide-what-to-build"&gt;Decide what to build&lt;/h2&gt;&#10;&lt;p&gt;Reach for a framework when (a) your team has fewer than two engineers who can own a memory subsystem long-term, (b) your workload fits within ±20% of one of the four frameworks&amp;rsquo; opinions, and (c) you don&amp;rsquo;t have an existing storage layer the framework would fight. Hand-roll when (a) you have those engineers, (b) your defining write or read pattern isn&amp;rsquo;t covered (per-document write policies for a legal-research agent, custom segmentation for a code-review agent, or a &lt;a href="https://jatinbansal.com/ai-engineering/knowledge-graphs-memory/"&gt;graph schema&lt;/a&gt; that doesn&amp;rsquo;t fit Graphiti&amp;rsquo;s entity-relation model), or (c) you already operate a vector store and a graph store and the framework&amp;rsquo;s opinions about both fight your data model.&lt;/p&gt;&#10;&lt;p&gt;The most common mistake: adopting a framework, then writing so much code around it to make it fit that you would have been better off rolling your own. mem0 and Letta both have escape hatches; custom prompts, overrideable extraction, custom tools; but every escape hatch is a place the next breaking change will land. If you find yourself overriding more than two defaults, the framework is wrong for your workload.&lt;/p&gt;&#10;&lt;p&gt;The escape from the binary: roll your own on a store primitive and adopt a framework only for the layer where its opinions are critical. &lt;a href="https://docs.langchain.com/oss/python/langgraph/persistence"&gt;LangGraph stores&lt;/a&gt; give you a tuple-namespaced KV/vector store; you build the write policy, retrieval blend, and tier topology yourself, and adopt Graphiti only for the graph layer if your workload needs bi-temporal queries. Most teams converge on this after a year.&lt;/p&gt;&#10;&lt;h2 id="letta-integration-in-python"&gt;Letta integration in Python&lt;/h2&gt;&#10;&lt;p&gt;The Letta integration pattern is &lt;em&gt;the framework owns the agent state, the application owns the message routing&lt;/em&gt;. You call the SDK with messages; Letta manages the memory blocks, the recall/archival tiers, and the persistence behind a &lt;code&gt;client.agents.create&lt;/code&gt; / &lt;code&gt;client.agents.messages.create&lt;/code&gt; surface. Install: &lt;code&gt;pip install letta-client&lt;/code&gt; and run a local server (&lt;code&gt;docker run -d -p 8283:8283 letta/letta:latest&lt;/code&gt;) or use Letta Cloud.&lt;/p&gt;&#10;&lt;figure class="nb-code"&gt;&#10; &lt;header class="nb-code-chrome"&gt;&#10; &lt;span class="nb-code-label"&gt;python&lt;/span&gt;&#10; &#10; &lt;/header&gt;&#10; &lt;div class="nb-code-body"&gt;&lt;div class="highlight"&gt;&lt;div class="chroma"&gt;&#10;&lt;table class="lntable"&gt;&lt;tr&gt;&lt;td class="lntd"&gt;&#10;&lt;pre tabindex="0" class="chroma"&gt;&lt;code&gt;&lt;span class="lnt"&gt; 1&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 2&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 3&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 4&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 5&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 6&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 7&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 8&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 9&#10;&lt;/span&gt;&lt;span class="lnt"&gt;10&#10;&lt;/span&gt;&lt;span class="lnt"&gt;11&#10;&lt;/span&gt;&lt;span class="lnt"&gt;12&#10;&lt;/span&gt;&lt;span class="lnt"&gt;13&#10;&lt;/span&gt;&lt;span class="lnt"&gt;14&#10;&lt;/span&gt;&lt;span class="lnt"&gt;15&#10;&lt;/span&gt;&lt;span class="lnt"&gt;16&#10;&lt;/span&gt;&lt;span class="lnt"&gt;17&#10;&lt;/span&gt;&lt;span class="lnt"&gt;18&#10;&lt;/span&gt;&lt;span class="lnt"&gt;19&#10;&lt;/span&gt;&lt;span class="lnt"&gt;20&#10;&lt;/span&gt;&lt;span class="lnt"&gt;21&#10;&lt;/span&gt;&lt;span class="lnt"&gt;22&#10;&lt;/span&gt;&lt;span class="lnt"&gt;23&#10;&lt;/span&gt;&lt;span class="lnt"&gt;24&#10;&lt;/span&gt;&lt;span class="lnt"&gt;25&#10;&lt;/span&gt;&lt;span class="lnt"&gt;26&#10;&lt;/span&gt;&lt;span class="lnt"&gt;27&#10;&lt;/span&gt;&lt;span class="lnt"&gt;28&#10;&lt;/span&gt;&lt;span class="lnt"&gt;29&#10;&lt;/span&gt;&lt;span class="lnt"&gt;30&#10;&lt;/span&gt;&lt;span class="lnt"&gt;31&#10;&lt;/span&gt;&lt;span class="lnt"&gt;32&#10;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/td&gt;&#10;&lt;td class="lntd"&gt;&#10;&lt;pre tabindex="0" class="chroma"&gt;&lt;code class="language-python" data-lang="python"&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="c1"&gt;# pip install letta-client&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="kn"&gt;from&lt;/span&gt; &lt;span class="nn"&gt;letta_client&lt;/span&gt; &lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;Letta&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="n"&gt;client&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;Letta&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;base_url&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;#34;http://localhost:8283&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="c1"&gt;# Create an agent with hierarchical memory: core memory blocks are always in context,&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="c1"&gt;# recall and archival tiers live outside the prompt and the agent pages them in via tools.&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="n"&gt;agent&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;client&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;agents&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;create&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;model&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;#34;openai/gpt-4o-mini&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;embedding&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;#34;openai/text-embedding-3-small&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;memory_blocks&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="s2"&gt;&amp;#34;label&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;&amp;#34;persona&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;&amp;#34;value&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;&amp;#34;You are a customer-support agent for Acme.&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="s2"&gt;&amp;#34;limit&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;1000&lt;/span&gt;&lt;span class="p"&gt;},&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="s2"&gt;&amp;#34;label&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;&amp;#34;human&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;&amp;#34;value&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;&amp;#34;Unknown user.&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;&amp;#34;limit&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;2000&lt;/span&gt;&lt;span class="p"&gt;},&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="p"&gt;],&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;tools&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;&amp;#34;web_search&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt; &lt;span class="c1"&gt;# core_memory_append, recall_search, archival_search are auto-injected&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="p"&gt;)&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="c1"&gt;# Each call returns the full step trace; Letta has already updated memory blocks,&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="c1"&gt;# inserted into recall/archival, and persisted state. No explicit write call needed.&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="n"&gt;reply&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;client&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;agents&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;messages&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;create&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;agent_id&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;agent&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;id&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;messages&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="p"&gt;[{&lt;/span&gt;&lt;span class="s2"&gt;&amp;#34;role&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;&amp;#34;user&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;&amp;#34;content&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;&amp;#34;I&amp;#39;m Priya, my order #4321 hasn&amp;#39;t shipped.&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;}],&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="p"&gt;)&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="n"&gt;assistant_msg&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nb"&gt;next&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;m&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;m&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="nb"&gt;reversed&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;reply&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;messages&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;m&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;message_type&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="s2"&gt;&amp;#34;assistant_message&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="nb"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;assistant_msg&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;content&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="c1"&gt;# Inspect the updated memory blocks — the agent should have written &amp;#34;Priya&amp;#34; and &amp;#34;#4321&amp;#34;&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="c1"&gt;# into the human block via core_memory_append on its own.&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="n"&gt;state&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;client&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;agents&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;retrieve&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;agent_id&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;agent&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;id&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;block&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;state&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;memory&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;blocks&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="nb"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;block&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;label&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;&amp;#34;→&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;block&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;value&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/td&gt;&lt;/tr&gt;&lt;/table&gt;&#10;&lt;/div&gt;&#10;&lt;/div&gt;&lt;/div&gt;&#10;&lt;/figure&gt;&#10;&lt;p&gt;The key property: there is no explicit &lt;code&gt;memory.add()&lt;/code&gt; call. The agent decides what to remember, and the framework records what it decided. This is Letta&amp;rsquo;s central opinion; &lt;em&gt;agent-driven memory management&lt;/em&gt;; and it is either exactly right (long-running stateful agents that learn from their interactions) or exactly wrong (workflows where the harness, not the agent, owns the write policy).&lt;/p&gt;&#10;&lt;h2 id="mem0-integration-in-typescript"&gt;mem0 integration in TypeScript&lt;/h2&gt;&#10;&lt;p&gt;The mem0 integration pattern is the inverse: &lt;em&gt;the application owns the message routing, mem0 owns the memory write/read on every turn&lt;/em&gt;. You call &lt;code&gt;memory.add(messages, { userId })&lt;/code&gt; after each user turn and &lt;code&gt;memory.search(query, { userId })&lt;/code&gt; before each model call. The framework extracts facts during &lt;code&gt;add&lt;/code&gt; and serves the relevant subset during &lt;code&gt;search&lt;/code&gt;. Install: &lt;code&gt;npm install mem0ai&lt;/code&gt; (open-source mode) or use &lt;a href="https://app.mem0.ai/"&gt;Mem0 Cloud&lt;/a&gt;.&lt;/p&gt;&#10;&lt;figure class="nb-code"&gt;&#10; &lt;header class="nb-code-chrome"&gt;&#10; &lt;span class="nb-code-label"&gt;typescript&lt;/span&gt;&#10; &#10; &lt;/header&gt;&#10; &lt;div class="nb-code-body"&gt;&lt;div class="highlight"&gt;&lt;div class="chroma"&gt;&#10;&lt;table class="lntable"&gt;&lt;tr&gt;&lt;td class="lntd"&gt;&#10;&lt;pre tabindex="0" class="chroma"&gt;&lt;code&gt;&lt;span class="lnt"&gt; 1&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 2&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 3&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 4&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 5&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 6&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 7&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 8&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 9&#10;&lt;/span&gt;&lt;span class="lnt"&gt;10&#10;&lt;/span&gt;&lt;span class="lnt"&gt;11&#10;&lt;/span&gt;&lt;span class="lnt"&gt;12&#10;&lt;/span&gt;&lt;span class="lnt"&gt;13&#10;&lt;/span&gt;&lt;span class="lnt"&gt;14&#10;&lt;/span&gt;&lt;span class="lnt"&gt;15&#10;&lt;/span&gt;&lt;span class="lnt"&gt;16&#10;&lt;/span&gt;&lt;span class="lnt"&gt;17&#10;&lt;/span&gt;&lt;span class="lnt"&gt;18&#10;&lt;/span&gt;&lt;span class="lnt"&gt;19&#10;&lt;/span&gt;&lt;span class="lnt"&gt;20&#10;&lt;/span&gt;&lt;span class="lnt"&gt;21&#10;&lt;/span&gt;&lt;span class="lnt"&gt;22&#10;&lt;/span&gt;&lt;span class="lnt"&gt;23&#10;&lt;/span&gt;&lt;span class="lnt"&gt;24&#10;&lt;/span&gt;&lt;span class="lnt"&gt;25&#10;&lt;/span&gt;&lt;span class="lnt"&gt;26&#10;&lt;/span&gt;&lt;span class="lnt"&gt;27&#10;&lt;/span&gt;&lt;span class="lnt"&gt;28&#10;&lt;/span&gt;&lt;span class="lnt"&gt;29&#10;&lt;/span&gt;&lt;span class="lnt"&gt;30&#10;&lt;/span&gt;&lt;span class="lnt"&gt;31&#10;&lt;/span&gt;&lt;span class="lnt"&gt;32&#10;&lt;/span&gt;&lt;span class="lnt"&gt;33&#10;&lt;/span&gt;&lt;span class="lnt"&gt;34&#10;&lt;/span&gt;&lt;span class="lnt"&gt;35&#10;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/td&gt;&#10;&lt;td class="lntd"&gt;&#10;&lt;pre tabindex="0" class="chroma"&gt;&lt;code class="language-typescript" data-lang="typescript"&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="c1"&gt;// npm install mem0ai&#10;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="kr"&gt;import&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="nx"&gt;Memory&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt; &lt;span class="kr"&gt;from&lt;/span&gt; &lt;span class="s2"&gt;&amp;#34;mem0ai/oss&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="c1"&gt;// Vector store + optional graph (Mem0g). Set OPENAI_API_KEY in the environment.&#10;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="kr"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;memory&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;new&lt;/span&gt; &lt;span class="nx"&gt;Memory&lt;/span&gt;&lt;span class="p"&gt;({&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="nx"&gt;vectorStore&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="nx"&gt;provider&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;&amp;#34;qdrant&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nx"&gt;config&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="nx"&gt;collectionName&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;&amp;#34;agent_memory&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nx"&gt;host&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;&amp;#34;localhost&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nx"&gt;port&lt;/span&gt;: &lt;span class="kt"&gt;6333&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt; &lt;span class="p"&gt;},&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="c1"&gt;// Uncomment to enable Mem0g — adds graph store for relational queries.&#10;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="c1"&gt;// graphStore: { provider: &amp;#34;neo4j&amp;#34;, config: { url: &amp;#34;bolt://localhost:7687&amp;#34;, username: &amp;#34;neo4j&amp;#34;, password: &amp;#34;password&amp;#34; } },&#10;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="p"&gt;});&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="c1"&gt;// Write: pass the conversation; mem0 runs LLM fact extraction internally&#10;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="c1"&gt;// and stores the distilled claims, not the raw turns.&#10;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="nx"&gt;memory&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;add&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="p"&gt;[&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="nx"&gt;role&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;&amp;#34;user&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nx"&gt;content&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;&amp;#34;I&amp;#39;m Priya. Order #4321 hasn&amp;#39;t shipped yet.&amp;#34;&lt;/span&gt; &lt;span class="p"&gt;},&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="nx"&gt;role&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;&amp;#34;assistant&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nx"&gt;content&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;&amp;#34;Hi Priya, let me check on order #4321.&amp;#34;&lt;/span&gt; &lt;span class="p"&gt;},&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="p"&gt;],&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="nx"&gt;userId&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;&amp;#34;user_priya_123&amp;#34;&lt;/span&gt; &lt;span class="p"&gt;},&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="p"&gt;);&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="c1"&gt;// Read: before the next model call, pull the relevant memories for the user.&#10;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="c1"&gt;// search() returns the distilled facts mem0 chose to store.&#10;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="kr"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;results&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="nx"&gt;memory&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;search&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;&amp;#34;What&amp;#39;s the user&amp;#39;s name and pending order?&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="nx"&gt;userId&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;&amp;#34;user_priya_123&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="nx"&gt;limit&lt;/span&gt;: &lt;span class="kt"&gt;5&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="p"&gt;});&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="kr"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;r&lt;/span&gt; &lt;span class="k"&gt;of&lt;/span&gt; &lt;span class="nx"&gt;results&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;results&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="nx"&gt;console&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;log&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sb"&gt;`[score=&lt;/span&gt;&lt;span class="si"&gt;${&lt;/span&gt;&lt;span class="nx"&gt;r&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;score&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;toFixed&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;2&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="sb"&gt;]`&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nx"&gt;r&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;memory&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="p"&gt;}&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="c1"&gt;// Inject results into the next system prompt. The harness, not the framework,&#10;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="c1"&gt;// owns this rendering step — mem0 returns the facts; you decide how to use them.&#10;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="kr"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;memoryContext&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nx"&gt;results&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;results&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;map&lt;/span&gt;&lt;span class="p"&gt;((&lt;/span&gt;&lt;span class="nx"&gt;r&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;=&amp;gt;&lt;/span&gt; &lt;span class="sb"&gt;`- &lt;/span&gt;&lt;span class="si"&gt;${&lt;/span&gt;&lt;span class="nx"&gt;r&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;memory&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="sb"&gt;`&lt;/span&gt;&lt;span class="p"&gt;).&lt;/span&gt;&lt;span class="nx"&gt;join&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;&amp;#34;\n&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="kr"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;systemPrompt&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="sb"&gt;`## Known facts about the user:&lt;/span&gt;&lt;span class="err"&gt;\&lt;/span&gt;&lt;span class="sb"&gt;n&lt;/span&gt;&lt;span class="si"&gt;${&lt;/span&gt;&lt;span class="nx"&gt;memoryContext&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="err"&gt;\&lt;/span&gt;&lt;span class="sb"&gt;n&lt;/span&gt;&lt;span class="err"&gt;\&lt;/span&gt;&lt;span class="sb"&gt;nReply to the user.`&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="c1"&gt;// ... pass systemPrompt to your LLM SDK of choice.&#10;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/td&gt;&lt;/tr&gt;&lt;/table&gt;&#10;&lt;/div&gt;&#10;&lt;/div&gt;&lt;/div&gt;&#10;&lt;/figure&gt;&#10;&lt;p&gt;The opinion this framework ships: &lt;em&gt;the unit of long-term memory is the distilled fact, not the raw turn&lt;/em&gt;. If you want the raw turns preserved verbatim, mem0 fights you; that&amp;rsquo;s not what the framework optimizes for. The escape hatch is &lt;code&gt;memory.add&lt;/code&gt; with &lt;code&gt;infer: false&lt;/code&gt;, which skips extraction and stores the raw text, but that path is not what the LoCoMo and LongMemEval numbers in the marketing are measured on.&lt;/p&gt;&#10;&lt;h2 id="graphiti-integration-sketch"&gt;Graphiti integration sketch&lt;/h2&gt;&#10;&lt;p&gt;Graphiti&amp;rsquo;s contract is &lt;em&gt;give me episodes with timestamps, I&amp;rsquo;ll give you a temporally-correct knowledge graph and a fused vector+BM25+graph retriever&lt;/em&gt;. Install: &lt;code&gt;pip install graphiti-core&lt;/code&gt; and run Neo4j (&lt;code&gt;docker run -d -p 7687:7687 -e NEO4J_AUTH=neo4j/password neo4j:5&lt;/code&gt;).&lt;/p&gt;&#10;&lt;figure class="nb-code"&gt;&#10; &lt;header class="nb-code-chrome"&gt;&#10; &lt;span class="nb-code-label"&gt;python&lt;/span&gt;&#10; &#10; &lt;/header&gt;&#10; &lt;div class="nb-code-body"&gt;&lt;div class="highlight"&gt;&lt;div class="chroma"&gt;&#10;&lt;table class="lntable"&gt;&lt;tr&gt;&lt;td class="lntd"&gt;&#10;&lt;pre tabindex="0" class="chroma"&gt;&lt;code&gt;&lt;span class="lnt"&gt; 1&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 2&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 3&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 4&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 5&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 6&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 7&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 8&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 9&#10;&lt;/span&gt;&lt;span class="lnt"&gt;10&#10;&lt;/span&gt;&lt;span class="lnt"&gt;11&#10;&lt;/span&gt;&lt;span class="lnt"&gt;12&#10;&lt;/span&gt;&lt;span class="lnt"&gt;13&#10;&lt;/span&gt;&lt;span class="lnt"&gt;14&#10;&lt;/span&gt;&lt;span class="lnt"&gt;15&#10;&lt;/span&gt;&lt;span class="lnt"&gt;16&#10;&lt;/span&gt;&lt;span class="lnt"&gt;17&#10;&lt;/span&gt;&lt;span class="lnt"&gt;18&#10;&lt;/span&gt;&lt;span class="lnt"&gt;19&#10;&lt;/span&gt;&lt;span class="lnt"&gt;20&#10;&lt;/span&gt;&lt;span class="lnt"&gt;21&#10;&lt;/span&gt;&lt;span class="lnt"&gt;22&#10;&lt;/span&gt;&lt;span class="lnt"&gt;23&#10;&lt;/span&gt;&lt;span class="lnt"&gt;24&#10;&lt;/span&gt;&lt;span class="lnt"&gt;25&#10;&lt;/span&gt;&lt;span class="lnt"&gt;26&#10;&lt;/span&gt;&lt;span class="lnt"&gt;27&#10;&lt;/span&gt;&lt;span class="lnt"&gt;28&#10;&lt;/span&gt;&lt;span class="lnt"&gt;29&#10;&lt;/span&gt;&lt;span class="lnt"&gt;30&#10;&lt;/span&gt;&lt;span class="lnt"&gt;31&#10;&lt;/span&gt;&lt;span class="lnt"&gt;32&#10;&lt;/span&gt;&lt;span class="lnt"&gt;33&#10;&lt;/span&gt;&lt;span class="lnt"&gt;34&#10;&lt;/span&gt;&lt;span class="lnt"&gt;35&#10;&lt;/span&gt;&lt;span class="lnt"&gt;36&#10;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/td&gt;&#10;&lt;td class="lntd"&gt;&#10;&lt;pre tabindex="0" class="chroma"&gt;&lt;code class="language-python" data-lang="python"&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="c1"&gt;# pip install graphiti-core neo4j&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="nn"&gt;asyncio&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="kn"&gt;from&lt;/span&gt; &lt;span class="nn"&gt;datetime&lt;/span&gt; &lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;datetime&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="kn"&gt;from&lt;/span&gt; &lt;span class="nn"&gt;graphiti_core&lt;/span&gt; &lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;Graphiti&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="n"&gt;graphiti&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;Graphiti&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;uri&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;#34;bolt://localhost:7687&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;user&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;#34;neo4j&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;password&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;#34;password&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="p"&gt;)&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="k"&gt;async&lt;/span&gt; &lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;main&lt;/span&gt;&lt;span class="p"&gt;():&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="n"&gt;graphiti&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;build_indices_and_constraints&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="c1"&gt;# add_episode triggers entity extraction, relation extraction, dedup,&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="c1"&gt;# bi-temporal stamping, and contradiction detection in one call.&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="n"&gt;graphiti&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;add_episode&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;name&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;#34;march_check_in&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;episode_body&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;#34;Priya is my manager. She approved the Q1 forecast on March 15.&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;source_description&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;#34;user message&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;reference_time&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;datetime&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;2026&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;3&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;15&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;group_id&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;#34;tenant_acme&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="c1"&gt;# multi-tenant scope&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="p"&gt;)&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="n"&gt;graphiti&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;add_episode&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;name&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;#34;april_update&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;episode_body&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;#34;Devansh is my new manager starting April 15. Priya moved to a different team.&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;source_description&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;#34;user message&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;reference_time&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;datetime&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;2026&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;4&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;15&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;group_id&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;#34;tenant_acme&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="p"&gt;)&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="c1"&gt;# The read path: search() is hybrid (graph traversal + vector + BM25 fused with RRF),&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="c1"&gt;# no LLM call in the loop. Filter by group_id for tenant isolation.&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;results&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="n"&gt;graphiti&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;search&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;query&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;#34;Who manages this user as of March 20?&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;group_ids&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;&amp;#34;tenant_acme&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;])&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;edge&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;results&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="nb"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;edge&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;fact&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;&amp;#34;valid:&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;edge&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;valid_at&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;&amp;#34;-&amp;gt;&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;edge&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;invalid_at&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="n"&gt;asyncio&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;run&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;main&lt;/span&gt;&lt;span class="p"&gt;())&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/td&gt;&lt;/tr&gt;&lt;/table&gt;&#10;&lt;/div&gt;&#10;&lt;/div&gt;&lt;/div&gt;&#10;&lt;/figure&gt;&#10;&lt;p&gt;The bi-temporal property is critical: the &amp;ldquo;as of March 20&amp;rdquo; query returns &lt;em&gt;Priya&lt;/em&gt; (the manager as-of that date), not Devansh (the current one). Vector-only stores cannot answer that question correctly regardless of how their retrieval is scored. If your workload doesn&amp;rsquo;t include point-in-time queries, this property is wasted complexity; if it does, no other framework ships it as a first-class concept.&lt;/p&gt;&#10;&lt;h2 id="further-reading"&gt;Further reading&lt;/h2&gt;&#10;&lt;ul&gt;&#10;&lt;li&gt;&lt;a href="https://mem0.ai/blog/state-of-ai-agent-memory-2026"&gt;State of AI Agent Memory 2026 (mem0.ai)&lt;/a&gt;: the most current cross-framework benchmark report, with explicit protocols. Pair with &lt;a href="https://www.letta.com/blog/benchmarking-ai-agent-memory"&gt;Letta&amp;rsquo;s controlled-benchmark response&lt;/a&gt; for the inevitable vendor-disagreement view.&lt;/li&gt;&#10;&lt;li&gt;&lt;a href="https://arxiv.org/abs/2310.08560"&gt;MemGPT: Towards LLMs as Operating Systems (Packer et al., 2023)&lt;/a&gt;: the founding paper for hierarchical memory and the OS-paging analogy. Letta is its production embodiment.&lt;/li&gt;&#10;&lt;li&gt;&lt;a href="https://arxiv.org/abs/2504.19413"&gt;Mem0: Building Production-Ready AI Agents with Scalable Long-Term Memory (Chhikara et al., 2025)&lt;/a&gt;: the standard reference for distill-at-write and Mem0g graph-augmented retrieval.&lt;/li&gt;&#10;&lt;li&gt;&lt;a href="https://arxiv.org/abs/2501.13956"&gt;Zep: A Temporal Knowledge Graph Architecture for Agent Memory (Rasmussen et al., 2025)&lt;/a&gt;: the bi-temporal-graph-as-primary-substrate case, with the LongMemEval breakdown that motivates the design.&lt;/li&gt;&#10;&lt;/ul&gt;&#10;</content:encoded></item><item><title>Memory Evaluation: Benchmarks and Custom Evals</title><link>https://jatinbansal.com/ai-engineering/memory-evaluation/</link><guid isPermaLink="true">https://jatinbansal.com/ai-engineering/memory-evaluation/</guid><pubDate>Tue, 19 May 2026 03:09:51 +0000</pubDate><dc:creator>Jatin Bansal</dc:creator><category>AI Engineering</category><description>How public benchmarks and custom evals measure memory recall, updates, temporal reasoning, and cost.</description><content:encoded>&lt;p&gt;Memory dashboards can report writes, reads, latency, and store size without measuring whether the system recalls the right fact. Evaluation needs multi-session traces, later questions with known answers, relevant episode IDs, and separate scores for temporal updates and abstention.&lt;/p&gt;&#10;&lt;h2 id="the-memory-eval-contract"&gt;The memory-eval contract&lt;/h2&gt;&#10;&lt;p&gt;A memory eval ingests a fixed corpus of multi-session interactions before presenting its questions. Questions are tagged by ability, including single-hop recall, multi-hop reasoning, temporal reasoning, knowledge updates, and abstention. Reports should include retrieval and answer quality alongside tokens, latency, and cost.&lt;/p&gt;&#10;&lt;p&gt;&lt;a href="https://jatinbansal.com/ai-engineering/rag-evaluation/"&gt;RAG evaluation&lt;/a&gt; scores retrieval against a mostly static corpus. Memory evaluation scores a corpus the system created during prior interactions. A &lt;a href="https://github.com/gkamradt/LLMTest_NeedleInAHaystack"&gt;needle-in-a-haystack test&lt;/a&gt; asks whether a model can find a literal fact already present in its prompt; a memory test asks whether the full system can recover a paraphrased fact across sessions.&lt;/p&gt;&#10;&lt;h2 id="public-benchmarks"&gt;Public benchmarks&lt;/h2&gt;&#10;&lt;p&gt;Five benchmarks define the field in 2026. Each probes a different facet. A defensible memory system has numbers on at least the first two; serious teams run all five.&lt;/p&gt;&#10;&lt;p&gt;&lt;strong&gt;LoCoMo&lt;/strong&gt; (&lt;a href="https://snap-research.github.io/locomo/"&gt;Maharana et al., 2024&lt;/a&gt;). Fifty multi-session conversations between two fictional speakers, each spanning up to 32 sessions and averaging ~600 turns (~16K tokens), with timestamps and multimodal context. The 1,540-question QA set covers five categories: single-hop (intra-session recall), multi-hop (cross-session synthesis), temporal reasoning, open-domain, and adversarial (questions that should be refused). The paper reports GPT-4 at ~32.1 F1 against a human ceiling of ~87.9; the field&amp;rsquo;s headline gap. Recent framework numbers: Mem0&amp;rsquo;s 2026 token-efficient algorithm reports &lt;a href="https://mem0.ai/research"&gt;92.5% on LoCoMo&lt;/a&gt;; the 2025 &lt;a href="https://arxiv.org/abs/2504.19413"&gt;Mem0 paper&lt;/a&gt; reports 66.9% vector-only and 68.4% with the Mem0g graph extension, both versus 52.9% for OpenAI&amp;rsquo;s built-in memory feature. The wide range reflects different LLM judges, prompts, and ingest pipelines; read the protocol before comparing.&lt;/p&gt;&#10;&lt;p&gt;&lt;strong&gt;LongMemEval&lt;/strong&gt; (&lt;a href="https://arxiv.org/abs/2410.10813"&gt;Wu et al., ICLR 2025&lt;/a&gt;). Five hundred manually-curated questions embedded in freely-scalable chat histories, probing five memory abilities: information extraction, multi-session reasoning, temporal reasoning, knowledge updates, and abstention. The construction is more rigorous than LoCoMo&amp;rsquo;s; each question is human-authored, with histories scalable to test ingest-time behavior at multiple sizes. The paper documents commercial chat assistants dropping ~30% on sustained interactions versus single-session baselines. &lt;a href="https://arxiv.org/abs/2501.13956"&gt;Zep&amp;rsquo;s January 2025 paper&lt;/a&gt; reports 71.2% overall on LongMemEval with GPT-4o (versus 60.2% for vanilla full-context at 29s latency), with the largest gains in temporal-reasoning (+17.3pp) and multi-session (+13.6pp). Mem0&amp;rsquo;s 2026 algorithm reports &lt;a href="https://mem0.ai/research"&gt;94.4% on LongMemEval&lt;/a&gt;.&lt;/p&gt;&#10;&lt;p&gt;&lt;strong&gt;MemoryAgentBench&lt;/strong&gt; (&lt;a href="https://arxiv.org/abs/2507.05257"&gt;Hu et al., 2025&lt;/a&gt;). Probes four cognitive competencies; accurate retrieval, test-time learning, long-range understanding, and &lt;em&gt;selective forgetting&lt;/em&gt;. The selective-forgetting category finally exposed the contradiction-resolution failure mode every framework had been papering over: &lt;em&gt;all&lt;/em&gt; paradigms fail dramatically on multi-hop conflict resolution, with best accuracy at or below 6% on CR-MH (Contradiction Resolution, Multi-Hop) even in the most advanced models. Run it if your product depends on the agent updating beliefs as facts change.&lt;/p&gt;&#10;&lt;p&gt;&lt;strong&gt;BEAM&lt;/strong&gt; (&lt;a href="https://mem0.ai/blog/state-of-ai-agent-memory-2026"&gt;Mem0&amp;rsquo;s State of AI Agent Memory 2026&lt;/a&gt;). Scales to 1M and 10M token regimes, with ten categories including preference following, knowledge update, event ordering, abstention, and contradiction resolution. Mem0&amp;rsquo;s 2026 numbers: 64.1 and 48.6 on BEAM 1M/10M. The contradiction-resolution and event-ordering categories are where production systems silently regress as corpora grow.&lt;/p&gt;&#10;&lt;p&gt;&lt;strong&gt;∞Bench&lt;/strong&gt; (&lt;a href="https://arxiv.org/abs/2402.13718"&gt;Zhang et al., 2024&lt;/a&gt;), &lt;strong&gt;NoLiMa&lt;/strong&gt; (&lt;a href="https://arxiv.org/abs/2502.05167"&gt;Modarressi et al., ICML 2025&lt;/a&gt;), and &lt;strong&gt;RULER&lt;/strong&gt; (&lt;a href="https://arxiv.org/abs/2404.06654"&gt;Hsieh et al., 2024&lt;/a&gt;) are long-context evals, not memory evals; they measure whether a model can find a fact buried in a 100K+ or 32K+ token prompt, not whether a system can recall across sessions. They calibrate the &lt;em&gt;long-context fallback&lt;/em&gt;: NoLiMa&amp;rsquo;s finding that 11 of 13 tested 128K-claiming models drop below 50% of their short-context baseline at 32K is the empirical case for memory-as-a-system over context stuffing.&lt;/p&gt;&#10;&lt;p&gt;A defensible benchmark story: numbers on LoCoMo and LongMemEval for the headline; MemoryAgentBench or BEAM if contradiction and selective forgetting matter; ∞Bench/NoLiMa/RULER numbers to justify why the system uses retrieval rather than bigger context.&lt;/p&gt;&#10;&lt;h2 id="retrieval-and-answer-metrics"&gt;Retrieval and answer metrics&lt;/h2&gt;&#10;&lt;p&gt;Two metric families do the work. &lt;strong&gt;Precision/recall/F1 over retrieved memories&lt;/strong&gt; measures the &lt;em&gt;retrieval substrate&lt;/em&gt;; given a query, how many of the top-K retrieved episodes are actually relevant, and how many of the relevant episodes in the store made it into the top-K? &lt;strong&gt;End-to-end answer correctness&lt;/strong&gt; (&lt;a href="https://jatinbansal.com/ai-engineering/llm-as-judge/"&gt;LLM-as-judge&lt;/a&gt;, sometimes string-match against a normalized gold) measures whether the &lt;em&gt;whole system&lt;/em&gt;; retrieval plus generation; answers right.&lt;/p&gt;&#10;&lt;p&gt;Both matter and fail differently. A retrieval substrate at 90% precision/recall with a generator that fumbles synthesis scores low end-to-end. A substrate that retrieves garbage paired with a generator whose parametric memory carries the answer scores high. The &lt;a href="https://arxiv.org/abs/2504.15965"&gt;Memory for Autonomous LLM Agents survey (2025)&lt;/a&gt; lays out the taxonomy; the production rule is &lt;em&gt;measure both, separately, and pay attention when they diverge&lt;/em&gt;.&lt;/p&gt;&#10;&lt;p&gt;The precision/recall mechanics, for episodic recall against a known gold:&lt;/p&gt;&#10;&lt;ul&gt;&#10;&lt;li&gt;Top-K retrieval: the system returns K episodes for a query.&lt;/li&gt;&#10;&lt;li&gt;Gold-relevant set: the eval pre-computes which episodes are relevant. For LoCoMo this is human-annotated; for a custom eval, you build it.&lt;/li&gt;&#10;&lt;li&gt;Precision@K: (relevant retrieved) / K. &lt;strong&gt;Recall@K:&lt;/strong&gt; (relevant retrieved) / (total relevant in store). &lt;strong&gt;F1@K:&lt;/strong&gt; harmonic mean.&lt;/li&gt;&#10;&lt;/ul&gt;&#10;&lt;p&gt;Pair these with &lt;strong&gt;temporal-correctness&lt;/strong&gt; for the temporal category; &amp;ldquo;what did the agent believe last Tuesday?&amp;rdquo; should return the belief that held &lt;em&gt;as of last Tuesday&lt;/em&gt;, not the current one. &lt;a href="https://jatinbansal.com/ai-engineering/temporal-reasoning-provenance/"&gt;Bi-temporal stores&lt;/a&gt; like Graphiti score this category much higher than vector-only stores.&lt;/p&gt;&#10;&lt;p&gt;For &lt;em&gt;contradiction resolution&lt;/em&gt;, the metric is &lt;a href="https://jatinbansal.com/ai-engineering/memory-conflict-and-forgetting/"&gt;supersession behavior&lt;/a&gt;: when two episodes claim contradictory facts, does the answer reflect the most recent (or most-trusted) claim, or average / pick at random? MemoryAgentBench CR-MH scores this; for a custom eval, the simplest version is hand-built &amp;ldquo;fact A at session N; ¬A at N+5; question at N+10&amp;rdquo; tuples.&lt;/p&gt;&#10;&lt;h2 id="code-a-precisionrecall-harness-python"&gt;Code: a precision/recall harness (Python)&lt;/h2&gt;&#10;&lt;p&gt;Phase-separated ingest-then-query, category-tagged questions, per-category metrics with token-cost alongside. Store-interchangeable across &lt;a href="https://docs.mem0.ai/"&gt;Mem0&lt;/a&gt;, &lt;a href="https://docs.langchain.com/oss/python/langgraph/persistence"&gt;LangGraph store&lt;/a&gt;, &lt;a href="https://docs.letta.com/"&gt;Letta&lt;/a&gt;, or a homegrown vector store.&lt;/p&gt;&#10;&lt;figure class="nb-code"&gt;&#10; &lt;header class="nb-code-chrome"&gt;&#10; &lt;span class="nb-code-label"&gt;python&lt;/span&gt;&#10; &#10; &lt;/header&gt;&#10; &lt;div class="nb-code-body"&gt;&lt;div class="highlight"&gt;&lt;div class="chroma"&gt;&#10;&lt;table class="lntable"&gt;&lt;tr&gt;&lt;td class="lntd"&gt;&#10;&lt;pre tabindex="0" class="chroma"&gt;&lt;code&gt;&lt;span class="lnt"&gt; 1&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 2&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 3&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 4&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 5&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 6&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 7&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 8&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 9&#10;&lt;/span&gt;&lt;span class="lnt"&gt;10&#10;&lt;/span&gt;&lt;span class="lnt"&gt;11&#10;&lt;/span&gt;&lt;span class="lnt"&gt;12&#10;&lt;/span&gt;&lt;span class="lnt"&gt;13&#10;&lt;/span&gt;&lt;span class="lnt"&gt;14&#10;&lt;/span&gt;&lt;span class="lnt"&gt;15&#10;&lt;/span&gt;&lt;span class="lnt"&gt;16&#10;&lt;/span&gt;&lt;span class="lnt"&gt;17&#10;&lt;/span&gt;&lt;span class="lnt"&gt;18&#10;&lt;/span&gt;&lt;span class="lnt"&gt;19&#10;&lt;/span&gt;&lt;span class="lnt"&gt;20&#10;&lt;/span&gt;&lt;span class="lnt"&gt;21&#10;&lt;/span&gt;&lt;span class="lnt"&gt;22&#10;&lt;/span&gt;&lt;span class="lnt"&gt;23&#10;&lt;/span&gt;&lt;span class="lnt"&gt;24&#10;&lt;/span&gt;&lt;span class="lnt"&gt;25&#10;&lt;/span&gt;&lt;span class="lnt"&gt;26&#10;&lt;/span&gt;&lt;span class="lnt"&gt;27&#10;&lt;/span&gt;&lt;span class="lnt"&gt;28&#10;&lt;/span&gt;&lt;span class="lnt"&gt;29&#10;&lt;/span&gt;&lt;span class="lnt"&gt;30&#10;&lt;/span&gt;&lt;span class="lnt"&gt;31&#10;&lt;/span&gt;&lt;span class="lnt"&gt;32&#10;&lt;/span&gt;&lt;span class="lnt"&gt;33&#10;&lt;/span&gt;&lt;span class="lnt"&gt;34&#10;&lt;/span&gt;&lt;span class="lnt"&gt;35&#10;&lt;/span&gt;&lt;span class="lnt"&gt;36&#10;&lt;/span&gt;&lt;span class="lnt"&gt;37&#10;&lt;/span&gt;&lt;span class="lnt"&gt;38&#10;&lt;/span&gt;&lt;span class="lnt"&gt;39&#10;&lt;/span&gt;&lt;span class="lnt"&gt;40&#10;&lt;/span&gt;&lt;span class="lnt"&gt;41&#10;&lt;/span&gt;&lt;span class="lnt"&gt;42&#10;&lt;/span&gt;&lt;span class="lnt"&gt;43&#10;&lt;/span&gt;&lt;span class="lnt"&gt;44&#10;&lt;/span&gt;&lt;span class="lnt"&gt;45&#10;&lt;/span&gt;&lt;span class="lnt"&gt;46&#10;&lt;/span&gt;&lt;span class="lnt"&gt;47&#10;&lt;/span&gt;&lt;span class="lnt"&gt;48&#10;&lt;/span&gt;&lt;span class="lnt"&gt;49&#10;&lt;/span&gt;&lt;span class="lnt"&gt;50&#10;&lt;/span&gt;&lt;span class="lnt"&gt;51&#10;&lt;/span&gt;&lt;span class="lnt"&gt;52&#10;&lt;/span&gt;&lt;span class="lnt"&gt;53&#10;&lt;/span&gt;&lt;span class="lnt"&gt;54&#10;&lt;/span&gt;&lt;span class="lnt"&gt;55&#10;&lt;/span&gt;&lt;span class="lnt"&gt;56&#10;&lt;/span&gt;&lt;span class="lnt"&gt;57&#10;&lt;/span&gt;&lt;span class="lnt"&gt;58&#10;&lt;/span&gt;&lt;span class="lnt"&gt;59&#10;&lt;/span&gt;&lt;span class="lnt"&gt;60&#10;&lt;/span&gt;&lt;span class="lnt"&gt;61&#10;&lt;/span&gt;&lt;span class="lnt"&gt;62&#10;&lt;/span&gt;&lt;span class="lnt"&gt;63&#10;&lt;/span&gt;&lt;span class="lnt"&gt;64&#10;&lt;/span&gt;&lt;span class="lnt"&gt;65&#10;&lt;/span&gt;&lt;span class="lnt"&gt;66&#10;&lt;/span&gt;&lt;span class="lnt"&gt;67&#10;&lt;/span&gt;&lt;span class="lnt"&gt;68&#10;&lt;/span&gt;&lt;span class="lnt"&gt;69&#10;&lt;/span&gt;&lt;span class="lnt"&gt;70&#10;&lt;/span&gt;&lt;span class="lnt"&gt;71&#10;&lt;/span&gt;&lt;span class="lnt"&gt;72&#10;&lt;/span&gt;&lt;span class="lnt"&gt;73&#10;&lt;/span&gt;&lt;span class="lnt"&gt;74&#10;&lt;/span&gt;&lt;span class="lnt"&gt;75&#10;&lt;/span&gt;&lt;span class="lnt"&gt;76&#10;&lt;/span&gt;&lt;span class="lnt"&gt;77&#10;&lt;/span&gt;&lt;span class="lnt"&gt;78&#10;&lt;/span&gt;&lt;span class="lnt"&gt;79&#10;&lt;/span&gt;&lt;span class="lnt"&gt;80&#10;&lt;/span&gt;&lt;span class="lnt"&gt;81&#10;&lt;/span&gt;&lt;span class="lnt"&gt;82&#10;&lt;/span&gt;&lt;span class="lnt"&gt;83&#10;&lt;/span&gt;&lt;span class="lnt"&gt;84&#10;&lt;/span&gt;&lt;span class="lnt"&gt;85&#10;&lt;/span&gt;&lt;span class="lnt"&gt;86&#10;&lt;/span&gt;&lt;span class="lnt"&gt;87&#10;&lt;/span&gt;&lt;span class="lnt"&gt;88&#10;&lt;/span&gt;&lt;span class="lnt"&gt;89&#10;&lt;/span&gt;&lt;span class="lnt"&gt;90&#10;&lt;/span&gt;&lt;span class="lnt"&gt;91&#10;&lt;/span&gt;&lt;span class="lnt"&gt;92&#10;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/td&gt;&#10;&lt;td class="lntd"&gt;&#10;&lt;pre tabindex="0" class="chroma"&gt;&lt;code class="language-python" data-lang="python"&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="c1"&gt;# pip install mem0ai openai&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="kn"&gt;from&lt;/span&gt; &lt;span class="nn"&gt;dataclasses&lt;/span&gt; &lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;dataclass&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;field&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="kn"&gt;from&lt;/span&gt; &lt;span class="nn"&gt;typing&lt;/span&gt; &lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;Any&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;Callable&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;Literal&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="nn"&gt;json&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt; &lt;span class="nn"&gt;time&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="kn"&gt;from&lt;/span&gt; &lt;span class="nn"&gt;collections&lt;/span&gt; &lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;defaultdict&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="n"&gt;Category&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;Literal&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;&amp;#34;single_hop&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;&amp;#34;multi_hop&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;&amp;#34;temporal&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;&amp;#34;knowledge_update&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;&amp;#34;abstention&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="nd"&gt;@dataclass&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="k"&gt;class&lt;/span&gt; &lt;span class="nc"&gt;EvalQuestion&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;qid&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;str&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;user_id&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;str&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;question&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;str&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;gold_answer&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;str&lt;/span&gt; &lt;span class="o"&gt;|&lt;/span&gt; &lt;span class="kc"&gt;None&lt;/span&gt; &lt;span class="c1"&gt;# None for abstention category&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;relevant_episode_ids&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;set&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="nb"&gt;str&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="c1"&gt;# ground truth for retrieval metrics&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;category&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;Category&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="nd"&gt;@dataclass&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="k"&gt;class&lt;/span&gt; &lt;span class="nc"&gt;IngestEvent&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;user_id&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;str&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;text&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;str&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;episode_id&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;str&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;timestamp&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;float&lt;/span&gt; &lt;span class="c1"&gt;# seconds since epoch; preserves multi-session structure&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="nd"&gt;@dataclass&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="k"&gt;class&lt;/span&gt; &lt;span class="nc"&gt;EvalRun&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;by_category&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;dict&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;Category&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nb"&gt;list&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="nb"&gt;dict&lt;/span&gt;&lt;span class="p"&gt;]]&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;field&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;default_factory&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="k"&gt;lambda&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;defaultdict&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nb"&gt;list&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;add&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="bp"&gt;self&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;cat&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;Category&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="o"&gt;**&lt;/span&gt;&lt;span class="n"&gt;fields&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="bp"&gt;self&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;by_category&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;cat&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;append&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;fields&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;summary&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="bp"&gt;self&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;-&amp;gt;&lt;/span&gt; &lt;span class="nb"&gt;dict&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;out&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;{}&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;cat&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;rows&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="bp"&gt;self&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;by_category&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;items&lt;/span&gt;&lt;span class="p"&gt;():&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="ow"&gt;not&lt;/span&gt; &lt;span class="n"&gt;rows&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="k"&gt;continue&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;n&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nb"&gt;len&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;rows&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;out&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;cat&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="s2"&gt;&amp;#34;n&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;n&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="s2"&gt;&amp;#34;precision_at_k&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;sum&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;&amp;#34;precision&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;r&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;rows&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt; &lt;span class="n"&gt;n&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="s2"&gt;&amp;#34;recall_at_k&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;sum&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;&amp;#34;recall&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;r&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;rows&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt; &lt;span class="n"&gt;n&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="s2"&gt;&amp;#34;f1_at_k&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;sum&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;&amp;#34;f1&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;r&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;rows&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt; &lt;span class="n"&gt;n&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="s2"&gt;&amp;#34;answer_correct&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;sum&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;&amp;#34;answer_correct&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;r&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;rows&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt; &lt;span class="n"&gt;n&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="s2"&gt;&amp;#34;avg_tokens&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;sum&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;&amp;#34;tokens&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;r&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;rows&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt; &lt;span class="n"&gt;n&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="s2"&gt;&amp;#34;avg_latency_ms&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;sum&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;&amp;#34;latency_ms&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;r&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;rows&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt; &lt;span class="n"&gt;n&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="p"&gt;}&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;out&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;f1&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;p&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;float&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;float&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;-&amp;gt;&lt;/span&gt; &lt;span class="nb"&gt;float&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="mf"&gt;0.0&lt;/span&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;p&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt; &lt;span class="k"&gt;else&lt;/span&gt; &lt;span class="mi"&gt;2&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="n"&gt;p&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="n"&gt;r&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;p&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;run_eval&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;ingest_events&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;list&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;IngestEvent&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;questions&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;list&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;EvalQuestion&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;memory_write&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;Callable&lt;/span&gt;&lt;span class="p"&gt;[[&lt;/span&gt;&lt;span class="n"&gt;IngestEvent&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt; &lt;span class="kc"&gt;None&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;memory_search&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;Callable&lt;/span&gt;&lt;span class="p"&gt;[[&lt;/span&gt;&lt;span class="nb"&gt;str&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nb"&gt;str&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nb"&gt;int&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt; &lt;span class="nb"&gt;list&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="nb"&gt;dict&lt;/span&gt;&lt;span class="p"&gt;]],&lt;/span&gt; &lt;span class="c1"&gt;# (user_id, query, k) -&amp;gt; [{episode_id, text, score}]&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;answer_with_context&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;Callable&lt;/span&gt;&lt;span class="p"&gt;[[&lt;/span&gt;&lt;span class="nb"&gt;str&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nb"&gt;list&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="nb"&gt;str&lt;/span&gt;&lt;span class="p"&gt;]],&lt;/span&gt; &lt;span class="nb"&gt;tuple&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="nb"&gt;str&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nb"&gt;int&lt;/span&gt;&lt;span class="p"&gt;]],&lt;/span&gt; &lt;span class="c1"&gt;# (q, retrieved_texts) -&amp;gt; (answer, tokens_used)&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;judge_correct&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;Callable&lt;/span&gt;&lt;span class="p"&gt;[[&lt;/span&gt;&lt;span class="nb"&gt;str&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nb"&gt;str&lt;/span&gt; &lt;span class="o"&gt;|&lt;/span&gt; &lt;span class="kc"&gt;None&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;Category&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt; &lt;span class="nb"&gt;bool&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt; &lt;span class="c1"&gt;# (pred, gold, category) -&amp;gt; bool&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;k&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;int&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mi"&gt;5&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;-&amp;gt;&lt;/span&gt; &lt;span class="n"&gt;EvalRun&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="c1"&gt;# Phase 1: ingest. The memory system has no access to the questions here.&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;ev&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="nb"&gt;sorted&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;ingest_events&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;key&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="k"&gt;lambda&lt;/span&gt; &lt;span class="n"&gt;e&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;e&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;timestamp&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;memory_write&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;ev&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="c1"&gt;# Phase 2: query. Per-question retrieval, scoring, and category attribution.&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;run&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;EvalRun&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;q&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;questions&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;t0&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;time&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;perf_counter&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;retrieved&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;memory_search&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;q&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;user_id&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;q&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;question&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;k&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;retrieved_ids&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;&amp;#34;episode_id&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;r&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;retrieved&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;relevant_retrieved&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;retrieved_ids&lt;/span&gt; &lt;span class="o"&gt;&amp;amp;&lt;/span&gt; &lt;span class="n"&gt;q&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;relevant_episode_ids&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="c1"&gt;# Edge: abstention questions have no relevant episodes; recall is 1 by convention,&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="c1"&gt;# precision is 0 if any episodes were retrieved (system should have abstained on empty).&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;q&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;category&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="s2"&gt;&amp;#34;abstention&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;precision&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mf"&gt;0.0&lt;/span&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;retrieved&lt;/span&gt; &lt;span class="k"&gt;else&lt;/span&gt; &lt;span class="mf"&gt;1.0&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;recall&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mf"&gt;1.0&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="k"&gt;else&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;precision&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nb"&gt;len&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;relevant_retrieved&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt; &lt;span class="nb"&gt;len&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;retrieved_ids&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;retrieved_ids&lt;/span&gt; &lt;span class="k"&gt;else&lt;/span&gt; &lt;span class="mf"&gt;0.0&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;recall&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nb"&gt;len&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;relevant_retrieved&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt; &lt;span class="nb"&gt;len&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;q&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;relevant_episode_ids&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;q&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;relevant_episode_ids&lt;/span&gt; &lt;span class="k"&gt;else&lt;/span&gt; &lt;span class="mf"&gt;0.0&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;answer&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;tokens&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;answer_with_context&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;q&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;question&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;&amp;#34;text&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;r&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;retrieved&lt;/span&gt;&lt;span class="p"&gt;])&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;correct&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;judge_correct&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;answer&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;q&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;gold_answer&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;q&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;category&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;run&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;add&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;q&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;category&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;qid&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;q&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;qid&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;precision&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;precision&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;recall&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;recall&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;f1&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;f1&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;precision&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;recall&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;answer_correct&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mf"&gt;1.0&lt;/span&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;correct&lt;/span&gt; &lt;span class="k"&gt;else&lt;/span&gt; &lt;span class="mf"&gt;0.0&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;tokens&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;tokens&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;latency_ms&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;time&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;perf_counter&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt; &lt;span class="n"&gt;t0&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="mf"&gt;1000.0&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="p"&gt;)&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;run&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/td&gt;&lt;/tr&gt;&lt;/table&gt;&#10;&lt;/div&gt;&#10;&lt;/div&gt;&lt;/div&gt;&#10;&lt;/figure&gt;&#10;&lt;p&gt;The &lt;code&gt;memory_write&lt;/code&gt; and &lt;code&gt;memory_search&lt;/code&gt; callables are the only contract; the same harness scores Mem0, LangGraph, and a homegrown vector store identically. For abstention questions, the judge should accept any refusal phrasing as correct when the gold is &lt;code&gt;None&lt;/code&gt;. &lt;code&gt;EvalRun.summary()&lt;/code&gt; is the publishable artifact: per-category precision, recall, F1, end-to-end correctness, and token/latency budget in one dict.&lt;/p&gt;&#10;&lt;h2 id="designing-a-custom-eval"&gt;Designing a custom eval&lt;/h2&gt;&#10;&lt;p&gt;Public benchmarks are necessary, not sufficient. They probe generic conversational memory; they don&amp;rsquo;t probe your product&amp;rsquo;s recall patterns. A customer-support agent has return-policy preferences and account-tier context that LoCoMo doesn&amp;rsquo;t touch; a medical-history agent has drug interactions and prior diagnoses that LongMemEval doesn&amp;rsquo;t touch. The custom eval is where workload-specific failure modes show up.&lt;/p&gt;&#10;&lt;p&gt;The minimum shape:&lt;/p&gt;&#10;&lt;ol&gt;&#10;&lt;li&gt;20-50 real or near-real multi-session traces: Real if privacy permits; otherwise synthetic in the product&amp;rsquo;s actual conversational shape. Curated 20-50 beats lazy 500.&lt;/li&gt;&#10;&lt;li&gt;Per-trace question set, 10-20 questions: Tagged by the LongMemEval five-ability frame. Each has a gold answer (or &lt;code&gt;None&lt;/code&gt; for abstention) and relevant-episode IDs for retrieval metrics.&lt;/li&gt;&#10;&lt;li&gt;An ingest-then-query harness: The Python/TS code above is the template.&lt;/li&gt;&#10;&lt;li&gt;A weekly cron run: Every memory-policy change passes the eval before shipping. A per-category regression is blocking even if the aggregate holds.&lt;/li&gt;&#10;&lt;li&gt;A growing adversarial subset: Every production incident caused by a memory bug becomes a regression question. Over time this set becomes more useful than the seed corpus.&lt;/li&gt;&#10;&lt;/ol&gt;&#10;&lt;p&gt;The trap to avoid: scoring only end-to-end answer correctness and ignoring retrieval precision/recall. If retrieval is at 30% recall but the generator&amp;rsquo;s parametric knowledge masks it for 80% of questions, the system will fail catastrophically the moment the &lt;a href="https://jatinbansal.com/ai-engineering/drift-detection/"&gt;question distribution shifts&lt;/a&gt;. Retrieval metrics are the leading indicator; end-to-end is the lagging one.&lt;/p&gt;&#10;&lt;p&gt;For &lt;a href="https://docs.ragas.io/"&gt;Ragas-style&lt;/a&gt; metrics applied to memory: context precision and context recall translate directly, and faithfulness catches &amp;ldquo;generator confabulates a memory.&amp;rdquo; The multi-session-specific categories (temporal correctness, contradiction resolution) diverge; Ragas doesn&amp;rsquo;t have those out of the box.&lt;/p&gt;&#10;&lt;h2 id="further-reading"&gt;Further reading&lt;/h2&gt;&#10;&lt;ul&gt;&#10;&lt;li&gt;&lt;a href="https://arxiv.org/abs/2410.10813"&gt;LongMemEval (Wu et al., ICLR 2025)&lt;/a&gt;; the most rigorously-constructed public memory eval. The five-ability framing is the right shape to copy for a custom eval; pair with the &lt;a href="https://github.com/xiaowu0162/LongMemEval"&gt;GitHub repo&lt;/a&gt; for prompt templates and judge protocol.&lt;/li&gt;&#10;&lt;li&gt;&lt;a href="https://snap-research.github.io/locomo/"&gt;LoCoMo (Maharana et al., 2024)&lt;/a&gt;; the canonical multi-session benchmark. The 32.1 F1 for GPT-4 vs 87.9 human ceiling is the field&amp;rsquo;s headline gap; recent framework numbers (Mem0 92.5%) only make sense against that baseline.&lt;/li&gt;&#10;&lt;li&gt;&lt;a href="https://arxiv.org/abs/2504.15965"&gt;Memory for Autonomous LLM Agents (2025)&lt;/a&gt;; comprehensive survey with the single best treatment of the metric taxonomy and LLM-as-judge bias.&lt;/li&gt;&#10;&lt;li&gt;&lt;a href="https://mem0.ai/blog/state-of-ai-agent-memory-2026"&gt;State of AI Agent Memory 2026 (mem0.ai)&lt;/a&gt;; production-framework view on current numbers across LoCoMo, LongMemEval, BEAM, with explicit protocol notes. Pair with &lt;a href="https://www.letta.com/blog/benchmarking-ai-agent-memory"&gt;Letta&amp;rsquo;s controlled-benchmark post&lt;/a&gt; for a different vendor&amp;rsquo;s read.&lt;/li&gt;&#10;&lt;/ul&gt;&#10;</content:encoded></item><item><title>Conversation Compaction: Keeping Long Sessions Alive</title><link>https://jatinbansal.com/ai-engineering/conversation-compaction/</link><guid isPermaLink="true">https://jatinbansal.com/ai-engineering/conversation-compaction/</guid><pubDate>Tue, 19 May 2026 02:54:25 +0000</pubDate><dc:creator>Jatin Bansal</dc:creator><category>AI Engineering</category><description>How long agent sessions compact context using preemptive triggers, cache-aware deletion, rollback, and journals.</description><content:encoded>&lt;p&gt;A large tool result can push an agent beyond its context limit before a reactive compactor runs. If the summarizer has a smaller window, it may reject the same buffer and leave the session unable to continue. A reliable harness compacts with headroom, commits changes atomically, and keeps a degraded truncation path for summarizer failure.&lt;/p&gt;&#10;&lt;h2 id="when-compaction-runs"&gt;When compaction runs&lt;/h2&gt;&#10;&lt;p&gt;&lt;strong&gt;Conversation compaction is the harness operation that, when triggered, reduces the size of the live conversation buffer in place so that the next model call fits.&lt;/strong&gt; Three properties distinguish &lt;em&gt;conversation&lt;/em&gt; compaction in a long session from the broader compression operation. It is &lt;em&gt;in-place mutation of the live buffer&lt;/em&gt;, not an offline pass; the next call must succeed against the new buffer. It is &lt;em&gt;non-optional once triggered&lt;/em&gt;: there is no path forward if it fails. And it is &lt;em&gt;cache-disruptive by default&lt;/em&gt;: any rewrite invalidates the &lt;a href="https://jatinbansal.com/ai-engineering/prompt-caching/"&gt;prompt-cache prefix&lt;/a&gt; from the rewrite boundary onward, so a naive implementation pays full prefill on every subsequent call. The orchestration question is how to fire it as rarely and cheaply as possible, and how to keep the session alive when it fails.&lt;/p&gt;&#10;&lt;h2 id="reactive-vs-preemptive-triggers"&gt;Reactive vs preemptive triggers&lt;/h2&gt;&#10;&lt;p&gt;Two philosophies, and most production harnesses end up running both.&lt;/p&gt;&#10;&lt;p&gt;&lt;strong&gt;Reactive compaction&lt;/strong&gt; fires &lt;em&gt;after&lt;/em&gt; the buffer crosses a watermark. Anthropic&amp;rsquo;s &lt;a href="https://platform.claude.com/docs/en/build-with-claude/compaction"&gt;compaction beta&lt;/a&gt; is the cleanest example; the server detects when input tokens exceed the configured trigger (default 150,000), runs compaction inline, emits a compaction block, and continues. OpenAI&amp;rsquo;s &lt;a href="https://developers.openai.com/api/docs/guides/compaction"&gt;server-side compaction via the Responses API&lt;/a&gt; (shipped February 11, 2026) is conceptually identical: pass &lt;code&gt;context_management.compact_threshold&lt;/code&gt; and the server fires compaction when the rendered token count crosses it. Claude Code&amp;rsquo;s auto-compact at ~95% is the client-side version. The pros are operational simplicity: a single number, a single conditional. The cons: the user-facing turn that crosses the watermark pays 1-5s of compaction latency, and there&amp;rsquo;s no head-room; if the model emits a 50K-token tool result at 90K of a 100K budget, the compaction must succeed on the first try because there&amp;rsquo;s no room for retry tokens.&lt;/p&gt;&#10;&lt;p&gt;&lt;strong&gt;Preemptive compaction&lt;/strong&gt; fires &lt;em&gt;before&lt;/em&gt; the buffer is dangerous, by projecting whether the next turn will breach the budget after the model&amp;rsquo;s reply. The estimate is the current buffer size plus a conservative &lt;code&gt;max_tokens&lt;/code&gt; bound on output plus the expected tool-result size from the most recent &lt;code&gt;tool_use&lt;/code&gt; block. The pros: the user-facing turn never gets surprise latency, and there is always head-room for the model to reply. The cons: some compactions are wasted (the prediction was conservative), and the trigger requires a reasonably accurate token estimator plus policy about max output size, which often lives outside the core compaction module.&lt;/p&gt;&#10;&lt;p&gt;The defensible production pattern is &lt;strong&gt;preemptive as primary, reactive as fallback&lt;/strong&gt;. Run preemptive at ~70% of effective context; keep reactive as a backstop at ~95% for cases where the estimator misses a heavy-tailed tool result. This mirrors generational garbage collection: frequent cheap cleanup with a less frequent full fallback.&lt;/p&gt;&#10;&lt;h2 id="cache-aware-compaction-surgical-deletion"&gt;Cache-aware compaction: surgical deletion&lt;/h2&gt;&#10;&lt;p&gt;Compaction should preserve stable cached prefixes. Rewriting the whole history invalidates &lt;a href="https://jatinbansal.com/ai-engineering/prompt-caching/"&gt;prompt-cache&lt;/a&gt; reuse after the first changed token. Prefer removing superseded tool results or replacing an old range with an immutable summary block.&lt;/p&gt;&#10;&lt;p&gt;Three patterns ship in production:&lt;/p&gt;&#10;&lt;p&gt;&lt;strong&gt;Surgical tool-result deletion&lt;/strong&gt;, system prompt and assistant turns preserved. The cheapest operation: identify tool-call/tool-result pairs where the result has since been superseded (e.g., five &lt;code&gt;Read&lt;/code&gt; calls on the same file; keep the latest, replace the earlier ones with &lt;code&gt;[result superseded; see turn N]&lt;/code&gt;). The system prompt, assistant reasoning, and the recent tail are unchanged; the cacheable prefix grows monotonically. This is what &lt;a href="https://platform.claude.com/docs/en/build-with-claude/context-editing"&gt;Anthropic&amp;rsquo;s &lt;code&gt;clear_tool_uses_20250919&lt;/code&gt;&lt;/a&gt; does at the API level and what Claude Code&amp;rsquo;s micro-compact does client-side. Compression ratio is modest (10-30%), but the cache hit rate stays high, and the operation can run frequently.&lt;/p&gt;&#10;&lt;p&gt;&lt;strong&gt;Append-only summary block, untouched tail.&lt;/strong&gt; When micro-compaction isn&amp;rsquo;t enough: write a new &amp;ldquo;session summary&amp;rdquo; block at a stable position (right after the system prompt), and drop the messages it summarizes. The summary, once written, is treated as immutable for the next K turns; it doesn&amp;rsquo;t get re-summarized on every turn; the cache treats it as a stable prefix. The conversation tail after the summary keeps growing and benefits from incremental caching. The Anthropic beta&amp;rsquo;s &lt;code&gt;pause_after_compaction&lt;/code&gt; option exists for this pattern: pause after the summary is generated, let the client preserve any instruction-oriented messages, then continue. Compression ratio is high (80-95%); cache invalidation cost is paid once per K turns rather than every turn.&lt;/p&gt;&#10;&lt;p&gt;&lt;strong&gt;Anti-pattern: summary-as-system-prompt mutation.&lt;/strong&gt; Some harnesses fold the running summary into the system prompt, mutating it every cycle. This is the worst thing you can do for the cache; the system prompt is the most-cached prefix, and rewriting it every turn recomputes the entire prefix on every turn. The cost graph reads &amp;ldquo;we turned on caching&amp;rdquo; but the bill stays flat. The &lt;a href="https://jatinbansal.com/ai-engineering/agent-harness-anatomy/"&gt;harness anatomy article&lt;/a&gt; flagged this; it bears repeating because every team seems to invent it independently.&lt;/p&gt;&#10;&lt;p&gt;Rule of thumb: &lt;strong&gt;compress as far from the cache-hit prefix as possible; rewrite as little of the prefix as possible; treat the summary block as an immutable record between compactions, not a running state.&lt;/strong&gt;&lt;/p&gt;&#10;&lt;h2 id="error-recovery-circuit-breakers-and-snapshot-rollback"&gt;Error recovery: circuit breakers and snapshot-rollback&lt;/h2&gt;&#10;&lt;p&gt;Compaction can fail because the summarizer is unavailable, its output violates the schema, required fields are empty, its own context limit is exceeded, or the compacted buffer remains too large. Each case needs a typed recovery path.&lt;/p&gt;&#10;&lt;p&gt;Take a snapshot before mutating the live buffer. Run the summarizer against that snapshot, validate the schema and required fields, then swap the compacted buffer atomically. On failure, retain the original. &lt;a href="https://jatinbansal.com/ai-engineering/long-horizon-reliability/"&gt;Long-horizon checkpointing&lt;/a&gt; applies the same commit discipline at a larger scope.&lt;/p&gt;&#10;&lt;p&gt;&lt;strong&gt;Circuit breaker around compaction failures.&lt;/strong&gt; Maintain a per-session counter of consecutive failures. Trip at N=3 (a defensible default; lower if throughput-bounded, higher if the summarizer is flaky). When tripped, do not retry compaction; fall back to a degraded strategy and emit a structured failure. The breaker&amp;rsquo;s purpose is to break the &lt;em&gt;infinite-failure loop&lt;/em&gt;: a session that fails once is recoverable; a session that fails repeatedly while burning the full timeout window each time is a service outage. After M turns of cool-down, reset.&lt;/p&gt;&#10;&lt;p&gt;&lt;strong&gt;Lossy truncation as the last-resort fallback.&lt;/strong&gt; When the breaker is tripped or the summarizer is unreachable, fall back to unconditional head-tail truncation: keep the system prompt, keep the most recent K turns, drop everything in the middle. This is the &lt;a href="https://jatinbansal.com/ai-engineering/short-term-memory/"&gt;default eviction policy from short-term memory&lt;/a&gt;, used here as the fallback when the smarter policy fails. The agent loses semantic context but the session stays alive; and a degraded session is recoverable; a crashed one is not. The fallback should be loud: log the event, expose a UI indicator, mark the failure for downstream review.&lt;/p&gt;&#10;&lt;p&gt;&lt;strong&gt;The wedged-buffer escape hatch.&lt;/strong&gt; For the case where the summarizer itself can&amp;rsquo;t fit the buffer in its context: either keep a secondary summarizer with a larger context window, or run a &lt;em&gt;chunked-and-merge&lt;/em&gt; pass that splits the buffer in half, summarizes each half independently, then summarizes the summaries. The chunked-and-merge pattern is the same shape as &lt;a href="https://python.langchain.com/docs/how_to/summarize_map_reduce/"&gt;map-reduce summarization&lt;/a&gt; and is the right recovery primitive when the foreground model&amp;rsquo;s context exceeds the summarizer&amp;rsquo;s.&lt;/p&gt;&#10;&lt;h2 id="append-only-memory-journals-the-architectural-alternative"&gt;Append-only memory journals: the architectural alternative&lt;/h2&gt;&#10;&lt;p&gt;A radically different design ports the database log-vs-LSM-tree decision: instead of compacting the live buffer, run the live buffer at &lt;em&gt;near-zero retention&lt;/em&gt; and write everything important to an external &lt;strong&gt;append-only memory journal&lt;/strong&gt;, queried at retrieval time rather than loaded wholesale.&lt;/p&gt;&#10;&lt;p&gt;Mechanically: every turn, before the model call, the harness extracts decisions, files, errors, or load-bearing facts and appends them to a per-session journal (JSONL file, Postgres table, vector store; substrate-agnostic). The live buffer stays short; last 10-20 turns; by aggressive truncation. When the model needs earlier context, it queries the journal via a &lt;code&gt;recall&lt;/code&gt; tool the harness exposes. The journal grows linearly; the live buffer is bounded by design.&lt;/p&gt;&#10;&lt;p&gt;The journal model &lt;em&gt;replaces the orchestration-failure surface with a retrieval-quality surface&lt;/em&gt;: no compaction to fail, but a recall query whose quality determines whether the agent finds the relevant fact. It &lt;em&gt;makes the trade-off explicit&lt;/em&gt;: the operator can inspect, query, and audit the journal; versus a summary block whose contents are at the summarizer&amp;rsquo;s discretion. And it &lt;em&gt;plays well with &lt;a href="https://jatinbansal.com/ai-engineering/sleep-time-compute/"&gt;sleep-time compute&lt;/a&gt;&lt;/em&gt;: the journal is exactly the artifact an offline consolidation pass needs.&lt;/p&gt;&#10;&lt;p&gt;The pattern shows up under several names in 2026 production systems. &lt;a href="https://doug.sh/posts/give-your-coding-agent-a-journal/"&gt;Doug Turnbull&amp;rsquo;s &amp;ldquo;give your coding agent a journal&amp;rdquo;&lt;/a&gt; is the cleanest articulation; an agent maintains a journal file in the working directory, one entry per significant action, queried when it needs to remember. &lt;a href="https://github.com/joshuadavidthomas/opencode-agent-memory"&gt;OpenCode&amp;rsquo;s append-only journal blocks&lt;/a&gt; productize the idea with semantic search. &lt;a href="https://docs.langchain.com/oss/python/langgraph/add-memory"&gt;LangGraph&amp;rsquo;s checkpoint-and-store split&lt;/a&gt; implements the architectural distinction at the framework level. The unifying claim: &lt;em&gt;aggressive truncation plus a queryable external journal beats summarization-in-place for any task where audit and reproducibility matter more than the gist surviving in prose&lt;/em&gt;.&lt;/p&gt;&#10;&lt;p&gt;The journal wins for coding agents (audit trail needs to survive verbatim), compliance workflows (regulators want exact actions, not paraphrases), long-horizon research (the journal is the research log), and multi-session agents (journals are naturally cross-session). Compaction still wins for open-ended conversational agents where the gist is what matters, latency-sensitive interactive agents where the journal round-trip dominates, and single-session ephemeral workflows. The patterns are complementary; the production answer for most agents is both; a journal for durable state, compaction for the live buffer&amp;rsquo;s working set.&lt;/p&gt;&#10;&lt;h2 id="a-preemptive-cache-aware-compactor-in-python"&gt;A preemptive, cache-aware compactor in Python&lt;/h2&gt;&#10;&lt;p&gt;Realizes the full orchestration shape: preemptive triggering, snapshot-and-rollback, circuit-breaker recovery, cache-aware mutation. Uses the &lt;a href="https://github.com/anthropics/anthropic-sdk-python"&gt;Anthropic SDK&lt;/a&gt;.&lt;/p&gt;&#10;&lt;figure class="nb-code"&gt;&#10; &lt;header class="nb-code-chrome"&gt;&#10; &lt;span class="nb-code-label"&gt;python&lt;/span&gt;&#10; &#10; &lt;/header&gt;&#10; &lt;div class="nb-code-body"&gt;&lt;div class="highlight"&gt;&lt;div class="chroma"&gt;&#10;&lt;table class="lntable"&gt;&lt;tr&gt;&lt;td class="lntd"&gt;&#10;&lt;pre tabindex="0" class="chroma"&gt;&lt;code&gt;&lt;span class="lnt"&gt; 1&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 2&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 3&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 4&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 5&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 6&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 7&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 8&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 9&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 10&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 11&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 12&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 13&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 14&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 15&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 16&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 17&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 18&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 19&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 20&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 21&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 22&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 23&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 24&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 25&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 26&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 27&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 28&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 29&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 30&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 31&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 32&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 33&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 34&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 35&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 36&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 37&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 38&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 39&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 40&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 41&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 42&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 43&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 44&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 45&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 46&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 47&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 48&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 49&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 50&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 51&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 52&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 53&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 54&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 55&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 56&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 57&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 58&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 59&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 60&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 61&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 62&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 63&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 64&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 65&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 66&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 67&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 68&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 69&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 70&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 71&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 72&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 73&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 74&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 75&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 76&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 77&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 78&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 79&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 80&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 81&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 82&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 83&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 84&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 85&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 86&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 87&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 88&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 89&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 90&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 91&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 92&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 93&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 94&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 95&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 96&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 97&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 98&#10;&lt;/span&gt;&lt;span class="lnt"&gt; 99&#10;&lt;/span&gt;&lt;span class="lnt"&gt;100&#10;&lt;/span&gt;&lt;span class="lnt"&gt;101&#10;&lt;/span&gt;&lt;span class="lnt"&gt;102&#10;&lt;/span&gt;&lt;span class="lnt"&gt;103&#10;&lt;/span&gt;&lt;span class="lnt"&gt;104&#10;&lt;/span&gt;&lt;span class="lnt"&gt;105&#10;&lt;/span&gt;&lt;span class="lnt"&gt;106&#10;&lt;/span&gt;&lt;span class="lnt"&gt;107&#10;&lt;/span&gt;&lt;span class="lnt"&gt;108&#10;&lt;/span&gt;&lt;span class="lnt"&gt;109&#10;&lt;/span&gt;&lt;span class="lnt"&gt;110&#10;&lt;/span&gt;&lt;span class="lnt"&gt;111&#10;&lt;/span&gt;&lt;span class="lnt"&gt;112&#10;&lt;/span&gt;&lt;span class="lnt"&gt;113&#10;&lt;/span&gt;&lt;span class="lnt"&gt;114&#10;&lt;/span&gt;&lt;span class="lnt"&gt;115&#10;&lt;/span&gt;&lt;span class="lnt"&gt;116&#10;&lt;/span&gt;&lt;span class="lnt"&gt;117&#10;&lt;/span&gt;&lt;span class="lnt"&gt;118&#10;&lt;/span&gt;&lt;span class="lnt"&gt;119&#10;&lt;/span&gt;&lt;span class="lnt"&gt;120&#10;&lt;/span&gt;&lt;span class="lnt"&gt;121&#10;&lt;/span&gt;&lt;span class="lnt"&gt;122&#10;&lt;/span&gt;&lt;span class="lnt"&gt;123&#10;&lt;/span&gt;&lt;span class="lnt"&gt;124&#10;&lt;/span&gt;&lt;span class="lnt"&gt;125&#10;&lt;/span&gt;&lt;span class="lnt"&gt;126&#10;&lt;/span&gt;&lt;span class="lnt"&gt;127&#10;&lt;/span&gt;&lt;span class="lnt"&gt;128&#10;&lt;/span&gt;&lt;span class="lnt"&gt;129&#10;&lt;/span&gt;&lt;span class="lnt"&gt;130&#10;&lt;/span&gt;&lt;span class="lnt"&gt;131&#10;&lt;/span&gt;&lt;span class="lnt"&gt;132&#10;&lt;/span&gt;&lt;span class="lnt"&gt;133&#10;&lt;/span&gt;&lt;span class="lnt"&gt;134&#10;&lt;/span&gt;&lt;span class="lnt"&gt;135&#10;&lt;/span&gt;&lt;span class="lnt"&gt;136&#10;&lt;/span&gt;&lt;span class="lnt"&gt;137&#10;&lt;/span&gt;&lt;span class="lnt"&gt;138&#10;&lt;/span&gt;&lt;span class="lnt"&gt;139&#10;&lt;/span&gt;&lt;span class="lnt"&gt;140&#10;&lt;/span&gt;&lt;span class="lnt"&gt;141&#10;&lt;/span&gt;&lt;span class="lnt"&gt;142&#10;&lt;/span&gt;&lt;span class="lnt"&gt;143&#10;&lt;/span&gt;&lt;span class="lnt"&gt;144&#10;&lt;/span&gt;&lt;span class="lnt"&gt;145&#10;&lt;/span&gt;&lt;span class="lnt"&gt;146&#10;&lt;/span&gt;&lt;span class="lnt"&gt;147&#10;&lt;/span&gt;&lt;span class="lnt"&gt;148&#10;&lt;/span&gt;&lt;span class="lnt"&gt;149&#10;&lt;/span&gt;&lt;span class="lnt"&gt;150&#10;&lt;/span&gt;&lt;span class="lnt"&gt;151&#10;&lt;/span&gt;&lt;span class="lnt"&gt;152&#10;&lt;/span&gt;&lt;span class="lnt"&gt;153&#10;&lt;/span&gt;&lt;span class="lnt"&gt;154&#10;&lt;/span&gt;&lt;span class="lnt"&gt;155&#10;&lt;/span&gt;&lt;span class="lnt"&gt;156&#10;&lt;/span&gt;&lt;span class="lnt"&gt;157&#10;&lt;/span&gt;&lt;span class="lnt"&gt;158&#10;&lt;/span&gt;&lt;span class="lnt"&gt;159&#10;&lt;/span&gt;&lt;span class="lnt"&gt;160&#10;&lt;/span&gt;&lt;span class="lnt"&gt;161&#10;&lt;/span&gt;&lt;span class="lnt"&gt;162&#10;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/td&gt;&#10;&lt;td class="lntd"&gt;&#10;&lt;pre tabindex="0" class="chroma"&gt;&lt;code class="language-python" data-lang="python"&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="c1"&gt;# pip install &amp;#34;anthropic&amp;gt;=0.89.0&amp;#34;&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="nn"&gt;copy&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="nn"&gt;json&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="nn"&gt;logging&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="nn"&gt;time&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="kn"&gt;from&lt;/span&gt; &lt;span class="nn"&gt;dataclasses&lt;/span&gt; &lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;dataclass&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;field&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="kn"&gt;from&lt;/span&gt; &lt;span class="nn"&gt;typing&lt;/span&gt; &lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;Any&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="kn"&gt;from&lt;/span&gt; &lt;span class="nn"&gt;anthropic&lt;/span&gt; &lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;Anthropic&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="n"&gt;log&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;logging&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;getLogger&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;&amp;#34;compactor&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="n"&gt;client&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;Anthropic&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="n"&gt;SUMMARIZER_MODEL&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s2"&gt;&amp;#34;claude-haiku-4-5&amp;#34;&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="n"&gt;SUMMARY_PROMPT&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s2"&gt;&amp;#34;&amp;#34;&amp;#34;You are compacting an agent conversation. Read the messages and emit a JSON object with these keys exactly. Never paraphrase file paths, error codes, function names, or version numbers — copy them verbatim.&#10;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="s2"&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="s2"&gt;{&amp;#34;session_intent&amp;#34;: &amp;#34;&amp;#34;, &amp;#34;files_touched&amp;#34;: [], &amp;#34;decisions&amp;#34;: [], &amp;#34;pending_questions&amp;#34;: [], &amp;#34;next_steps&amp;#34;: []}&#10;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="s2"&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="s2"&gt;MESSAGES:&#10;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="si"&gt;{messages}&lt;/span&gt;&lt;span class="s2"&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="s2"&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="s2"&gt;Respond with the JSON object only.&#10;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="s2"&gt;&amp;#34;&amp;#34;&amp;#34;&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="nd"&gt;@dataclass&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="k"&gt;class&lt;/span&gt; &lt;span class="nc"&gt;CompactionConfig&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;soft_watermark&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;float&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mf"&gt;0.70&lt;/span&gt; &lt;span class="c1"&gt;# preemptive trigger&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;hard_watermark&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;float&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mf"&gt;0.95&lt;/span&gt; &lt;span class="c1"&gt;# reactive fallback&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;tail_keep&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;int&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mi"&gt;10&lt;/span&gt; &lt;span class="c1"&gt;# messages preserved verbatim after compaction&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;max_failures&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;int&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mi"&gt;3&lt;/span&gt; &lt;span class="c1"&gt;# circuit-breaker trip threshold&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;cooldown_turns&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;int&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mi"&gt;5&lt;/span&gt; &lt;span class="c1"&gt;# cool-down before breaker resets&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;summarizer_max_tokens&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;int&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mi"&gt;1024&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;output_headroom_tokens&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;int&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mi"&gt;4096&lt;/span&gt; &lt;span class="c1"&gt;# reserved for the foreground reply&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="nd"&gt;@dataclass&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="k"&gt;class&lt;/span&gt; &lt;span class="nc"&gt;CompactorState&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;consecutive_failures&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;int&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;breaker_tripped_at_turn&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;int&lt;/span&gt; &lt;span class="o"&gt;|&lt;/span&gt; &lt;span class="kc"&gt;None&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="kc"&gt;None&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;last_summary&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;dict&lt;/span&gt; &lt;span class="o"&gt;|&lt;/span&gt; &lt;span class="kc"&gt;None&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="kc"&gt;None&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;last_summary_at_turn&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;int&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;turn&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;int&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;estimate_tokens&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;messages&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;list&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="nb"&gt;dict&lt;/span&gt;&lt;span class="p"&gt;])&lt;/span&gt; &lt;span class="o"&gt;-&amp;gt;&lt;/span&gt; &lt;span class="nb"&gt;int&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="c1"&gt;# ~4 chars per token; replace with the provider&amp;#39;s tokenizer in production.&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="nb"&gt;sum&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nb"&gt;len&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;json&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;dumps&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;m&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;m&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;messages&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;//&lt;/span&gt; &lt;span class="mi"&gt;4&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;project_next_turn&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;messages&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;list&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="nb"&gt;dict&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;pending_tool_result_estimate&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;int&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;cfg&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;CompactionConfig&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;-&amp;gt;&lt;/span&gt; &lt;span class="nb"&gt;int&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;estimate_tokens&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;messages&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="n"&gt;pending_tool_result_estimate&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="n"&gt;cfg&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;output_headroom_tokens&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="p"&gt;)&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="k"&gt;class&lt;/span&gt; &lt;span class="nc"&gt;CompactorError&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="ne"&gt;Exception&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt; &lt;span class="o"&gt;...&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="k"&gt;class&lt;/span&gt; &lt;span class="nc"&gt;WedgedBufferError&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;CompactorError&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt; &lt;span class="o"&gt;...&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="k"&gt;class&lt;/span&gt; &lt;span class="nc"&gt;BreakerTrippedError&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;CompactorError&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt; &lt;span class="o"&gt;...&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;_run_summarizer&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;messages&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;list&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="nb"&gt;dict&lt;/span&gt;&lt;span class="p"&gt;])&lt;/span&gt; &lt;span class="o"&gt;-&amp;gt;&lt;/span&gt; &lt;span class="nb"&gt;dict&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="c1"&gt;# Validate output shape; raise on any malformed result so the caller can&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="c1"&gt;# snapshot-rollback rather than commit a broken summary.&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;body&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;json&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;dumps&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;messages&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;ensure_ascii&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="kc"&gt;False&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;resp&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;client&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;messages&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;create&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;model&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;SUMMARIZER_MODEL&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;max_tokens&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;1024&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;messages&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="p"&gt;[{&lt;/span&gt;&lt;span class="s2"&gt;&amp;#34;role&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;&amp;#34;user&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;&amp;#34;content&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;SUMMARY_PROMPT&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;format&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;messages&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;body&lt;/span&gt;&lt;span class="p"&gt;)}],&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="p"&gt;)&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;text&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;resp&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;content&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;text&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;strip&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;text&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;startswith&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;&amp;#34;```&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;text&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;text&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;split&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;&amp;#34;```&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;2&lt;/span&gt;&lt;span class="p"&gt;)[&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;lstrip&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;&amp;#34;json&lt;/span&gt;&lt;span class="se"&gt;\n&lt;/span&gt;&lt;span class="s2"&gt;&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;rstrip&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;&amp;#34;`&lt;/span&gt;&lt;span class="se"&gt;\n&lt;/span&gt;&lt;span class="s2"&gt;&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;obj&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;json&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;loads&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;text&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="c1"&gt;# malformed JSON raises here&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;required&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="s2"&gt;&amp;#34;session_intent&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;&amp;#34;files_touched&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;&amp;#34;decisions&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="s2"&gt;&amp;#34;pending_questions&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;&amp;#34;next_steps&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="ow"&gt;not&lt;/span&gt; &lt;span class="n"&gt;required&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;issubset&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;obj&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;keys&lt;/span&gt;&lt;span class="p"&gt;()):&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="k"&gt;raise&lt;/span&gt; &lt;span class="n"&gt;CompactorError&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sa"&gt;f&lt;/span&gt;&lt;span class="s2"&gt;&amp;#34;summarizer missing keys: &lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;required&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt; &lt;span class="n"&gt;obj&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;keys&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s2"&gt;&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="c1"&gt;# Empty-load-bearing-field check: a summary with no decisions and no files&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="c1"&gt;# touched after a 50-turn coding session is almost certainly a silent failure.&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="ow"&gt;not&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;obj&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;&amp;#34;decisions&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="ow"&gt;or&lt;/span&gt; &lt;span class="n"&gt;obj&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;&amp;#34;files_touched&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;])&lt;/span&gt; &lt;span class="ow"&gt;and&lt;/span&gt; &lt;span class="nb"&gt;len&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;messages&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;&amp;gt;&lt;/span&gt; &lt;span class="mi"&gt;20&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="k"&gt;raise&lt;/span&gt; &lt;span class="n"&gt;CompactorError&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;&amp;#34;summarizer returned empty load-bearing fields&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;obj&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;_chunked_and_merge&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;messages&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;list&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="nb"&gt;dict&lt;/span&gt;&lt;span class="p"&gt;])&lt;/span&gt; &lt;span class="o"&gt;-&amp;gt;&lt;/span&gt; &lt;span class="nb"&gt;dict&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="c1"&gt;# Wedged-buffer escape hatch: split, summarize halves, summarize summaries.&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;mid&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nb"&gt;len&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;messages&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;//&lt;/span&gt; &lt;span class="mi"&gt;2&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;left&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;_run_summarizer&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;messages&lt;/span&gt;&lt;span class="p"&gt;[:&lt;/span&gt;&lt;span class="n"&gt;mid&lt;/span&gt;&lt;span class="p"&gt;])&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;right&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;_run_summarizer&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;messages&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;mid&lt;/span&gt;&lt;span class="p"&gt;:])&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;merged_input&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="s2"&gt;&amp;#34;role&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;&amp;#34;user&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;&amp;#34;content&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;&amp;#34;PARTIAL SUMMARY (first half):&lt;/span&gt;&lt;span class="se"&gt;\n&lt;/span&gt;&lt;span class="s2"&gt;&amp;#34;&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="n"&gt;json&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;dumps&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;left&lt;/span&gt;&lt;span class="p"&gt;)},&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="s2"&gt;&amp;#34;role&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;&amp;#34;user&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;&amp;#34;content&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;&amp;#34;PARTIAL SUMMARY (second half):&lt;/span&gt;&lt;span class="se"&gt;\n&lt;/span&gt;&lt;span class="s2"&gt;&amp;#34;&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="n"&gt;json&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;dumps&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;right&lt;/span&gt;&lt;span class="p"&gt;)},&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="p"&gt;]&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;_run_summarizer&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;merged_input&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;_lossy_truncate&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;messages&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;list&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="nb"&gt;dict&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt; &lt;span class="n"&gt;tail_keep&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;int&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;-&amp;gt;&lt;/span&gt; &lt;span class="nb"&gt;list&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="nb"&gt;dict&lt;/span&gt;&lt;span class="p"&gt;]:&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="c1"&gt;# Last-resort fallback when the circuit breaker is tripped or summarizer fails.&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;system&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;m&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;m&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;messages&lt;/span&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;m&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;&amp;#34;role&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="s2"&gt;&amp;#34;system&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;tail&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;m&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;m&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;messages&lt;/span&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;m&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;&amp;#34;role&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;!=&lt;/span&gt; &lt;span class="s2"&gt;&amp;#34;system&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;][&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;tail_keep&lt;/span&gt;&lt;span class="p"&gt;:]&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;system&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="p"&gt;[{&lt;/span&gt;&lt;span class="s2"&gt;&amp;#34;role&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;&amp;#34;user&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;&amp;#34;content&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;&amp;#34;[earlier history truncated; &amp;#34;&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="s2"&gt;&amp;#34;compactor unavailable]&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;}]&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="n"&gt;tail&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&lt;span class="k"&gt;class&lt;/span&gt; &lt;span class="nc"&gt;Compactor&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="fm"&gt;__init__&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="bp"&gt;self&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;cfg&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;CompactionConfig&lt;/span&gt; &lt;span class="o"&gt;|&lt;/span&gt; &lt;span class="kc"&gt;None&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="kc"&gt;None&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;model_context_limit&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;int&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mi"&gt;200_000&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="bp"&gt;self&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;cfg&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;cfg&lt;/span&gt; &lt;span class="ow"&gt;or&lt;/span&gt; &lt;span class="n"&gt;CompactionConfig&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="bp"&gt;self&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;state&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;CompactorState&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="bp"&gt;self&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;limit&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;model_context_limit&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;maybe_compact&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="bp"&gt;self&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;messages&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;list&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="nb"&gt;dict&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;pending_tool_result_estimate&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;int&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;-&amp;gt;&lt;/span&gt; &lt;span class="nb"&gt;list&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="nb"&gt;dict&lt;/span&gt;&lt;span class="p"&gt;]:&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="bp"&gt;self&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;state&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;turn&lt;/span&gt; &lt;span class="o"&gt;+=&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;projected&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;project_next_turn&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;messages&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;pending_tool_result_estimate&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="bp"&gt;self&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;cfg&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="c1"&gt;# Breaker cool-down: reset after the configured number of turns.&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="bp"&gt;self&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;state&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;breaker_tripped_at_turn&lt;/span&gt; &lt;span class="ow"&gt;is&lt;/span&gt; &lt;span class="ow"&gt;not&lt;/span&gt; &lt;span class="kc"&gt;None&lt;/span&gt; &lt;span class="ow"&gt;and&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="bp"&gt;self&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;state&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;turn&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt; &lt;span class="bp"&gt;self&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;state&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;breaker_tripped_at_turn&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="o"&gt;&amp;gt;=&lt;/span&gt; &lt;span class="bp"&gt;self&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;cfg&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;cooldown_turns&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;log&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;info&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;&amp;#34;compactor.breaker_reset turn=&lt;/span&gt;&lt;span class="si"&gt;%d&lt;/span&gt;&lt;span class="s2"&gt;&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="bp"&gt;self&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;state&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;turn&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="bp"&gt;self&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;state&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;breaker_tripped_at_turn&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="kc"&gt;None&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="bp"&gt;self&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;state&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;consecutive_failures&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;projected&lt;/span&gt; &lt;span class="o"&gt;&amp;lt;&lt;/span&gt; &lt;span class="bp"&gt;self&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;limit&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="bp"&gt;self&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;cfg&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;soft_watermark&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;messages&lt;/span&gt; &lt;span class="c1"&gt;# no compaction needed&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="c1"&gt;# If the breaker is tripped, go straight to lossy truncation.&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="bp"&gt;self&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;state&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;breaker_tripped_at_turn&lt;/span&gt; &lt;span class="ow"&gt;is&lt;/span&gt; &lt;span class="ow"&gt;not&lt;/span&gt; &lt;span class="kc"&gt;None&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;log&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;warning&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;&amp;#34;compactor.breaker_open fallback=truncate turn=&lt;/span&gt;&lt;span class="si"&gt;%d&lt;/span&gt;&lt;span class="s2"&gt;&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="bp"&gt;self&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;state&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;turn&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;_lossy_truncate&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;messages&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="bp"&gt;self&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;cfg&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;tail_keep&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="c1"&gt;# Snapshot the live buffer before any mutation.&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;snapshot&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;copy&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;deepcopy&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;messages&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="k"&gt;try&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;estimate_tokens&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;messages&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;&amp;gt;&lt;/span&gt; &lt;span class="bp"&gt;self&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;limit&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="c1"&gt;# wedged-buffer escape&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;summary&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;_chunked_and_merge&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;messages&lt;/span&gt;&lt;span class="p"&gt;[:&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="bp"&gt;self&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;cfg&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;tail_keep&lt;/span&gt;&lt;span class="p"&gt;])&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="k"&gt;else&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;summary&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;_run_summarizer&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;messages&lt;/span&gt;&lt;span class="p"&gt;[:&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="bp"&gt;self&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;cfg&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;tail_keep&lt;/span&gt;&lt;span class="p"&gt;])&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="k"&gt;except&lt;/span&gt; &lt;span class="ne"&gt;Exception&lt;/span&gt; &lt;span class="k"&gt;as&lt;/span&gt; &lt;span class="n"&gt;e&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="bp"&gt;self&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;state&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;consecutive_failures&lt;/span&gt; &lt;span class="o"&gt;+=&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;log&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;error&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;&amp;#34;compactor.failure n=&lt;/span&gt;&lt;span class="si"&gt;%d&lt;/span&gt;&lt;span class="s2"&gt; err=&lt;/span&gt;&lt;span class="si"&gt;%s&lt;/span&gt;&lt;span class="s2"&gt;&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="bp"&gt;self&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;state&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;consecutive_failures&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;e&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="bp"&gt;self&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;state&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;consecutive_failures&lt;/span&gt; &lt;span class="o"&gt;&amp;gt;=&lt;/span&gt; &lt;span class="bp"&gt;self&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;cfg&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;max_failures&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="bp"&gt;self&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;state&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;breaker_tripped_at_turn&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="bp"&gt;self&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;state&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;turn&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;log&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;error&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;&amp;#34;compactor.breaker_tripped turn=&lt;/span&gt;&lt;span class="si"&gt;%d&lt;/span&gt;&lt;span class="s2"&gt;&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="bp"&gt;self&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;state&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;turn&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="c1"&gt;# Atomic rollback: original buffer is unchanged.&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;_lossy_truncate&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;snapshot&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="bp"&gt;self&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;cfg&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;tail_keep&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="c1"&gt;# Success: build the new buffer atomically.&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="bp"&gt;self&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;state&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;consecutive_failures&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="bp"&gt;self&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;state&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;last_summary&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;summary&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="bp"&gt;self&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;state&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;last_summary_at_turn&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="bp"&gt;self&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;state&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;turn&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;system&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;m&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;m&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;messages&lt;/span&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;m&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;&amp;#34;role&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="s2"&gt;&amp;#34;system&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;tail&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;messages&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="bp"&gt;self&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;cfg&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;tail_keep&lt;/span&gt;&lt;span class="p"&gt;:]&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="n"&gt;summary_block&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="s2"&gt;&amp;#34;role&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;&amp;#34;user&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="s2"&gt;&amp;#34;content&amp;#34;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;&amp;#34;PRIOR SESSION SUMMARY (immutable until next compaction):&lt;/span&gt;&lt;span class="se"&gt;\n&lt;/span&gt;&lt;span class="s2"&gt;&amp;#34;&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="n"&gt;json&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;dumps&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;summary&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;indent&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;2&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="p"&gt;}&lt;/span&gt;&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;system&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;summary_block&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="n"&gt;tail&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/td&gt;&lt;/tr&gt;&lt;/table&gt;&#10;&lt;/div&gt;&#10;&lt;/div&gt;&lt;/div&gt;&#10;&lt;/figure&gt;&#10;&lt;p&gt;The interesting parts aren&amp;rsquo;t the summarizer call; that&amp;rsquo;s the &lt;a href="https://jatinbansal.com/ai-engineering/context-compression/"&gt;context-compression article&lt;/a&gt;&amp;rsquo;s domain; but the orchestration. The snapshot-and-rollback cannot leave the buffer half-rewritten. The breaker prevents the infinite-failure loop. The lossy-truncation fallback keeps the session moving. The chunked-and-merge escape hatch handles the wedged case. The preemptive trigger gates on &lt;code&gt;projected&lt;/code&gt;, not on current buffer size, so head-room is reserved before the call.&lt;/p&gt;&#10;&lt;h2 id="further-reading"&gt;Further reading&lt;/h2&gt;&#10;&lt;ul&gt;&#10;&lt;li&gt;&lt;a href="https://platform.claude.com/docs/en/build-with-claude/compaction"&gt;Anthropic; Compaction documentation&lt;/a&gt;; the reference for the &lt;code&gt;compact_20260112&lt;/code&gt; server-managed strategy. The &lt;code&gt;pause_after_compaction&lt;/code&gt; option and the custom-instructions surface are the production levers worth knowing before you build a client-side compactor.&lt;/li&gt;&#10;&lt;li&gt;&lt;a href="https://developers.openai.com/api/docs/guides/compaction"&gt;OpenAI; Compaction guide for the Responses API&lt;/a&gt;; the parallel server-side primitive (&lt;code&gt;context_management.compact_threshold&lt;/code&gt;), shipped February 11, 2026. The shape is conceptually identical to Anthropic&amp;rsquo;s beta; the trigger semantics differ slightly (token-count threshold vs. trigger-with-strategy).&lt;/li&gt;&#10;&lt;li&gt;&lt;a href="https://doug.sh/posts/give-your-coding-agent-a-journal/"&gt;Doug Turnbull; &amp;ldquo;Give your coding agent a journal&amp;rdquo;&lt;/a&gt;; the cleanest articulation of the append-only-journal alternative. The &amp;ldquo;journal-file-in-the-working-directory&amp;rdquo; pattern is the operational shape every coding-agent team eventually converges on, and the post explains why.&lt;/li&gt;&#10;&lt;li&gt;&lt;a href="https://www.morphllm.com/claude-code-auto-compact"&gt;Morph LLM; &amp;ldquo;Claude Code auto-compact: what triggers it, what it loses, how to fix it&amp;rdquo;&lt;/a&gt;; the production-incident perspective on auto-compaction. The empirical observations on micro-compact vs full-compact thresholds and the cataloged loss modes are the closest thing to a public post-mortem on a widely-deployed compactor.&lt;/li&gt;&#10;&lt;li&gt;&lt;a href="https://www.philschmid.de/agent-harness-2026"&gt;Phil Schmid; &amp;ldquo;The importance of Agent Harness in 2026&amp;rdquo;&lt;/a&gt;; the broader framing of the harness as the new competitive surface. The &amp;ldquo;build-to-delete&amp;rdquo; argument and the trajectory-data point are load-bearing background for any team deciding whether to invest in a custom compactor.&lt;/li&gt;&#10;&lt;/ul&gt;&#10;</content:encoded></item></channel></rss>