Drift Detection and Regression Testing for LLM Systems

An LLM application can deteriorate while its code and aggregate score remain unchanged. The user mix may shift toward a weak category, a provider may change a model, or the evaluator may move. Drift monitoring compares current production distributions with a trusted reference; regression testing compares system versions on matched inputs.

Three kinds of drift

The signals factor into three axes. Detecting any one without the others is half-blind.

Input drift. What users are sending. The cheap shadow is category mix: every user message is classified into an error-analysis category at ingest, the per-category count is logged, and a Population Stability Index (PSI) between the current 24-hour window and the reference 7-day window is computed once an hour. PSI above 0.2 flags material drift in the literature, above 0.1 is worth a glance. The expensive shadow is embedding centroid distance: the text-embedding of every user message goes into the trace store, the current-window centroid is the mean of those embeddings, and the Euclidean (or cosine) distance to the reference-window centroid is the drift signal. Arize Phoenix’s embedding-drift module is the canonical implementation; it clusters with HDBSCAN, projects with UMAP, and exposes the centroid distance as a per-cluster trend line. The dual signal; category PSI plus centroid distance; catches both the “users are asking new things” failure mode (PSI fires; centroid moves) and the “users are asking similar things differently” mode (PSI flat; centroid moves) that a single signal would miss.

Output drift. What the model is returning. The shadow at this layer is per-output features that don’t require judging; output length distribution, refusal rate, tool-call frequency per turn, structured-output validity rate, cache hit rate. These are cheap to compute over every span, and the Kolmogorov-Smirnov test on the per-window distribution against the reference distribution is the alerting signal. A 10-point drop in tool-call frequency is the kind of failure that doesn’t show in any quality metric until support escalates it, but shows immediately in the KS test on tool-use rate. Output drift is the easiest axis to instrument and the easiest to miss because none of these signals are quality measurements; they’re correlates of quality, and a system that monitors only the judged quality score will see the regression two days after the cheap output-feature signal would have fired.

Concept drift. Whether the relationship between input and output is what it used to be; the actual quality measurement. The shadow at this layer is the judged score on a fixed evaluation slice, run continuously over a rolling sample of production traces. Sample 1% of traces per category per hour, run the pinned judge against them, compute the per-category mean score and bootstrapped confidence interval, alert when the current-window CI lower bound falls below the reference-window CI upper bound. This is the most expensive signal; every alert costs you the judge-call price for the sampled traces, times the categories, times the windows; but it’s the only one that measures the thing you care about. The cheaper signals on the other two axes are correlates; this one is the target.

The discipline is to gate the expensive signal on the cheap ones. The hourly judge-replay runs against full samples; the category-PSI and centroid-distance checks run continuously and, when they fire, trigger an immediate judge re-score on the drifted slice. The combination; continuous cheap monitoring plus on-demand expensive judging; keeps the bill bounded while keeping the latency to alert under an hour for the drift modes that matter.

Statistical tests and embedding shift

A handful of statistical tools cover the LLM drift surface; pick by data shape.

• Population Stability Index (PSI) for categorical distributions; error-analysis category mix, tool-call distribution, finish-reason distribution. PSI is Σ (current_pct - reference_pct) * ln(current_pct / reference_pct) over the bins. Material drift threshold ~0.2, alert threshold ~0.1, fine-grained inspection above 0.05. Cheap; runs on aggregates.
• Kolmogorov-Smirnov (KS) test for continuous distributions; output length, latency, per-span cost, judge score. KS reports the max absolute difference between the two empirical CDFs and a p-value; the p-value is the alert signal but the effect size (the D statistic) is what matters at scale. Cheap; runs on samples.
• Kullback-Leibler (KL) divergence for probability distributions where you have explicit probabilities; token-level distributions if you have logprobs, soft-label outputs from a classifier head. KL is Σ p(i) * log(p(i) / q(i)); it’s asymmetric, and the symmetric variant (Jensen-Shannon) is the more useful drift metric in practice. Cheaper than KS; needs probabilities.
• Embedding centroid distance: Euclidean or cosine distance between current-window and reference-window mean embeddings. The bare metric is the cheap version; cluster-aware variants (compute per-cluster centroids after HDBSCAN, alert when any cluster’s centroid moves or a new cluster appears with mass above a threshold) are the production version. Arize’s documented implementation is the cleanest reference.
• Paired-bootstrap confidence intervals for regression testing; the right tool when you’re comparing two systems on the same fixed inputs. Sample with replacement from the golden-set rows, recompute the per-system mean score on each bootstrap sample, take the 2.5th and 97.5th percentiles of the difference distribution as the 95% CI on the delta. A non-zero-containing CI is the significance gate. Bootstrap because the per-row scores aren’t normally distributed and t-tests overstate confidence.
• Domain-classifier drift: Evidently’s approach for raw text: train a binary classifier to distinguish reference-window text from current-window text; if the classifier hits ROC AUC > ~0.7, the distributions are separable and drift is happening. Useful when the input doesn’t reduce cleanly to categories or embeddings (e.g. raw markdown documents, long-form code).

Pick the tool by the data shape, not by what’s fashionable. Don’t reach for KL divergence on a categorical mix where PSI is the bog-standard answer, and don’t run a KS test on a 7-bin distribution where PSI is more interpretable.

Test a model upgrade

A model upgrade; Sonnet 4.6 to Opus 4.7, Haiku 4.5 to whatever Anthropic ships next; is the most common shipping event that breaks an LLM app, because the provider’s pre-release evals tell you about their benchmarks, not yours. Anthropic’s recent deprecation cadence; Sonnet 4 and Opus 4 retired June 15, 2026; Sonnet 4.5 cutover by May 18; means most production teams will run this protocol every few months for the lifetime of the application. The protocol that holds up has four phases.

Phase 1: offline regression on the golden set. Run the golden set against both the baseline model and the candidate. Score every row with the same pinned judge (the LLM-as-judge article is load-bearing here; judge calibration changes invalidate the comparison). Compute paired-bootstrap CIs on the per-category quality delta. The gate: every category’s delta CI either contains zero (no regression) or is strictly above zero (improvement); any category whose CI is strictly negative is a regression and the candidate fails the gate. Aggregate-only gating misses category-level regressions; gate per category.

Phase 2: shadow run on historical production traffic. Pull the last 7 days of production traces, replay the inputs through the candidate without serving the outputs, and run the same judge against both baseline outputs (already scored, in the trace store) and candidate outputs. The signal here is two-fold: the same paired-bootstrap CI on per-category deltas (now over real production distribution, not the curated golden set), and a behavioral diff; output length, tool-call frequency, refusal rate, structured-output validity; that catches behavioral regressions the judge isn’t sensitive to. A model that emits 40% more tokens to say the same thing is a cost regression even if the quality is identical, and the behavioral diff is what surfaces it.

Phase 3: canary deploy. Route 1% of live traffic to the candidate, monitor for an hour, judge a 10% sample, gate on the same per-category CI test. If it passes, step to 5%, monitor for an hour, judge. Continue to 20%, 50%, 100% with the same checkpoint at each stop. The increments and dwell times are the progressive-delivery primitives borrowed verbatim from microservices; the only LLM-specific adjustment is that the per-window judged sample is the gate, not request-level quality. A clean canary fails into rollback by flipping a single config flag, which is why the routing has to be a config knob, not a deploy step.

Phase 4: dual-run window and decommission. After cutover, keep the baseline warm for at least a week; same prompt, same routing infrastructure, just not the default route. The dual-run window catches the slower drift modes that the canary missed: user behavior changing in response to the new model’s verbosity, downstream consumers depending on a specific output shape, cache patterns that the canary’s short window didn’t exercise. After the dual-run window passes clean, decommission the baseline and reclaim the resources; the provider’s pricing often makes the dual-run window a small cost relative to the rollback insurance it buys.

A subtler point. The protocol works for model upgrades and prompt edits with no modification; both are changes to the per-call surface and both fail in the same way. It does not directly cover retrieval changes (a new chunking strategy, a re-embedded vector store, a reranker swap) because the change affects the context the model sees and the offline golden set’s recall metrics matter more than the judged output. The RAG evaluation article is the right specialization there; the protocol structure is the same, the metrics are different. Similarly, embedding-model upgrades need the dual-index migration pattern; the per-call regression protocol catches the symptom but not the cause.

Code: an embedding-drift detector over a trace store in Python

The detector below pulls user messages from the Langfuse trace store, embeds them, computes PSI over the categorical mix, and centroid-distance over the embeddings, and emits a structured drift report. Install: pip install langfuse openai numpy scipy.

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121

# pip install langfuse openai numpy scipy
import os
from datetime import datetime, timedelta, timezone
from collections import Counter

import numpy as np
from langfuse import get_client
from openai import OpenAI
from scipy.stats import ks_2samp

langfuse = get_client()
openai_client = OpenAI()
EMBED_MODEL = "text-embedding-3-large"


def fetch_user_messages(start: datetime, end: datetime, limit: int = 5000):
    """Pull user-turn inputs from the trace store as (text, category, output_len) tuples."""
    traces = langfuse.api.trace.list(
        from_timestamp=start, to_timestamp=end, limit=limit,
    ).data
    rows = []
    for t in traces:
        if not t.input:
            continue
        text = str(t.input.get("user_msg", ""))
        category = (t.metadata or {}).get("category", "unknown")
        output_len = len(str(t.output or ""))
        rows.append((text, category, output_len))
    return rows


def psi(reference: list, current: list, bins: int | None = None) -> float:
    """Population Stability Index over categorical mixes.

    For a categorical input (list of labels), bins is the union of labels.
    For a continuous input, pass bins as an integer and we histogram.
    """
    if bins is None:
        # Categorical: bins are the union of unique values.
        labels = sorted(set(reference) | set(current))
        ref_counts = Counter(reference)
        cur_counts = Counter(current)
        ref_total = max(1, len(reference))
        cur_total = max(1, len(current))
        psi_total = 0.0
        for label in labels:
            ref_pct = (ref_counts[label] + 1e-6) / ref_total  # smoothing
            cur_pct = (cur_counts[label] + 1e-6) / cur_total
            psi_total += (cur_pct - ref_pct) * np.log(cur_pct / ref_pct)
        return float(psi_total)
    # Continuous: equal-width bins over the reference range.
    edges = np.linspace(min(reference), max(reference), bins + 1)
    ref_hist, _ = np.histogram(reference, bins=edges)
    cur_hist, _ = np.histogram(current, bins=edges)
    ref_pct = (ref_hist + 1e-6) / max(1, ref_hist.sum())
    cur_pct = (cur_hist + 1e-6) / max(1, cur_hist.sum())
    return float(np.sum((cur_pct - ref_pct) * np.log(cur_pct / ref_pct)))


def embed_batch(texts: list[str]) -> np.ndarray:
    """Batch-embed texts; production should retry/backoff."""
    resp = openai_client.embeddings.create(model=EMBED_MODEL, input=texts)
    return np.array([e.embedding for e in resp.data])


def centroid_distance(ref_emb: np.ndarray, cur_emb: np.ndarray) -> float:
    """Cosine distance between the two centroids."""
    ref_c = ref_emb.mean(axis=0)
    cur_c = cur_emb.mean(axis=0)
    cos = np.dot(ref_c, cur_c) / (np.linalg.norm(ref_c) * np.linalg.norm(cur_c))
    return float(1.0 - cos)


def run_drift_check(reference_days: int = 7, current_hours: int = 24) -> dict:
    end = datetime.now(timezone.utc)
    ref_start = end - timedelta(days=reference_days) - timedelta(hours=current_hours)
    ref_end = end - timedelta(hours=current_hours)
    ref = fetch_user_messages(ref_start, ref_end)
    cur = fetch_user_messages(ref_end, end)
    if not ref or not cur:
        return {"status": "insufficient_data", "ref_n": len(ref), "cur_n": len(cur)}

    # Category PSI
    cat_psi = psi([r[1] for r in ref], [c[1] for c in cur])

    # Output-length KS
    ks_stat, ks_p = ks_2samp([r[2] for r in ref], [c[2] for c in cur])

    # Embedding centroid distance — sample to bound cost
    ref_sample = [r[0] for r in ref[: min(500, len(ref))]]
    cur_sample = [c[0] for c in cur[: min(500, len(cur))]]
    ref_emb = embed_batch(ref_sample)
    cur_emb = embed_batch(cur_sample)
    cent_dist = centroid_distance(ref_emb, cur_emb)

    # Decision: any single signal can fire, but the action is "investigate the slice".
    alerts = []
    if cat_psi > 0.2:
        alerts.append(f"category_psi={cat_psi:.3f} (material)")
    elif cat_psi > 0.1:
        alerts.append(f"category_psi={cat_psi:.3f} (worth_glance)")
    if ks_p < 0.01 and ks_stat > 0.1:
        alerts.append(f"output_length_ks_D={ks_stat:.3f}, p={ks_p:.4f}")
    if cent_dist > 0.05:
        alerts.append(f"embedding_centroid_distance={cent_dist:.4f}")

    return {
        "status": "drift" if alerts else "stable",
        "alerts": alerts,
        "category_psi": cat_psi,
        "output_length_ks_stat": float(ks_stat),
        "output_length_ks_p": float(ks_p),
        "embedding_centroid_distance": cent_dist,
        "n_reference": len(ref),
        "n_current": len(cur),
    }


if __name__ == "__main__":
    import json
    print(json.dumps(run_drift_check(), indent=2))

The three signals; category PSI, output-length KS, embedding-centroid distance; are independent. Any one firing is enough to investigate; the conjunction is the strong signal. The thresholds are starting points (PSI 0.2 / 0.1 from the scorecard literature, KS p<0.01 and D>0.1 to filter weak effects); recalibrate against your own false-positive rate after a month. The embedding step is sampled to 500 messages per window; at $0.13 per million tokens for text-embedding-3-large and ~80 tokens per message, an hourly run costs around $0.50 a day, which is the cheap shadow signal. The expensive judged signal sits behind this and runs only when an alert fires.

Further reading

• Chip Huyen; Data Distribution Shifts and Monitoring; the ML-systems reference for the conceptual frame: covariate shift, concept drift, label shift, and the monitoring pipeline that catches each. Pre-LLM but the taxonomy transfers cleanly, and the discussion of when to recompute baselines and how to gate retraining is the load-bearing piece for the LLM application.
• Evidently; Monitoring NLP models in production: detecting drift in text data; the practitioner’s tutorial on text-data drift, including the domain-classifier approach, descriptor monitoring, and the embedding-drift comparison. The cleanest single resource for the input-distribution-drift problem on raw text.
• Arize; Embedding Drift documentation; Arize’s canonical writeup of centroid-distance-based drift on embedding inputs, with the HDBSCAN clustering pipeline and the UMAP projection that the Phoenix UI exposes. Pairs well with Aparna Dhinakaran’s measuring embedding drift for the data-science framing.
• Eugene Yan; Patterns for Building LLM-Based Systems; the cross-pillar pattern catalogue, with the guardrails and evals sections covering the steady-state monitoring discipline this article instantiates. The connective tissue piece for evals-plus-observability-plus-drift as one stack.
• Ma et al.; (Why) Is My Prompt Getting Worse? Rethinking Regression Testing for Evolving LLM APIs; the academic case for prompt-level regression testing across silent model updates, with measurements of how much the same prompt drifts across provider versions and the case for paired, fixed-input regression suites. Maps directly to the protocol in this article.