$ cat ai-engineering/eval-driven-development.md

Eval-Driven Development for LLM Systems

Evaluation-driven development with error analysis, golden sets, layered checks, and CI gates.

Jatin Bansal@blog:~/ai-engineering$ open eval-driven-development

LLM application regressions often appear as distribution shifts rather than deterministic failures. A model upgrade may alter tool calls, a prompt edit may break one query class, and a retriever change may shift the supplied context. A versioned eval suite makes these changes visible before promotion.

Treat evals as the release contract

Eval-driven development treats a versioned eval suite as the release contract for an LLM application. The suite grows from failures found in real traces because the input distribution is too broad to specify in advance. It measures quality together with cost and latency, so an expensive quality gain can still count as a regression. A deterministic slice runs on every pull request, while slower LLM-judged checks run nightly or before release. Both enforce explicit promotion thresholds.

A note on the name. “Eval-driven development” gestures at TDD, and that gesture is partly misleading. Hamel Husain and Shreya Shankar argue strongly that pure write-evals-first does not work for LLM systems, because the surface area is unbounded and you cannot enumerate failure modes from a spec sheet. The right loop is error analysis first, evals second; look at traces, categorize what is actually breaking, then write an eval that catches the category. The TDD shape (write the failing test, fix the bug, commit both together) survives; the “imagine all the failures” shape does not. Call it eval-driven development as long as you remember which half of the analogy holds.

Start with error analysis

The loop that produces a useful eval suite is concrete enough to write down. Five steps, repeated weekly during early development, monthly once the system stabilizes:

Capture traces: Log every model call with its inputs, retrieved context, output, latency, token counts, and cost. Sample randomly or stratify by query type. The production tracing layer covers span shape, OTel GenAI semantic conventions, sampling, and platform choices. Evals only require that the traces are queryable.
Open-coded review. A single domain expert (not three) reads 50–100 traces back-to-back and writes free-text notes about every problem they see. No category schema yet. This is the “journaling” pass Hamel and Shreya borrow from qualitative research; the goal is to surface failure modes you didn’t know existed.
Axial coding. Cluster the open-coded notes into categories: “wrong product name,” “ignored the prior turn,” “output JSON missed a required field,” “took the long path through the tool call graph.” Most categories will be small; a few will dominate the volume. The categories are now your taxonomy.
Write the eval row. For each category, pick the cheapest assertion that catches it. Schema validation for the JSON case. Substring match for the product-name case. Tool-call-sequence assertion for the wrong-path case. LLM-as-judge with a specific rubric for the “ignored prior turn” case where no cheaper assertion exists. Add a row to the suite for each.
Wire to CI and watch the trend. The deterministic rows gate the next merge. The judged rows trend on the dashboard. When a new failure shows up in production, return to step 1 with that trace.

The trap that kills internal evals is going straight from “we want quality” to “let’s write a faithfulness judge” without the open-coded pass. You end up with a metric that measures a generic property nobody on the team can characterize, scores in the 0.7–0.85 band on every change, and has no relationship to the bugs your users actually report. Error analysis grounds the eval in the system’s actual failure distribution.

Combine golden sets with live evaluation

The eval inputs come from two places, and both are necessary. The golden set is a frozen, curated list of inputs; typically 50–500; checked into version control, never modified except to add new rows or deprecate old ones. It’s the regression unit. Every change to the system is compared against the suite’s previous score on exactly the same inputs. Without this, you cannot tell whether a metric movement is system change or input change. The golden set is small enough to be hand-curated and stable enough to be a contract.

Live evals sample real production traffic, redact PII, and score the sample asynchronously. They catch distribution drift; the kinds of queries users started sending after Wednesday’s marketing push; that the golden set, frozen six months ago, doesn’t represent. The dashboard shows both: golden score (changes only when the system changes) and live score (changes when the system or the input distribution changes). The gap between them is the distribution drift signal.

A practical rule for golden-set construction: every row should map to a specific category from your error analysis. A row that exists “for coverage” of some imagined edge case is a row you can’t interpret when it regresses. The NurtureBoss case study in Hamel’s evals piece is the canonical worked example; they grew their golden set by mining the cases their date-handling logic had previously failed on, not by brainstorming dates a user might type.

Measure cost and latency with quality

A common failure mode is to track quality alone, leaving cost and latency in a separate dashboard nobody opens. The day you ship a model upgrade that improves faithfulness by three points and triples per-call cost, the quality-only dashboard cheers and the finance team panics. Treat each eval row as emitting three numbers; quality score, p50 token cost, p95 latency; and require the suite’s release-gating threshold to include all three. The right framing is the Pareto frontier: a change that moves quality up and cost down is unambiguous; a change that moves only one and worsens the other is a judgment call that has to happen on the PR, not after deploy. This is the same idea the prompt-caching article develops for the inference layer and the RAG-evaluation article develops for retrieval pipelines.

Code: a minimal Python eval harness

The simplest working harness is a couple hundred lines of Python that ingests a golden set, runs the system under test, evaluates each row with a mix of deterministic and judged checks, and writes a result row per commit SHA. Install: pip install anthropic pydantic (and your test runner; pytest works fine).

python

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
# pip install anthropic pydantic
import json
import time
from dataclasses import dataclass
from typing import Callable
from anthropic import Anthropic
from pydantic import BaseModel

client = Anthropic()

@dataclass
class EvalRow:
    id: str
    category: str           # from error analysis
    input: dict             # whatever shape the system takes
    must_contain: list[str] = None       # deterministic substring check
    must_not_contain: list[str] = None
    schema: type[BaseModel] | None = None  # deterministic schema check
    judge_rubric: str | None = None        # LLM-judged check
    max_latency_ms: int | None = None
    max_cost_usd: float | None = None

@dataclass
class EvalResult:
    row_id: str
    category: str
    passed_deterministic: bool
    judge_score: float | None
    latency_ms: int
    cost_usd: float
    error: str | None = None

def llm_judge(output: str, rubric: str) -> float:
    """1-5 score from a small judge model against a rubric."""
    prompt = (
        f"Rubric:\n{rubric}\n\n"
        f"Output:\n{output}\n\n"
        f"Score this 1-5 against the rubric. Reply with only the integer."
    )
    msg = client.messages.create(
        model="claude-haiku-4-5",
        max_tokens=4,
        messages=[{"role": "user", "content": prompt}],
    )
    try:
        return int(msg.content[0].text.strip()) / 5.0
    except (ValueError, IndexError):
        return float("nan")

def run_row(row: EvalRow, system_under_test: Callable) -> EvalResult:
    t0 = time.monotonic()
    try:
        out = system_under_test(row.input)
    except Exception as e:
        return EvalResult(row.id, row.category, False, None, 0, 0.0, str(e))
    latency_ms = int((time.monotonic() - t0) * 1000)

    # Deterministic checks
    passed = True
    if row.must_contain:
        passed &= all(s in out["text"] for s in row.must_contain)
    if row.must_not_contain:
        passed &= all(s not in out["text"] for s in row.must_not_contain)
    if row.schema:
        try:
            row.schema.model_validate_json(out["text"])
        except Exception:
            passed = False
    if row.max_latency_ms:
        passed &= latency_ms <= row.max_latency_ms

    # Judged check (only if deterministic passed; cheap-first ladder)
    judge_score = None
    if passed and row.judge_rubric:
        judge_score = llm_judge(out["text"], row.judge_rubric)

    return EvalResult(
        row_id=row.id,
        category=row.category,
        passed_deterministic=passed,
        judge_score=judge_score,
        latency_ms=latency_ms,
        cost_usd=out.get("cost_usd", 0.0),
    )

def run_suite(rows: list[EvalRow], system_under_test: Callable) -> dict:
    results = [run_row(r, system_under_test) for r in rows]
    by_cat = {}
    for r in results:
        by_cat.setdefault(r.category, []).append(r)
    return {
        "pass_rate": sum(r.passed_deterministic for r in results) / len(results),
        "judge_mean": (
            sum(r.judge_score for r in results if r.judge_score is not None)
            / max(1, sum(1 for r in results if r.judge_score is not None))
        ),
        "p95_latency_ms": sorted(r.latency_ms for r in results)[int(len(results) * 0.95)],
        "total_cost_usd": sum(r.cost_usd for r in results),
        "by_category": {
            cat: {
                "pass_rate": sum(r.passed_deterministic for r in rs) / len(rs),
                "n": len(rs),
            }
            for cat, rs in by_cat.items()
        },
        "results": results,
    }

The harness is deliberately minimal so the shape stays visible. Three things to flag. The cheap-first ladder; judged checks only run when deterministic checks pass; a row that fails schema validation doesn’t pay for a judge call. per-category breakdown; the aggregate is for the dashboard, the per-category numbers are for action. The cost and latency rollups are first-class outputs, not afterthoughts. A CI script wraps this with a pass/fail decision: result["pass_rate"] >= 0.95 and result["judge_mean"] >= 0.80 and result["p95_latency_ms"] <= 3000.

A worked row that exercises every assertion type: