$ cat ai-engineering/llm-as-judge.md

LLM-as-Judge: Pointwise and Pairwise

How to design and calibrate pointwise and pairwise LLM judges against human labels.

Jatin Bansal@blog:~/ai-engineering$ open llm-as-judge

A team ships a rubric-based judge that grades helpfulness 1–5. The first week, every PR’s mean score lands between 3.6 and 3.9. The suite looks healthy, the gate is green, the model upgrade ships. Two weeks later support escalates a wave of complaints about the assistant being curt and unhelpful, and a closer look at the traces shows the upgraded model emits half the tokens for the same questions. The judge had been counting tokens as effort and effort as quality; the verbosity bias was scoring length, not helpfulness, and the new model lost that bias dividend. Nobody on the team had calibrated the judge against a human reviewer in three months. The judge had drifted into vanity, and the eval gate was meaningless before anyone noticed.

The evaluation contract

LLM-as-judge uses one language model to score, rank, or critique another model’s output against a versioned rubric. Reliable use requires a fixed evaluation set, calibration against human labels, and explicit tests for position, verbosity, self-preference, and formatting bias.

The technique was canonized by Zheng et al.’s “Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena”, which established that a strong judge model reaches ~85% agreement with human experts on MT-Bench using pairwise comparison; at parity with the ~81% human–human agreement on the same task. The paper also catalogued the bias modes that the rest of the field has been mitigating since. The complementary lineage is G-Eval, which formalized chain-of-thought rubric grading with token-probability-weighted continuous scores and is the template most modern pointwise harnesses descend from.

Pointwise scoring

A pointwise judge scores one input/output pair against a rubric. Its prompt contains the rubric, input, candidate output, and a structured verdict format. Rubric design and calibration are harder than the call itself; broad properties such as “helpfulness” should be split into narrower criteria.

The discipline the field has converged on, traced back to Hamel Husain’s work and the G-Eval paper, looks like this:

Criterion-separated. One judge prompt per property; faithfulness, relevance, instruction-following, tone; not one prompt that scores all four. Each prompt’s rubric is two or three sentences naming the property and the failure modes.
Binary or low-cardinality: Pass/fail or 1–3 beats 1–5, which beats 1–10. A binary judge is easier to calibrate against a human reviewer because the confusion matrix has only two cells. Drawing on work across more than 30 companies, Hamel reports that domain-expert pass/fail labels correlate better with actual quality than granular scores and are harder to game with verbosity. Pass/fail loses ordinal information; use a 1–3 scale with anchored examples when near-threshold trends matter.
Anchored examples. The rubric includes one or two example outputs at each score level, drawn from your own traces. This is the equivalent of rater training in human evaluation: the judge needs to see what a “3” looks like in your domain, not in the abstract.
Chain-of-thought first, verdict last. The judge writes its reasoning before its verdict. G-Eval’s central trick is to ask the judge to enumerate the evaluation steps in CoT form before emitting the score; the score-after-reasoning ordering improves human correlation significantly compared to score-then-justify. Modern pointwise harnesses adopt this universally.
Token-probability scoring (optional). When the judge emits a single token score (e.g. “3”), the logits over the score tokens can be re-weighted to produce a continuous score: score = Σ p(k) · k for k in the score range. G-Eval shows this reduces score quantization noise and improves correlation with human ratings. This requires logprobs access; supported in the OpenAI SDK directly and now exposed as logprobs=True in the Anthropic Messages API, and is most useful for trend-tracking, less so for hard gates.

The pointwise judge’s failure mode is scale drift: the same property scored across two judge upgrades is not on the same scale, and historical comparisons silently lie unless you pin the judge or recalibrate. The cheap mitigation is to pin the judge model in the eval config (claude-haiku-4-5 is fine for most rubrics) and treat a judge upgrade as a metric reset.

Pairwise comparison

The pairwise judge takes two candidate outputs A and B for the same input and emits “A wins,” “B wins,” or “tie.” The protocol comes directly from Chatbot Arena, where pairwise crowd preferences feed Elo ratings, and from MT-Bench, where pairwise judge calls fed the agreement studies above. Pairwise is the right shape when you want to compare two systems against each other; model A vs model B, prompt v1 vs prompt v2, and when the property is too ambiguous for a stable absolute rubric.

Pairwise’s reliability advantage is real. The judge does not have to map an output onto an absolute scale; it only has to decide which of two outputs is better given the same context. That cancels out a lot of rubric-interpretation noise, which is why pairwise GPT-4 hits 85% human agreement on MT-Bench. But pairwise has two costs: quadratic call count; comparing N systems requires N·(N-1)/2 pairs per query, and position bias, which dominates the noise budget if untreated.

Position bias is the judge’s tendency to prefer the first option (or, depending on the model, the last) regardless of content. Zheng et al. reported Claude-v1 with 70% one-sided preference and GPT-3.5 with 50% positional bias on MT-Bench; recent frontier-model studies show the effect persists, with position bias remaining a 40% inconsistency rate for GPT-4-class judges on rubric-based pairwise tasks. The mitigation is non-optional and adds a 2x cost: run each comparison twice with positions swapped, and only count a “win” when both orderings agree. Disagreements become ties. This is the canonical “both-orderings” protocol and is what every production pairwise pipeline implements.

The second pairwise gotcha is calibration of partial wins. Two orderings that agree on “A wins” are a clean win for A. Two orderings that flip; A wins in (A, B) and B wins in (B, A); are not really a tie; they’re a judge whose verdict was determined by position, not content. The instrumented protocol counts these separately as “position-flipped” cases and gates the eval’s reliability on the fraction below ~10%. If you see 30% position-flipped cases, the judge is too noisy for pairwise on this rubric and you either fall back to pointwise or switch judge models.

Judge biases

Four biases appear in every untreated judge pipeline, and the discipline is to treat each one with an explicit mitigation rather than trust the judge’s calibration:

Position bias. Treated by both-orderings, above.
Verbosity bias. The judge prefers longer outputs even when the shorter one is more correct: length reads as effort, effort reads as quality. Eugene Yan’s survey reports Claude-v1 and GPT-3.5 preferring longer responses over 90% of the time on the verbosity probe. The mitigation is a rubric that explicitly rewards conciseness when the shorter response covers the same ground, plus length-normalized scoring on rubrics where length and quality are orthogonal. Don’t trust the judge to detect verbose padding on its own.
Self-preference bias. A judge from a model family scores outputs from its own family higher than outputs from other families. Zheng et al. measured GPT-4 with 10% self-enhancement and Claude-v1 with 25% self-preference. The mitigation is cross-family judging: use a judge from a different family than the systems being compared, or use a panel of judges (more below).
Length / authority / format biases. Catch-all category: the judge prefers outputs that look authoritative (citations, hedged claims), follow a familiar format (bullet lists over prose, even when prose is better), or contain technical jargon. These biases are domain-specific; the mitigation is to detect them via the calibration loop: if the judge and a domain expert disagree systematically on a category of output, that’s a bias signal, and the rubric needs an explicit constraint to counteract it.

These four are not a complete list; the Survey on LLM-as-a-Judge catalogues a dozen more. But the four above are the critical ones in practice; treat them and the rest is small noise on top.

Panels of judges

A panel of smaller judges from disjoint model families outperforms a single large judge on most tasks, costs less, and has lower intra-model bias. Verga et al. (“Replacing Judges with Juries”) measured a three-judge panel of smaller models (Command-R, Haiku, GPT-3.5) beating GPT-4 as a single judge on six datasets while costing 7x less. The mechanism is two-fold: cross-family panels neutralize each model’s self-preference, and averaging over independent estimators reduces the per-call variance the same way ensemble methods do everywhere else.

The trade-off is operational: more judge clients to manage, more credentials, more API surface to fail on. Production teams ship panels for the expensive nightly judged sweep where the variance reduction matters; the cheap per-PR layer often runs with a single pinned judge for simplicity. The cleanest panel is three judges from three families; pick one Anthropic, one OpenAI, one open-weights (e.g. a Llama or Qwen variant). Aggregate by majority vote for binary verdicts and by mean for continuous scores.

Calibration against humans

The judge has no meaning until you’ve measured its agreement with a domain expert on a held-out calibration set. The protocol is:

Sample 100–200 traces stratified by category from production or from the existing eval set.
Have one domain expert; not three, not a committee; score every sample on the rubric, using the same prompt the judge will see. The single-expert constraint is from Hamel’s work: averaging over multiple humans introduces variance that swamps the signal at small sample sizes.
Score the same samples with the candidate judge.
Compute Cohen’s kappa between expert and judge for categorical verdicts (or Spearman/Kendall correlation for continuous scores). Plain percentage agreement overstates: an 80% percentage agreement can correspond to a 0.62 kappa, which is fair but not strong (Eugene Yan reports exactly this gap on TriviaQA with Llama-3-8b as judge).
Iterate on the rubric until kappa exceeds ~0.7 on the calibration set, or; if the underlying property is genuinely ambiguous; accept that the judge has a floor and switch to a different property.

Re-run the calibration on a fresh sample monthly, or after any judge-model upgrade, prompt change, or rubric edit. The judge’s agreement number is the metric that says the metric is trustworthy.

Pointwise judge in Python

The Python harness below pins the judge model, runs the rubric in chain-of-thought mode, parses a structured verdict via Pydantic, and exposes a pairwise shim that wraps two pointwise calls. Install: pip install anthropic pydantic.

python

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
# pip install anthropic pydantic
from anthropic import Anthropic
from pydantic import BaseModel, Field
import json

JUDGE_MODEL = "claude-haiku-4-5"  # pinned; treat upgrade as metric reset
client = Anthropic()

class Verdict(BaseModel):
    reasoning: str = Field(..., description="Step-by-step rubric reasoning before the score.")
    score: int = Field(..., ge=1, le=3, description="1=fail, 2=partial, 3=pass")

POINTWISE_PROMPT = """\
You are an evaluator. Score the response against the rubric below.

Rubric (faithfulness): the response makes only claims that are explicitly supported by the provided context. Score:
- 3 (pass): every claim supported, no extrapolation
- 2 (partial): one claim under-supported but the rest are clean
- 1 (fail): hallucinated or contradicted claims

Anchored examples:
- A 3 looks like: <one trace from your domain, redacted>
- A 1 looks like: <one trace, redacted>

Reason step by step first, then emit a JSON object: {"reasoning": "...", "score": N}.

Context:
{context}

Response:
{response}
"""

def pointwise_judge(context: str, response: str) -> Verdict:
    msg = client.messages.create(
        model=JUDGE_MODEL,
        max_tokens=512,
        messages=[{
            "role": "user",
            "content": POINTWISE_PROMPT.format(context=context, response=response),
        }],
    )
    text = msg.content[0].text
    # Robust JSON extraction: find the last {...} block in the output.
    start = text.rfind("{")
    end = text.rfind("}")
    return Verdict.model_validate_json(text[start:end + 1])

# Pairwise via two pointwise calls plus the both-orderings adapter.
def pairwise_judge(context: str, response_a: str, response_b: str) -> str:
    """Returns 'A', 'B', 'tie', or 'position-flipped'."""
    a_first = _compare(context, response_a, response_b)  # asks: which is better?
    b_first = _compare(context, response_b, response_a)
    if a_first == "first" and b_first == "second":
        return "A"
    if a_first == "second" and b_first == "first":
        return "B"
    if a_first == b_first:
        return "position-flipped"  # judge prefers position, not content
    return "tie"

PAIRWISE_PROMPT = """\
You are an evaluator. Two responses to the same query are below. Decide which
better follows the rubric (faithfulness as above), or whether they're equivalent.

Reason step by step first, then emit a JSON object: {"reasoning": "...", "verdict": "first"|"second"|"tie"}.

Context:
{context}

Response 1:
{first}

Response 2:
{second}
"""

def _compare(context: str, first: str, second: str) -> str:
    msg = client.messages.create(
        model=JUDGE_MODEL,
        max_tokens=512,
        messages=[{
            "role": "user",
            "content": PAIRWISE_PROMPT.format(context=context, first=first, second=second),
        }],
    )
    text = msg.content[0].text
    start = text.rfind("{")
    end = text.rfind("}")
    return json.loads(text[start:end + 1])["verdict"]

The pointwise rubric evaluates one property per prompt, so four properties require four calls per row. The pairwise implementation records position-flipped verdicts separately from ties because they indicate judge noise. JSON parsing is permissive because judges can prepend prose; strict structured output is another valid implementation.