LLM-as-a-judge

A strong LLM grades or compares outputs against a rubric. Run pairwise verdicts in both orders, control for length, and validate against human labels with agreement and Cohen's κ before trusting the scores.

8 min read800 wordscore0 viewsupdated 2026-10-21

Definition

LLM-as-a-judge is the use of a language model as the evaluator of other models' outputs. Given a prompt, one or two candidate answers and a rubric, the judge returns a score or a preference, usually with a written rationale. Zheng et al. named and studied the practice in "Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena" (NeurIPS 2023 Datasets and Benchmarks), where GPT-4's pairwise verdicts agreed with expert humans over 80% of the time, about as often as two experts agreed with each other.

Open-ended outputs (chat answers, summaries, code reviews, agent traces) have no answer key, so exact-match metrics do not apply. Overlap metrics such as BLEU, ROUGE and METEOR and BERTScore need a reference and reward wording, not usefulness. Human raters are the gold standard but cost minutes per item (see Human evaluation and arena-style ranking). A judge model costs one API call per item, so it can run on every pull request, every prompt change and every fine-tuning checkpoint.

In a real system the judge sits in the offline eval harness and sometimes online. Offline it compares a candidate against a baseline on a fixed prompt set. Online it scores a sample of production traffic for groundedness, policy violations or task success, feeding dashboards and alerts. LLM evaluation and benchmarking covers the benchmark side; this article is about building and trusting the judge itself.

Three formats

Pairwise comparison: the judge sees two answers and picks one or a tie. It is the most sensitive format and matches how humans compare. Single-answer grading: the judge scores one answer on a scale, say 1–10. It is cheaper to scale (one call per answer, not per pair) but scores drift between judge versions. Reference-guided grading: the judge also gets a reference solution, which Zheng et al. found important for maths and reasoning, where the judge may fail to solve the problem itself.

pairwise judging: ask twice, swap the order, keep only consistent verdictsQ: Explain what a p-value is.A (short, correct)P(data at least this extreme| H₀ true).B (long, wrong)“The probability that the nullhypothesis is true. It is a keyconcept in statistics…”call 1: [1] = A, [2] = Brubric: accuracy, then help{"winner": "1"}→ A winscall 2: [1] = B, [2] = Asame prompt, same rubric{"winner": "1"}→ B winssplitverdict: tiescore 0.5slot 1 won twicesingle-answer grading: read the score distribution, not just the argmaxRate answer A for accuracy,1 (wrong) … 5 (fully correct).Explain, then output the scoreas one token.judge reads Q, A and rubric0.0010.0520.1030.4040.455p(score token)argmax: 5Σ p(s)·s= 4.25fewer tiesbefore trusting it: meta-evaluate the judge on human labelsagreementvs majority labelCohen's κchance-correctedconsistencyboth orders agreelength slopewin rate vs Δ length
fig.Top: a pairwise judgment run twice with the answer order swapped. Answer A is short and correct, B is long and wrong. Both calls pick slot 1, so the verdicts disagree and the pair is recorded as a tie, not as a win for either. Middle: single-answer grading that reads the score-token probabilities (0.05, 0.10, 0.40, 0.45 for 2–5) and reports Σ p(s)·s = 4.25 instead of the argmax 5. Bottom: the four numbers to measure against human labels before trusting the judge.

Known biases

Position bias: verdicts depend on which answer comes first. Wang et al. ("Large Language Models are not Fair Evaluators", 2023) showed that swapping the order could flip rankings between models, and proposed scoring both orders and averaging. Verbosity bias: longer answers win even when the extra text adds nothing or adds errors. Self-preference: a judge tends to favour outputs from its own model family. Limited competence: a judge cannot reliably grade a proof or a bug fix that it cannot produce itself.

sAB=12[v(A,B)+1−v(B,A)],logit⁡P(A≻B)=θ+φ Δℓ,LC win rate=σ(θ)s_{AB}=\tfrac12\big[v(A,B)+1-v(B,A)\big],\qquad \operatorname{logit}P(A\succ B)=\theta+\varphi\,\Delta\ell,\qquad \text{LC win rate}=\sigma(\theta)
v(X,Y)v(X,Y)
1 if the judge prefers the answer in slot 1 when X is shown first, 0 otherwise
sABs_{AB}
order-balanced score: 1 or 0 when both orders agree, 0.5 when they split
Δℓ\Delta\ell
length difference of A over B (standardised)
φ, θ\varphi,\ \theta
fitted length effect and the remaining preference; σ(θ)\sigma(\theta) is the win rate predicted at equal lengths

The length model is the idea behind length-controlled AlpacaEval (Dubois et al., 2024). Its regression has terms for the model, the length difference and the instruction, and the reported win rate is the prediction with the length term set to zero. Arena-style ratings use the same trick under the name style control (see Human evaluation and arena-style ranking).

Measuring the judge

κ=po−pe1−pe,pe=∑kpkjudge pkhuman,scoreG-Eval=∑sp(s) s\kappa=\frac{p_o-p_e}{1-p_e},\qquad p_e=\sum_k p^{\text{judge}}_k\,p^{\text{human}}_k,\qquad \text{score}_{\text{G-Eval}}=\sum_{s}p(s)\,s
pop_o
observed agreement between judge and human labels
pep_e
agreement expected by chance from the two label distributions; κ=0\kappa=0 is chance, 1 is perfect
p(s)p(s)
the judge's probability for score token s (G-Eval, Liu et al., EMNLP 2023)

Raw agreement flatters a judge when one label dominates; κ\kappa (Cohen, 1960) removes the chance part. Compare it with human–human κ\kappa on the same items: a judge cannot be expected to beat the humans' own consistency. Probability-weighted scores give a continuous value and fewer ties than reading the argmax token, which helps rank close systems.

Reference points

MT-Bench
80 multi-turn questions in 8 categories (writing, roleplay, extraction, reasoning, math, coding, STEM, humanities), graded by GPT-4 on a 1–10 scale (Zheng et al., 2023).
AlpacaEval
805 instructions; pairwise win rate against a reference model, judged by an LLM. The length-controlled version (2024) was introduced to remove the length bias.
G-Eval
Liu et al., EMNLP 2023: the judge first writes evaluation steps (chain of thought), then fills a form; scores are weighted by token probabilities.
Open judges
Fine-tuned evaluator models such as Prometheus (Kim et al., ICLR 2024) take a user-written rubric and can run locally.
Juries
Verga et al. (2024) found that a panel of smaller judges from different families can match a single large judge at lower cost and with less self-preference.
300 prompts · model A vs model B · judge protocol: A always firsthumansjudge boundarytrue quality difference A − B →A longer than B →verdict: A wins · B wins · tiewin rate of model Ahumans (truth)0.440judge, raw0.747judge, length-controlled0.6520.5agreement with humans198/300 decided = 0.660ties (orders disagreed)n/a (one call per pair)position consistencyn/aCohen's κ vs humans0.358fitted length slope φ0.685judge call: A wins if 1.5·d + 0.6·Δℓ + 0.8 (slot) + noise > 0; humans: d + noise > 0
protocol
fig.300 synthetic prompts where model A is slightly worse (humans prefer A 44.0% of the time) but longer. Each judge call adds position and length terms to the true quality gap. Defaults (position bias 0.8, length bias 0.6, A always first): the judge reports a win rate of 0.747, length control alone 0.652, agreement with humans 0.660 and κ = 0.358. Random order: 0.583 raw and 0.445 length-controlled. Both orders, tie if split: 117 ties (consistency 0.610), 0.555 raw, 0.459 length-controlled, and 158 of 183 decided verdicts (0.863) match the humans. A bias-free judge agrees on 242/300 (κ = 0.611).

Two lessons from the figure. Length control fixes length bias but not position bias: with A always first, it still reports 0.652. Swapping fixes position bias at the cost of ties; the decided verdicts get more accurate, but the tie rate itself is a warning that the judge is unreliable on those items.

Worked example: is the judge good enough?

Judge ↓ / human →Human: AHuman: BJudge total
Judge: A401555
Judge: B103545
Human total5050100
100 human-labelled pairs; judge vs majority human label
  1. 1Observed agreement: po=(40+35)/100=0.75p_o=(40+35)/100=0.75.
  2. 2Chance agreement: pe=0.55⋅0.50+0.45⋅0.50=0.50p_e=0.55\cdot0.50+0.45\cdot0.50=0.50.
  3. 3κ=(0.75−0.50)/(1−0.50)=0.50\kappa=(0.75-0.50)/(1-0.50)=0.50: moderate. If two humans reach κ=0.6\kappa=0.6 on these items, the judge is close to, but below, human consistency.
  4. 4The 15 + 10 disagreements are the useful output: read them, look for a pattern (longer answer wins, first answer wins, maths), then fix the rubric or add a reference answer and re-measure.
python
1import json
2
3JUDGE_PROMPT = """You are an impartial judge. Compare the two answers to the question.
4Judge accuracy first, then helpfulness. Ignore answer order and length.
5Question: {q}
6[Answer 1]
7{a1}
8[Answer 2]
9{a2}
10Reply with JSON: {{"reasoning": "...", "winner": "1" | "2" | "tie"}}"""
11
12def pairwise(ask, q, a, b):
13 """Score A vs B in both orders; returns 1 (A), 0 (B) or 0.5 (tie / inconsistent)."""
14 v1 = json.loads(ask(JUDGE_PROMPT.format(q=q, a1=a, a2=b)))["winner"]
15 v2 = json.loads(ask(JUDGE_PROMPT.format(q=q, a1=b, a2=a)))["winner"]
16 s1 = {"1": 1.0, "2": 0.0, "tie": 0.5}[v1] # A was in slot 1
17 s2 = {"1": 0.0, "2": 1.0, "tie": 0.5}[v2] # A was in slot 2
18 return s1 if s1 == s2 else 0.5
19
20# stand-in for an LLM API call: a judge that always prefers slot 1
21always_first = lambda prompt: '{"reasoning": "...", "winner": "1"}'
22print(pairwise(always_first, "Explain a p-value.", "P(data this extreme | H0).", "P(H0 is true)."))
23# 0.5: the verdict flips with the order, so it is recorded as a tie
Order-balanced pairwise judging. `ask` is any function that sends a prompt to an LLM API and returns its text; here a stand-in judge that always picks slot 1 shows the swap catching position bias (runs as is)

Check yourself

A judge picks A when A is shown first and B when B is shown first. What should an order-balanced protocol record?

Check yourself

Model B's answers are on average twice as long as A's, and an LLM judge prefers B in 70% of pairs. What is the first check?

Remember

  • ✓A judge LLM scores or compares outputs against a rubric; pairwise is the most sensitive format, single grading the cheapest.
  • ✓Biases: position, verbosity, self-preference and limited competence on hard maths or code.
  • ✓Run both orders, record a tie when they split; control for length with logit⁡P=θ+φΔℓ\operatorname{logit}P=\theta+\varphi\Delta\ell.
  • ✓Validate on human labels: agreement, Cohen's κ\kappa, position consistency, length slope; compare with human–human κ\kappa.
  • ✓Use specific rubrics, reference answers for reasoning tasks, a judge from another family, and a pinned judge version.
← back to Generative AI roadmap