Definition
LLM-as-a-judge is the use of a language model as the evaluator of other models' outputs. Given a prompt, one or two candidate answers and a rubric, the judge returns a score or a preference, usually with a written rationale. Zheng et al. named and studied the practice in "Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena" (NeurIPS 2023 Datasets and Benchmarks), where GPT-4's pairwise verdicts agreed with expert humans over 80% of the time, about as often as two experts agreed with each other.
Open-ended outputs (chat answers, summaries, code reviews, agent traces) have no answer key, so exact-match metrics do not apply. Overlap metrics such as BLEU, ROUGE and METEOR and BERTScore need a reference and reward wording, not usefulness. Human raters are the gold standard but cost minutes per item (see Human evaluation and arena-style ranking). A judge model costs one API call per item, so it can run on every pull request, every prompt change and every fine-tuning checkpoint.
In a real system the judge sits in the offline eval harness and sometimes online. Offline it compares a candidate against a baseline on a fixed prompt set. Online it scores a sample of production traffic for groundedness, policy violations or task success, feeding dashboards and alerts. LLM evaluation and benchmarking covers the benchmark side; this article is about building and trusting the judge itself.
Three formats
Pairwise comparison: the judge sees two answers and picks one or a tie. It is the most sensitive format and matches how humans compare. Single-answer grading: the judge scores one answer on a scale, say 1–10. It is cheaper to scale (one call per answer, not per pair) but scores drift between judge versions. Reference-guided grading: the judge also gets a reference solution, which Zheng et al. found important for maths and reasoning, where the judge may fail to solve the problem itself.
Known biases
Position bias: verdicts depend on which answer comes first. Wang et al. ("Large Language Models are not Fair Evaluators", 2023) showed that swapping the order could flip rankings between models, and proposed scoring both orders and averaging. Verbosity bias: longer answers win even when the extra text adds nothing or adds errors. Self-preference: a judge tends to favour outputs from its own model family. Limited competence: a judge cannot reliably grade a proof or a bug fix that it cannot produce itself.
- 1 if the judge prefers the answer in slot 1 when X is shown first, 0 otherwise
- order-balanced score: 1 or 0 when both orders agree, 0.5 when they split
- length difference of A over B (standardised)
- fitted length effect and the remaining preference; is the win rate predicted at equal lengths
The length model is the idea behind length-controlled AlpacaEval (Dubois et al., 2024). Its regression has terms for the model, the length difference and the instruction, and the reported win rate is the prediction with the length term set to zero. Arena-style ratings use the same trick under the name style control (see Human evaluation and arena-style ranking).
Measuring the judge
- observed agreement between judge and human labels
- agreement expected by chance from the two label distributions; is chance, 1 is perfect
- the judge's probability for score token s (G-Eval, Liu et al., EMNLP 2023)
Raw agreement flatters a judge when one label dominates; (Cohen, 1960) removes the chance part. Compare it with human–human on the same items: a judge cannot be expected to beat the humans' own consistency. Probability-weighted scores give a continuous value and fewer ties than reading the argmax token, which helps rank close systems.
Reference points
- MT-Bench
- 80 multi-turn questions in 8 categories (writing, roleplay, extraction, reasoning, math, coding, STEM, humanities), graded by GPT-4 on a 1–10 scale (Zheng et al., 2023).
- AlpacaEval
- 805 instructions; pairwise win rate against a reference model, judged by an LLM. The length-controlled version (2024) was introduced to remove the length bias.
- G-Eval
- Liu et al., EMNLP 2023: the judge first writes evaluation steps (chain of thought), then fills a form; scores are weighted by token probabilities.
- Open judges
- Fine-tuned evaluator models such as Prometheus (Kim et al., ICLR 2024) take a user-written rubric and can run locally.
- Juries
- Verga et al. (2024) found that a panel of smaller judges from different families can match a single large judge at lower cost and with less self-preference.
Two lessons from the figure. Length control fixes length bias but not position bias: with A always first, it still reports 0.652. Swapping fixes position bias at the cost of ties; the decided verdicts get more accurate, but the tie rate itself is a warning that the judge is unreliable on those items.
Worked example: is the judge good enough?
| Judge ↓ / human → | Human: A | Human: B | Judge total |
|---|---|---|---|
| Judge: A | 40 | 15 | 55 |
| Judge: B | 10 | 35 | 45 |
| Human total | 50 | 50 | 100 |
- 1Observed agreement: .
- 2Chance agreement: .
- 3: moderate. If two humans reach on these items, the judge is close to, but below, human consistency.
- 4The 15 + 10 disagreements are the useful output: read them, look for a pattern (longer answer wins, first answer wins, maths), then fix the rubric or add a reference answer and re-measure.
1import json23JUDGE_PROMPT = """You are an impartial judge. Compare the two answers to the question.4Judge accuracy first, then helpfulness. Ignore answer order and length.5Question: {q}6[Answer 1]7{a1}8[Answer 2]9{a2}10Reply with JSON: {{"reasoning": "...", "winner": "1" | "2" | "tie"}}"""1112def pairwise(ask, q, a, b):13 """Score A vs B in both orders; returns 1 (A), 0 (B) or 0.5 (tie / inconsistent)."""14 v1 = json.loads(ask(JUDGE_PROMPT.format(q=q, a1=a, a2=b)))["winner"]15 v2 = json.loads(ask(JUDGE_PROMPT.format(q=q, a1=b, a2=a)))["winner"]16 s1 = {"1": 1.0, "2": 0.0, "tie": 0.5}[v1] # A was in slot 117 s2 = {"1": 0.0, "2": 1.0, "tie": 0.5}[v2] # A was in slot 218 return s1 if s1 == s2 else 0.51920# stand-in for an LLM API call: a judge that always prefers slot 121always_first = lambda prompt: '{"reasoning": "...", "winner": "1"}'22print(pairwise(always_first, "Explain a p-value.", "P(data this extreme | H0).", "P(H0 is true)."))23# 0.5: the verdict flips with the order, so it is recorded as a tieCheck yourself
A judge picks A when A is shown first and B when B is shown first. What should an order-balanced protocol record?
Check yourself
Model B's answers are on average twice as long as A's, and an LLM judge prefers B in 70% of pairs. What is the first check?
Remember
- ✓A judge LLM scores or compares outputs against a rubric; pairwise is the most sensitive format, single grading the cheapest.
- ✓Biases: position, verbosity, self-preference and limited competence on hard maths or code.
- ✓Run both orders, record a tie when they split; control for length with .
- ✓Validate on human labels: agreement, Cohen's , position consistency, length slope; compare with human–human .
- ✓Use specific rubrics, reference answers for reasoning tasks, a judge from another family, and a pinned judge version.