Definition
Arena-style ranking ranks generative models from many pairwise human votes. A user sends a prompt, two anonymous models answer side by side, the user votes for one (or a tie), and only then are the names revealed. The votes are fitted with the Bradley–Terry model (Bradley and Terry, 1952) and reported on the Elo scale. The best-known instance is Chatbot Arena (Chiang et al., "Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference", ICML 2024), launched by LMSYS in 2023.
Human judgment is the reference that every automatic metric is validated against, including LLM-as-a-judge. It is also where product decisions are made: Ouyang et al. (2022) reported that labelers preferred outputs of a 1.3B-parameter InstructGPT model to those of the 175B GPT-3, a result no benchmark score would have shown. The cost is time and noise, so the design of the study decides whether its numbers mean anything.
Designing a human evaluation
| Format | Rater task | Good for | Watch out for |
|---|---|---|---|
| Absolute (Likert) | rate one output 1–5 on a criterion | tracking one system over time | raters use the scale differently; scores drift |
| Pairwise preference | pick the better of A and B, or tie | comparing systems; small differences | needs a model (Bradley–Terry) to rank many systems |
| Ranking | order k outputs | collecting preference data for RLHF | harder per item; expands to k(k−1)/2 pairs |
| Error annotation | mark error spans with type and severity | diagnosis (MQM in translation) | slow; needs trained annotators |
- 1Write the question you want answered: "is v2 more helpful than v1 on billing questions?" fixes the prompts, the criterion and the comparison.
- 2Sample prompts from real traffic or a versioned set, stratified by task; hold them fixed across systems.
- 3Blind and randomise: hide system names, shuffle left/right placement and item order per rater.
- 4Write guidelines with worked examples of each label, including ties and "both bad"; pilot on 20–50 items and revise.
- 5Collect overlapping labels (2–3 raters per item on at least a subset) and report inter-annotator agreement, e.g. Cohen's κ or Krippendorff's α.
- 6Report win rate with a confidence interval and the number of votes, not only a point estimate.
Who rates matters. Clark et al. ("All That's 'Human' Is Not Gold", ACL 2021) found that untrained evaluators distinguished GPT-3 text from human text at roughly chance level. Freitag et al. (TACL 2021) found that professional translators using MQM error annotation ranked MT systems differently from crowd raters. Domain tasks (medicine, law, code) need domain raters, and agreement between them sets the ceiling for any automatic metric.
From votes to ratings
- latent strength of model i; only differences are identified, so one model or the mean is anchored
- winner and loser of one battle; a tie is commonly split as half a win each
- Elo scale: 400/ln 10 ≈ 173.7 points per unit of log-odds; a 100-point gap means 64.0% wins
This is logistic regression with one and one per row, the same model as the reward model in RLHF. The maximum-likelihood fit uses all battles at once, so the result does not depend on their order. Chatbot Arena first published online Elo, , and later switched to the Bradley–Terry fit. Online Elo assumes players' strengths drift, as in chess. A frozen model's strength does not drift, so order-dependence is pure noise.
- observed win rate of one model over another in n votes
- votes on one pair for the 95% interval half-width to equal the true gap
A 20-point gap is a 52.9% win rate and needs about 1,162 votes on that pair; a 50-point gap (57.1%) needs about 188. With 1,000 votes the half-width is , about Elo points. Leaderboards therefore publish bootstrap intervals, and a rank is only meaningful as "1 + the number of models whose interval lies entirely above". Ties at the top are normal.
Arena practice
- Votes
- A better, B better, tie, both bad. Names are hidden until after the vote.
- Scale
- Chiang et al. (2024) reported over 240,000 votes at the time of the paper; the live counts are far larger and change daily.
- Fit
- Bradley–Terry by maximum likelihood, with bootstrap confidence intervals; it replaced the initial online Elo.
- Style control
- Answer length and markdown use enter the regression as covariates, so ratings are not driven by formatting (added in 2024).
- Bias
- Voters choose their own prompts, so the ranking reflects the user population's tasks and tastes, not a fixed benchmark.
Set with 200 battles: online Elo barely moves with the order (5.3 points) but stays compressed near its starting value of 1000, with errors up to 81.0. Small trades order noise for bias; Bradley–Terry has neither problem.
Worked example: three models
- 1Votes: A beats B 60–40, B beats C 60–40, A beats C 70–30 (300 battles).
- 2One pair alone: , i.e. Elo points.
- 3Joint Bradley–Terry fit (mean anchored at 1000): A 1072.4, B 1000.0, C 927.6. The gaps are 72.4, 72.4 and 144.8.
- 4Check: the model predicts and , close to the observed 0.60 and 0.70. Bradley–Terry is transitive, so it pools evidence: the A–C votes sharpen A–B.
- 5Bootstrap (200 resamples): A [1041, 1105], B [964, 1031], C [890, 957]. A and B overlap, so with 100 votes per pair A is not clearly better than B.
1import numpy as np23def fit_bt(battles, models, iters=50):4 """Bradley–Terry by Newton's method on logistic loss; battles = [(winner, loser), ...]."""5 idx = {m: i for i, m in enumerate(models)}6 X = np.zeros((len(battles), len(models)))7 for r, (w, l) in enumerate(battles):8 X[r, idx[w]], X[r, idx[l]] = 1.0, -1.0 # every row: winner +1, loser -1, y = 19 beta = np.zeros(len(models))10 for _ in range(iters):11 p = 1 / (1 + np.exp(-X @ beta))12 grad = X.T @ (1 - p) - 1e-6 * beta13 hess = (X * (p * (1 - p))[:, None]).T @ X + 1e-6 * np.eye(len(models))14 beta += np.linalg.solve(hess, grad)15 beta -= beta.mean()16 return 1000 + 400 / np.log(10) * beta # Elo scale, mean 10001718models = ["A", "B", "C"]19battles = ([("A", "B")] * 60 + [("B", "A")] * 40 + [("B", "C")] * 60 + [("C", "B")] * 4020 + [("A", "C")] * 70 + [("C", "A")] * 30)21print(np.round(fit_bt(battles, models), 1)) # [1072.4 1000. 927.6]2223rng = np.random.default_rng(0) # bootstrap 95% intervals24boot = np.array([fit_bt([battles[i] for i in rng.integers(0, len(battles), len(battles))], models)25 for _ in range(200)])26print(np.round(np.percentile(boot, [2.5, 97.5], axis=0), 0))Check yourself
Model X is rated 1205 [1188, 1224] and model Y 1191 [1177, 1208]. What should the leaderboard say?
Check yourself
Why did arena leaderboards move from online Elo to a Bradley–Terry fit?
Remember
- ✓Pairwise, blind, randomised comparisons are the most reliable human signal for ranking generative systems.
- ✓Report inter-annotator agreement (κ or α); it bounds how well any metric can match humans.
- ✓Bradley–Terry: , fitted by MLE on all votes; Elo scale .
- ✓Uncertainty is large: a 20-point gap needs about 1,200 votes on that pair; rank by bootstrap interval.
- ✓Control for length and formatting, and remember arena ratings reflect their own users' prompts.