Human evaluation and arena-style ranking

People compare two anonymous answers and vote; a Bradley–Terry fit turns votes into ratings with bootstrap intervals. Rank by interval, not point estimate: a 20-point gap needs about 1,200 votes on that pair.

8 min read762 wordscore0 viewsupdated 2026-10-21

Definition

Arena-style ranking ranks generative models from many pairwise human votes. A user sends a prompt, two anonymous models answer side by side, the user votes for one (or a tie), and only then are the names revealed. The votes are fitted with the Bradley–Terry model (Bradley and Terry, 1952) and reported on the Elo scale. The best-known instance is Chatbot Arena (Chiang et al., "Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference", ICML 2024), launched by LMSYS in 2023.

Human judgment is the reference that every automatic metric is validated against, including LLM-as-a-judge. It is also where product decisions are made: Ouyang et al. (2022) reported that labelers preferred outputs of a 1.3B-parameter InstructGPT model to those of the 175B GPT-3, a result no benchmark score would have shown. The cost is time and noise, so the design of the study decides whether its numbers mean anything.

Designing a human evaluation

FormatRater taskGood forWatch out for
Absolute (Likert)rate one output 1–5 on a criteriontracking one system over timeraters use the scale differently; scores drift
Pairwise preferencepick the better of A and B, or tiecomparing systems; small differencesneeds a model (Bradley–Terry) to rank many systems
Rankingorder k outputscollecting preference data for RLHFharder per item; expands to k(k−1)/2 pairs
Error annotationmark error spans with type and severitydiagnosis (MQM in translation)slow; needs trained annotators
Common formats. Pairwise is the default for ranking systems; error annotation for finding out why one is worse.
  1. 1Write the question you want answered: "is v2 more helpful than v1 on billing questions?" fixes the prompts, the criterion and the comparison.
  2. 2Sample prompts from real traffic or a versioned set, stratified by task; hold them fixed across systems.
  3. 3Blind and randomise: hide system names, shuffle left/right placement and item order per rater.
  4. 4Write guidelines with worked examples of each label, including ties and "both bad"; pilot on 20–50 items and revise.
  5. 5Collect overlapping labels (2–3 raters per item on at least a subset) and report inter-annotator agreement, e.g. Cohen's κ or Krippendorff's α.
  6. 6Report win rate with a confidence interval and the number of votes, not only a point estimate.

Who rates matters. Clark et al. ("All That's 'Human' Is Not Gold", ACL 2021) found that untrained evaluators distinguished GPT-3 text from human text at roughly chance level. Freitag et al. (TACL 2021) found that professional translators using MQM error annotation ranked MT systems differently from crowd raters. Domain tasks (medicine, law, code) need domain raters, and agreement between them sets the ceiling for any automatic metric.

one battle: blind side-by-side, vote first, names revealed afteruser promptWrite a haikuabout rain.Model A (hidden): …Model B (hidden): …A is betterB is bettertieboth badthen revealed: A = cedar, B = fjordbattle logidmodel Amodel Bvote#1cedarfjordA#2atlasbirchB#3duneatlastie#4birchcedarAtie → ½ win each sideBradley–Terry MLEmaximise Σ log σ(β_w − β_l)R = 400·log₁₀ p + cbootstrap battles→ 95% intervalrankmodelrating95% CI1atlas12421226–12602birch12051188–12242cedar11911177–12084dune11521137–1169rank = 1 + models clearly abovehuman-evaluation formats (pick per question, report agreement)absolute (Likert 1–5)one answer at a timescale drift across raterspairwise preferenceA vs B, or rankingeasier, needs a BT fiterror annotationmark spans + severityslow, but diagnosticinter-annotator agreement (Cohen's κ, Krippendorff's α) bounds how well any metric can match humans
fig.One arena battle: a prompt goes to two hidden models, the user votes A better, B better, tie or both bad, and only then are the names revealed. Battles accumulate in a log, a Bradley–Terry fit turns them into ratings, and bootstrapping the battles gives 95% intervals. Here birch (1205) and cedar (1191) share rank 2 because their intervals overlap, while atlas (1242) is clear of both. Below: the three basic human-evaluation formats.

From votes to ratings

P(i≻j)=σ(βi−βj),β^=arg⁡max⁡β∑(w,l)log⁡σ(βw−βl),Ri=1000+400ln⁡10 βiP(i\succ j)=\sigma(\beta_i-\beta_j),\qquad \hat\beta=\arg\max_\beta\sum_{(w,l)}\log\sigma(\beta_w-\beta_l),\qquad R_i=1000+\frac{400}{\ln 10}\,\beta_i
βi\beta_i
latent strength of model i; only differences are identified, so one model or the mean is anchored
(w,l)(w,l)
winner and loser of one battle; a tie is commonly split as half a win each
RiR_i
Elo scale: 400/ln 10 ≈ 173.7 points per unit of log-odds; a 100-point gap means 64.0% wins

This is logistic regression with one +1+1 and one −1-1 per row, the same model as the reward model in RLHF. The maximum-likelihood fit uses all battles at once, so the result does not depend on their order. Chatbot Arena first published online Elo, Ri←Ri+K(Si−Ei)R_i\leftarrow R_i+K(S_i-E_i), and later switched to the Bradley–Terry fit. Online Elo assumes players' strengths drift, as in chess. A frozen model's strength does not drift, so order-dependence is pure noise.

SE(p^)=p(1−p)n,n95%≈(1.96⋅0.5p−0.5)2\mathrm{SE}(\hat p)=\sqrt{\frac{p(1-p)}{n}},\qquad n_{95\%}\approx\left(\frac{1.96\cdot0.5}{p-0.5}\right)^{2}
p^\hat p
observed win rate of one model over another in n votes
n95%n_{95\%}
votes on one pair for the 95% interval half-width to equal the true gap p−0.5p-0.5

A 20-point gap is a 52.9% win rate and needs about 1,162 votes on that pair; a 50-point gap (57.1%) needs about 188. With 1,000 votes the half-width is ±0.031\pm0.031, about ±22\pm22 Elo points. Leaderboards therefore publish bootstrap intervals, and a rank is only meaningful as "1 + the number of models whose interval lies entirely above". Ties at the top are normal.

Arena practice

Votes
A better, B better, tie, both bad. Names are hidden until after the vote.
Scale
Chiang et al. (2024) reported over 240,000 votes at the time of the paper; the live counts are far larger and change daily.
Fit
Bradley–Terry by maximum likelihood, with bootstrap confidence intervals; it replaced the initial online Elo.
Style control
Answer length and markdown use enter the regression as covariates, so ratings are not driven by formatting (added in 2024).
Bias
Voters choose their own prompts, so the ranking reflects the user population's tasks and tastes, not a fixed benchmark.
1000 battles · ≈67 per pair · Bradley–Terry MLE + 100 bootstraps · online Elo K = 3290010001100120013001400model (true)BT [95% CI] rankatlas 12501254 [1224, 1297] #1birch 12101207 [1174, 1241] #3cedar 11901231 [1203, 1267] #2dune 11501124 [1088, 1160] #4ember 10801111 [1080, 1143] #5fjord 1000952 [908, 986] #6| true rating · ● Bradley–Terry, bar = bootstrap 95% CI · ◇ online Elo (start 1000)ranks correct 4/6 · pairs separated by CIs 11/15 · mean CI half-width ±34.8max error: Bradley–Terry 47.7 · online Elo 62.2same battles, reversed order: online Elo moves by up to 61.5; Bradley–Terry by 0
vote sample
fig.Six synthetic models with known ratings (1250, 1210, 1190, 1150, 1080, 1000) fight battles between uniformly chosen pairs. Defaults (1,000 battles, about 67 per pair, K = 32, seed 1): only 4 of 6 ranks are right, the intervals separate 11 of 15 pairs, the mean half-width is ±34.8 and the largest Bradley–Terry error is 47.7. Online Elo is off by up to 62.2, and replaying the same battles in reverse moves it by up to 61.5. At 4,000 battles the half-width shrinks to ±16.2 and 14 of 15 pairs separate, but birch and cedar, 20 points apart, are still swapped. With seed 2 at 1,000 battles all six ranks are right.

Set K=4K=4 with 200 battles: online Elo barely moves with the order (5.3 points) but stays compressed near its starting value of 1000, with errors up to 81.0. Small KK trades order noise for bias; Bradley–Terry has neither problem.

Worked example: three models

  1. 1Votes: A beats B 60–40, B beats C 60–40, A beats C 70–30 (300 battles).
  2. 2One pair alone: logit⁡(0.60)=ln⁡1.5=0.405\operatorname{logit}(0.60)=\ln1.5=0.405, i.e. 0.405⋅173.7=70.40.405\cdot173.7=70.4 Elo points.
  3. 3Joint Bradley–Terry fit (mean anchored at 1000): A 1072.4, B 1000.0, C 927.6. The gaps are 72.4, 72.4 and 144.8.
  4. 4Check: the model predicts P(A≻B)=0.603P(A\succ B)=0.603 and P(A≻C)=0.697P(A\succ C)=0.697, close to the observed 0.60 and 0.70. Bradley–Terry is transitive, so it pools evidence: the A–C votes sharpen A–B.
  5. 5Bootstrap (200 resamples): A [1041, 1105], B [964, 1031], C [890, 957]. A and B overlap, so with 100 votes per pair A is not clearly better than B.
python
1import numpy as np
2
3def fit_bt(battles, models, iters=50):
4 """Bradley–Terry by Newton's method on logistic loss; battles = [(winner, loser), ...]."""
5 idx = {m: i for i, m in enumerate(models)}
6 X = np.zeros((len(battles), len(models)))
7 for r, (w, l) in enumerate(battles):
8 X[r, idx[w]], X[r, idx[l]] = 1.0, -1.0 # every row: winner +1, loser -1, y = 1
9 beta = np.zeros(len(models))
10 for _ in range(iters):
11 p = 1 / (1 + np.exp(-X @ beta))
12 grad = X.T @ (1 - p) - 1e-6 * beta
13 hess = (X * (p * (1 - p))[:, None]).T @ X + 1e-6 * np.eye(len(models))
14 beta += np.linalg.solve(hess, grad)
15 beta -= beta.mean()
16 return 1000 + 400 / np.log(10) * beta # Elo scale, mean 1000
17
18models = ["A", "B", "C"]
19battles = ([("A", "B")] * 60 + [("B", "A")] * 40 + [("B", "C")] * 60 + [("C", "B")] * 40
20 + [("A", "C")] * 70 + [("C", "A")] * 30)
21print(np.round(fit_bt(battles, models), 1)) # [1072.4 1000. 927.6]
22
23rng = np.random.default_rng(0) # bootstrap 95% intervals
24boot = np.array([fit_bt([battles[i] for i in rng.integers(0, len(battles), len(battles))], models)
25 for _ in range(200)])
26print(np.round(np.percentile(boot, [2.5, 97.5], axis=0), 0))
Bradley–Terry ratings and bootstrap intervals from a battle log with numpy (runs as is)

Check yourself

Model X is rated 1205 [1188, 1224] and model Y 1191 [1177, 1208]. What should the leaderboard say?

Check yourself

Why did arena leaderboards move from online Elo to a Bradley–Terry fit?

Remember

  • ✓Pairwise, blind, randomised comparisons are the most reliable human signal for ranking generative systems.
  • ✓Report inter-annotator agreement (κ or α); it bounds how well any metric can match humans.
  • ✓Bradley–Terry: P(i≻j)=σ(βi−βj)P(i\succ j)=\sigma(\beta_i-\beta_j), fitted by MLE on all votes; Elo scale R=1000+173.7βR=1000+173.7\beta.
  • ✓Uncertainty is large: a 20-point gap needs about 1,200 votes on that pair; rank by bootstrap interval.
  • ✓Control for length and formatting, and remember arena ratings reflect their own users' prompts.
← back to Generative AI roadmap