# Human evaluation and arena-style ranking

> People compare two anonymous answers and vote; a Bradley–Terry fit turns votes into ratings with bootstrap intervals. Rank by interval, not point estimate: a 20-point gap needs about 1,200 votes on that pair.

Source: https://www.mlfoundry.in/learn/human-evaluation-and-arena-style-ranking  
Section: Generative AI › Evaluation of Generative Models  
Level: core · 8 min read · updated 2026-10-21

**Arena-style ranking:** ranks generative models from many pairwise human votes. A user sends a prompt, two anonymous models answer side by side, the user votes for one (or a tie), and only then are the names revealed. The votes are fitted with the Bradley–Terry model (Bradley and Terry, 1952) and reported on the Elo scale. The best-known instance is Chatbot Arena (Chiang et al., "Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference", ICML 2024), launched by LMSYS in 2023.

Human judgment is the reference that every automatic metric is validated against, including **LLM-as-a-judge**. It is also where product decisions are made: Ouyang et al. (2022) reported that labelers preferred outputs of a 1.3B-parameter InstructGPT model to those of the 175B GPT-3, a result no benchmark score would have shown. The cost is time and noise, so the design of the study decides whether its numbers mean anything.

## Designing a human evaluation

Common formats. Pairwise is the default for ranking systems; error annotation for finding out why one is worse.

| Format | Rater task | Good for | Watch out for |
| --- | --- | --- | --- |
| Absolute (Likert) | rate one output 1–5 on a criterion | tracking one system over time | raters use the scale differently; scores drift |
| Pairwise preference | pick the better of A and B, or tie | comparing systems; small differences | needs a model (Bradley–Terry) to rank many systems |
| Ranking | order k outputs | collecting preference data for **RLHF** | harder per item; expands to k(k−1)/2 pairs |
| Error annotation | mark error spans with type and severity | diagnosis (MQM in translation) | slow; needs trained annotators |

1. **Write the question you want answered**: "is v2 more helpful than v1 on billing questions?" fixes the prompts, the criterion and the comparison.
2. **Sample prompts from real traffic** or a versioned set, stratified by task; hold them fixed across systems.
3. **Blind and randomise**: hide system names, shuffle left/right placement and item order per rater.
4. **Write guidelines with worked examples** of each label, including ties and "both bad"; pilot on 20–50 items and revise.
5. **Collect overlapping labels** (2–3 raters per item on at least a subset) and report inter-annotator agreement, e.g. Cohen's κ or Krippendorff's α.
6. **Report win rate with a confidence interval** and the number of votes, not only a point estimate.

Who rates matters. Clark et al. ("All That's 'Human' Is Not Gold", ACL 2021) found that untrained evaluators distinguished GPT-3 text from human text at roughly chance level. Freitag et al. (TACL 2021) found that professional translators using MQM error annotation ranked MT systems differently from crowd raters. Domain tasks (medicine, law, code) need domain raters, and agreement between them sets the ceiling for any automatic metric.

> Figure: One arena battle: a prompt goes to two hidden models, the user votes A better, B better, tie or both bad, and only then are the names revealed. Battles accumulate in a log, a Bradley–Terry fit turns them into ratings, and bootstrapping the battles gives 95% intervals. Here birch (1205) and cedar (1191) share rank 2 because their intervals overlap, while atlas (1242) is clear of both. Below: the three basic human-evaluation formats.

## From votes to ratings

$$
P(i\succ j)=\sigma(\beta_i-\beta_j),\qquad \hat\beta=\arg\max_\beta\sum_{(w,l)}\log\sigma(\beta_w-\beta_l),\qquad R_i=1000+\frac{400}{\ln 10}\,\beta_i
$$

- $\beta_i$: latent strength of model i; only differences are identified, so one model or the mean is anchored
- $(w,l)$: winner and loser of one battle; a tie is commonly split as half a win each
- $R_i$: Elo scale: 400/ln 10 ≈ 173.7 points per unit of log-odds; a 100-point gap means 64.0% wins

This is logistic regression with one $+1$ and one $-1$ per row, the same model as the reward model in **RLHF**. The maximum-likelihood fit uses all battles at once, so the result does not depend on their order. Chatbot Arena first published **online Elo**, $R_i\leftarrow R_i+K(S_i-E_i)$, and later switched to the Bradley–Terry fit. Online Elo assumes players' strengths drift, as in chess. A frozen model's strength does not drift, so order-dependence is pure noise.

$$
\mathrm{SE}(\hat p)=\sqrt{\frac{p(1-p)}{n}},\qquad n_{95\%}\approx\left(\frac{1.96\cdot0.5}{p-0.5}\right)^{2}
$$

- $\hat p$: observed win rate of one model over another in n votes
- $n_{95\%}$: votes on one pair for the 95% interval half-width to equal the true gap $p-0.5$

A 20-point gap is a 52.9% win rate and needs about 1,162 votes on that pair; a 50-point gap (57.1%) needs about 188. With 1,000 votes the half-width is $\pm0.031$, about $\pm22$ Elo points. Leaderboards therefore publish bootstrap intervals, and a rank is only meaningful as "1 + the number of models whose interval lies entirely above". Ties at the top are normal.

**Arena practice**

- **Votes:** A better, B better, tie, both bad. Names are hidden until after the vote.
- **Scale:** Chiang et al. (2024) reported over 240,000 votes at the time of the paper; the live counts are far larger and change daily.
- **Fit:** Bradley–Terry by maximum likelihood, with bootstrap confidence intervals; it replaced the initial online Elo.
- **Style control:** Answer length and markdown use enter the regression as covariates, so ratings are not driven by formatting (added in 2024).
- **Bias:** Voters choose their own prompts, so the ranking reflects the user population's tasks and tastes, not a fixed benchmark.

> Figure: Six synthetic models with known ratings (1250, 1210, 1190, 1150, 1080, 1000) fight battles between uniformly chosen pairs. Defaults (1,000 battles, about 67 per pair, K = 32, seed 1): only 4 of 6 ranks are right, the intervals separate 11 of 15 pairs, the mean half-width is ±34.8 and the largest Bradley–Terry error is 47.7. Online Elo is off by up to 62.2, and replaying the same battles in reverse moves it by up to 61.5. At 4,000 battles the half-width shrinks to ±16.2 and 14 of 15 pairs separate, but birch and cedar, 20 points apart, are still swapped. With seed 2 at 1,000 battles all six ranks are right.

Set $K=4$ with 200 battles: online Elo barely moves with the order (5.3 points) but stays compressed near its starting value of 1000, with errors up to 81.0. Small $K$ trades order noise for bias; Bradley–Terry has neither problem.

## Worked example: three models

1. Votes: A beats B 60–40, B beats C 60–40, A beats C 70–30 (300 battles).
2. One pair alone: $\operatorname{logit}(0.60)=\ln1.5=0.405$, i.e. $0.405\cdot173.7=70.4$ Elo points.
3. Joint Bradley–Terry fit (mean anchored at 1000): A 1072.4, B 1000.0, C 927.6. The gaps are 72.4, 72.4 and 144.8.
4. Check: the model predicts $P(A\succ B)=0.603$ and $P(A\succ C)=0.697$, close to the observed 0.60 and 0.70. Bradley–Terry is transitive, so it pools evidence: the A–C votes sharpen A–B.
5. Bootstrap (200 resamples): A [1041, 1105], B [964, 1031], C [890, 957]. A and B overlap, so with 100 votes per pair A is not clearly better than B.

```python
import numpy as np

def fit_bt(battles, models, iters=50):
    """Bradley–Terry by Newton's method on logistic loss; battles = [(winner, loser), ...]."""
    idx = {m: i for i, m in enumerate(models)}
    X = np.zeros((len(battles), len(models)))
    for r, (w, l) in enumerate(battles):
        X[r, idx[w]], X[r, idx[l]] = 1.0, -1.0          # every row: winner +1, loser -1, y = 1
    beta = np.zeros(len(models))
    for _ in range(iters):
        p = 1 / (1 + np.exp(-X @ beta))
        grad = X.T @ (1 - p) - 1e-6 * beta
        hess = (X * (p * (1 - p))[:, None]).T @ X + 1e-6 * np.eye(len(models))
        beta += np.linalg.solve(hess, grad)
    beta -= beta.mean()
    return 1000 + 400 / np.log(10) * beta              # Elo scale, mean 1000

models = ["A", "B", "C"]
battles = ([("A", "B")] * 60 + [("B", "A")] * 40 + [("B", "C")] * 60 + [("C", "B")] * 40
           + [("A", "C")] * 70 + [("C", "A")] * 30)
print(np.round(fit_bt(battles, models), 1))            # [1072.4 1000.   927.6]

rng = np.random.default_rng(0)                         # bootstrap 95% intervals
boot = np.array([fit_bt([battles[i] for i in rng.integers(0, len(battles), len(battles))], models)
                 for _ in range(200)])
print(np.round(np.percentile(boot, [2.5, 97.5], axis=0), 0))
```

Bradley–Terry ratings and bootstrap intervals from a battle log with numpy (runs as is)

> **Style wins votes:** Raters, like LLM judges, prefer longer, well-formatted, confident answers. A model can climb a leaderboard by adding headers and bullet points without becoming more correct. Control for length and formatting in the regression, and check correctness separately on tasks with verifiable answers.

> **Arena ratings measure one population:** Self-selected users with self-chosen prompts skew toward chat, coding and curiosity questions. A rating gap there may not transfer to your support tickets or medical notes. For a product decision, run your own blind pairwise study on your own prompts; the same Bradley–Terry machinery applies.

**Question:** Model X is rated 1205 [1188, 1224] and model Y 1191 [1177, 1208]. What should the leaderboard say?

- A. X is better, since 1205 > 1191
- B. Y is better, since its interval is narrower
- C. X and Y share a rank: their intervals overlap
- D. Both ratings are invalid

**Answer:** C. X and Y share a rank: their intervals overlap — A 14-point gap is a 52% win rate, well inside the sampling noise of these vote counts. Ranking by "1 + number of models clearly above" gives both the same rank. More votes on the X–Y pair are the only way to separate them.

**Question:** Why did arena leaderboards move from online Elo to a Bradley–Terry fit?

- A. Bradley–Terry is faster to update after each vote
- B. Model weights are frozen, so order-dependent updates add noise; the joint MLE uses all votes at once and is order-independent
- C. Elo cannot handle ties
- D. Bradley–Terry does not need a scale

**Answer:** B. Model weights are frozen, so order-dependent updates add noise; the joint MLE uses all votes at once and is order-independent — Online Elo is designed for players whose skill changes over time; its result depends on the order of games and on $K$. A frozen model's strength is fixed, so the maximum-likelihood fit on all battles is the right estimator. In the figure, reversing the battle order moves online Elo by up to 61.5 points and Bradley–Terry by 0.

**Key points**

- Pairwise, blind, randomised comparisons are the most reliable human signal for ranking generative systems.
- Report inter-annotator agreement (κ or α); it bounds how well any metric can match humans.
- Bradley–Terry: $P(i\succ j)=\sigma(\beta_i-\beta_j)$, fitted by MLE on all votes; Elo scale $R=1000+173.7\beta$.
- Uncertainty is large: a 20-point gap needs about 1,200 votes on that pair; rank by bootstrap interval.
- Control for length and formatting, and remember arena ratings reflect their own users' prompts.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
