# LLM evaluation and benchmarking

> LLM evaluation scores a model by perplexity, by log-likelihood ranking of fixed choices (MMLU, HellaSwag), by checking generated answers (GSM8K exact match, unbiased pass@k on HumanEval) or by pairwise judgments fitted with Bradley–Terry; a delta counts only with a paired confidence interval.

Source: https://www.mlfoundry.in/learn/llm-evaluation-and-benchmarking  
Section: Deep Learning › Large Language Models (LLMs)  
Level: core · 16 min read · updated 2026-10-13

**LLM evaluation:** is the measurement of what a language model can do, under a fixed and fully specified protocol: a dataset, a prompt format, a scoring rule and a statistic with its uncertainty. There are four main protocols. **Intrinsic** evaluation reports perplexity or bits per byte on held-out text. **Likelihood-based multiple choice** ranks fixed answer options by their log-probability, as in MMLU (Hendrycks et al., ICLR 2021). **Generative** evaluation samples an answer and checks it, by exact match on GSM8K (Cobbe et al., 2021) or by unit tests on HumanEval with the pass@k estimator of Chen et al. ("Evaluating Large Language Models Trained on Code", 2021). **Preference** evaluation compares two models' answers using an LLM judge (MT-Bench) or crowd votes (Chatbot Arena; Zheng et al., NeurIPS 2023; Chiang et al., ICML 2024), fitted with a Bradley–Terry model.

A **Language modeling** loss tells you how well a model predicts text. It does not tell you whether the model can answer a chemistry question, write a working function or follow an instruction. Before GPT-3 that gap was closed by fine-tuning: GLUE and SuperGLUE scored a pre-trained encoder such as **BERT** after training it on each task. Brown et al. ("Language Models are Few-Shot Learners", 2020) changed the setup. They evaluated one frozen model on dozens of tasks by putting zero, one or a few solved examples in the prompt. Since then a benchmark score has meant "this checkpoint, this prompt, this scorer", with no task-specific training at all.

That shift created the problems this article is about. Scores depend on prompt wording and answer extraction. Test sets leak into web-scale training data. Benchmarks saturate within a year or two, so new ones keep appearing: MMLU (2020), HumanEval and GSM8K (2021), BIG-bench (2022), GPQA and SWE-bench (2023). Open-ended chat quality has no answer key at all, so it is judged by humans or by other models. Benchmarks are often small, a few hundred items, so a two-point gap can be pure noise.

> Figure: The four protocols side by side. Perplexity needs only text. Multiple choice never lets the model generate: it compares the log-probability of each fixed option. Generative evaluation samples, then parses an answer or runs tests. Preference evaluation compares two models on the same prompt and turns many win/tie/loss verdicts into ratings. Whatever the protocol, the reported number needs its sample size, its standard error and the exact prompt format.

## Intrinsic evaluation: perplexity and bits per byte

$$
\mathrm{PPL}=\exp\Big(-\frac1T\sum_{t=1}^{T}\log p_\theta(x_t\mid x_{<t})\Big),\qquad \mathrm{BPB}=\frac{-\sum_{t=1}^{T}\log p_\theta(x_t\mid x_{<t})}{B\,\ln 2}
$$

- $T$: number of tokens in the held-out text
- $B$: number of UTF-8 bytes in the same text
- $\mathrm{PPL}$: effective branching factor: a model with PPL 10 is as unsure as a uniform choice among 10 tokens
- $\mathrm{BPB}$: bits per byte; independent of the tokenizer, used by The Pile (Gao et al. 2020)

Perplexity is cheap, smooth and has no prompt to tune, so it is the standard signal during **Pretraining and continued pretraining**. It has two limits. It is **per token**, so two models with different tokenizers are not comparable: a tokenizer with longer tokens spreads the same information over fewer, harder tokens. Bits per byte normalises by the text instead. For example, 1,000 tokens at mean NLL 2.3 nats give PPL $e^{2.3}=9.97$; if the text is 4,200 bytes, that is $1000\cdot2.3/(4200\ln2)=0.790$ bits per byte. The second limit is that perplexity predicts downstream ability only loosely. Within one model family, lower loss usually means better benchmarks. Across families, data mixtures and post-training it often does not, and post-training such as **RLHF** often *raises* perplexity on generic web text while making the model far more useful.

## Multiple choice by log-likelihood

For a question with fixed options, the cleanest protocol never lets the model generate. The harness builds the context (instructions, few-shot examples, the question), appends each option in turn, and reads off the log-probability of the option's tokens. The prediction is the option with the highest score. No parsing is needed and the result is deterministic. EleutherAI's lm-evaluation-harness and the early Open LLM Leaderboard used this protocol for ARC, HellaSwag and MMLU.

$$
s_j=\sum_{t=1}^{|a_j|}\log p_\theta\big(a_{j,t}\mid c,\,a_{j,<t}\big),\qquad \hat\jmath=\arg\max_j s_j\ \ (\texttt{acc}),\qquad \hat\jmath=\arg\max_j \frac{s_j}{\mathrm{len}(a_j)}\ \ (\texttt{acc\_norm})
$$

- $c$: context: instructions, few-shot examples and the question
- $a_j$: answer option $j$, tokenized as a continuation of $c$
- $s_j$: total log-probability of the option; a sum, so longer options are penalised
- $\mathrm{len}(a_j)$: lm-evaluation-harness divides by the option's character count; GPT-3 divided by token count

The sum is biased toward short options: every extra token adds another negative log-probability. On HellaSwag, where the four endings are full sentences of different lengths, this matters. Say the wrong ending has 2 tokens with total $-2.0$ and the right one has 6 tokens with total $-4.2$. The raw sum picks the wrong one, while per-token averages ($-1.0$ vs $-0.7$) pick the right one. Brown et al. used per-token normalisation for most tasks. For ARC, OpenBookQA and RACE they divided by the option's unconditional probability given only "Answer:", $p(a_j\mid c)/p(a_j\mid\text{"Answer:"})$, which removes the prior that some answers are simply common strings.

MMLU is usually scored differently. The options are shown in the prompt labelled A–D, and the harness compares the probabilities of the single letter tokens " A", " B", " C", " D" after "Answer:". This tests whether the model can map its knowledge onto a letter, which small base models do poorly. The Hugging Face team showed in 2023 that three popular MMLU implementations (the original code, HELM and the Eleuther harness) gave noticeably different scores, and even different rankings, for the same models. A score without its protocol cannot be compared.

## Generative evaluation and pass@k

Math and code need the model to produce the answer. GSM8K, 8.5K grade-school word problems needing 2 to 8 steps, is scored by extracting the final number and checking exact match. Chain-of-thought prompting (Wei et al., 2022) uses 8 worked exemplars. MATH (Hendrycks et al., NeurIPS 2021) asks for a final answer in $\boxed{\cdot}$, which must be normalised before comparison. For code, matching against a reference is useless because many programs are correct. HumanEval runs the completion against hidden unit tests.

Sampling makes one more question natural: if the model may try $k$ times, how often does at least one attempt pass? Chen et al. define **pass@k** as the expected fraction of problems solved by at least one of $k$ samples. Drawing exactly $k$ samples per problem gives a high-variance estimate. Instead they draw $n\ge k$ samples (they used $n=200$), count the $c$ that pass, and compute:

$$
\text{pass@}k=\mathbb E_{\text{problems}}\left[1-\frac{\binom{n-c}{k}}{\binom{n}{k}}\right]=\mathbb E\left[1-\prod_{i=n-c+1}^{n}\Big(1-\frac{k}{i}\Big)\right]
$$

- $n$: samples drawn per problem; must be at least k
- $c$: samples that pass every unit test
- $\binom{n-c}{k}/\binom{n}{k}$: probability that a random $k$-subset of the $n$ samples contains no correct one
- $\prod(1-k/i)$: the numerically stable form from the paper; binomials overflow for n = 200

Why is it unbiased? The $n$ samples are i.i.d. with success probability $p$. A uniformly random $k$-subset of them is itself $k$ i.i.d. samples, so the probability that the subset has no success, averaged over the draw, is exactly $(1-p)^k$. The obvious plug-in alternative $1-(1-\hat p)^k$ with $\hat p=c/n$ is **biased low**. The function $p\mapsto1-(1-p)^k$ is concave, so by Jensen's inequality $\mathbb E[1-(1-\hat p)^k]\le1-(1-p)^k$. The bias grows with $k$ and shrinks with $n$. At $k=1$ both estimators reduce to $c/n$.

**pass@k for one problem with n = 10 samples, c = 3 correct**

1. $k=1$: $1-\binom{7}{1}/\binom{10}{1}=1-7/10=0.300$, the same as $c/n$.
2. $k=5$: $\binom{7}{5}=21$ and $\binom{10}{5}=252$, so pass@5 $=1-21/252=0.917$. The product form agrees: $1-(1-\tfrac5{8})(1-\tfrac59)(1-\tfrac5{10})=1-\tfrac38\cdot\tfrac49\cdot\tfrac12=0.917$.
3. Plug-in estimate: $1-(1-0.3)^5=1-0.168=0.832$, which is 8.5 points lower.
4. $k=8>n-c=7$: any 8 of the 10 samples must include a correct one, so pass@8 $=1$ exactly. The plug-in gives $1-0.7^8=0.942$.

Temperature matters. Chen et al. found the best temperature rose with $k$: about $T=0.2$ for pass@1 and $T=0.8$ for pass@100, since diversity helps when any one success counts. Codex-12B solved 28.8% of HumanEval with one sample. Many current reports give greedy pass@1 instead, which is a different number. SWE-bench (Jimenez et al., ICLR 2024) extends the idea to real repositories: the model must produce a patch that makes the issue's failing tests pass without breaking the others.

**Benchmarks most often reported**

- **MMLU:** Hendrycks et al., ICLR 2021. 57 subjects from elementary maths to law, 4 options, 14,042 test questions, usually 5-shot. GPT-3 175B scored 43.9% few-shot against 25% chance
- **HellaSwag:** Zellers et al., ACL 2019. Pick the right continuation of a scene among 4; wrong endings chosen by adversarial filtering against BERT-era models. Humans above 95%; 10,042 validation items. Scored by length-normalised log-likelihood
- **ARC:** Clark et al., 2018. 7,787 grade-school science questions; the Challenge set (2,590) keeps those that both a retrieval and a word co-occurrence baseline got wrong
- **GSM8K:** Cobbe et al., 2021. 8.5K grade-school word problems (7,473 train, 1,319 test), 2 to 8 steps, integer answers, exact match
- **MATH:** Hendrycks et al., NeurIPS 2021. 12,500 competition problems (7,500 train, 5,000 test), 7 subjects, difficulty levels 1 to 5, final answer in $\boxed{\cdot}$. MATH-500 is the 500-problem test subset of Lightman et al. (2023)
- **HumanEval:** Chen et al., 2021. 164 hand-written Python functions with docstrings, on average 7.7 unit tests each; pass@k with n = 200 samples
- **TruthfulQA:** Lin, Hilton and Evans, ACL 2022. 817 questions in 38 categories built to trigger common misconceptions. Best model then truthful on 58% vs 94% for humans; larger models were often less truthful
- **BIG-bench:** Srivastava et al., 2022. 204 tasks from 450 authors at 132 institutions. BIG-Bench Hard (Suzgun et al., 2022) keeps 23 tasks on which earlier models fell below the average human rater
- **GPQA:** Rein et al., 2023. 448 graduate-level multiple-choice questions in biology, physics and chemistry. Experts 65%, skilled non-experts 34% even with unrestricted web access for 30+ minutes. Diamond subset: 198 questions
- **SWE-bench:** Jimenez et al., ICLR 2024. 2,294 GitHub issues from 12 Python repositories; graded by the repository's own tests. SWE-bench Verified (OpenAI, 2024) is a human-validated subset of 500
- **MT-Bench:** Zheng et al., NeurIPS 2023. 80 two-turn questions in 8 categories (writing, roleplay, extraction, reasoning, math, coding, STEM, humanities), graded 1 to 10 by GPT-4

## LLM-as-a-judge and Chatbot Arena

Open-ended answers have no key. Zheng et al. ("Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena", NeurIPS 2023) asked GPT-4 to grade answers, either one at a time on a 1–10 scale or as a pairwise comparison. GPT-4's verdicts agreed with human experts over 80% of the time, about as often as two humans agreed with each other. The same paper catalogued the judge's biases. **Position bias**: the judge prefers whichever answer comes first (or second), so verdicts can flip when the order is swapped. **Verbosity bias**: longer answers win even when the extra length adds nothing. **Self-enhancement**: a judge may favour answers written by its own model family. Judges are also weak at grading maths and reasoning they cannot solve themselves. The standard fixes are to run both orders and call it a tie when they disagree, to give the judge a reference answer, and to control for length. Length-controlled AlpacaEval (Dubois et al., 2024) regresses out the length difference before computing win rates.

Chatbot Arena (Chiang et al., ICML 2024) replaces the judge with crowds. A user types a prompt, two anonymous models answer, and the user votes for one, a tie, or both bad. The model names are revealed only after the vote. The pairwise outcomes are fitted with the **Bradley–Terry** model, the same model that underlies the reward model in **RLHF**:

$$
P(i\succ j)=\frac{1}{1+e^{-(\beta_i-\beta_j)}},\qquad R_i=1000+\frac{400}{\ln 10}\,\beta_i,\qquad P(i\succ j)=\frac{1}{1+10^{(R_j-R_i)/400}}
$$

- $\beta_i$: latent strength of model i, fitted by maximum likelihood (logistic regression on battle outcomes)
- $R_i$: the same strength on the Elo scale; 400/ln 10 ≈ 173.7 points per unit of log-odds
- $R_i-R_j=100$: means i wins 64.0% of battles; 200 means 76.0%; 400 means 90.9%

The leaderboard first used online Elo updates, $R_i\leftarrow R_i+K(S_i-E_i)$. For example, with $K=32$ a 1000-rated model that beats a 1100-rated one had expected score $0.360$ and gains $32\cdot0.640=20.5$ points. Online Elo depends on the order of battles and assumes players' strengths drift, which static model weights do not. Bradley–Terry maximum likelihood uses all battles at once and does not depend on their order. The Arena reports bootstrap confidence intervals for each rating. Models whose intervals overlap are effectively tied. The Arena later added "style control", which adds answer length and markdown use as covariates so that ratings are not driven by formatting.

## Few-shot, zero-shot and prompt sensitivity

Base models are usually evaluated few-shot: $k$ solved examples in the prompt show the format and the answer space. MMLU is typically 5-shot and GSM8K 8-shot with chain-of-thought. Instruction-tuned and chat models are often evaluated zero-shot through their chat template, because few-shot examples formatted as plain text can confuse them. Neither is "the" score. Reports should state the shot count, the template and whether chain-of-thought was used.

Scores are sensitive to choices that should not matter. Zhao et al. ("Calibrate Before Use", ICML 2021) found that the prompt format, the choice of few-shot examples and even their order could move GPT-3 accuracy from near chance to near state of the art. They traced this to majority-label, recency and common-token biases. Lu et al. (ACL 2022) found the same for example order. Sclar et al. (2023) varied only formatting (separators, casing, spacing) and saw accuracy spreads of up to 76 points for LLaMA-2-13B on some tasks. Multiple-choice answers also depend on which letter holds the right option. Robust evaluations average over several prompts or permute the options, and report the spread as well as the mean.

## Contamination

A benchmark measures generalisation only if its test items were not in training. Web-scale corpora make that hard to guarantee: benchmarks are posted on GitHub, discussed on forums and copied into other datasets. Brown et al. checked GPT-3 with 13-gram overlap between benchmarks and training data, and reported results on "clean" subsets. Overlap checks miss paraphrases and translations. Other signals include unusually high likelihood of the exact test text (Min-K% Prob, Shi et al., 2023), accuracy that drops on freshly written look-alike problems (GSM1k, Zhang et al., 2024), and accuracy that falls for problems published after the training cutoff (LiveCodeBench, Jain et al., 2024). BIG-bench embeds a **canary string** in its files so that training pipelines can filter them out. Fine-tuning on benchmark-like data is a related, softer problem: the model gets better at the format, not at the skill.

## How big is a benchmark difference?

An accuracy is a mean of $N$ Bernoulli outcomes. Its standard error is $\sqrt{p(1-p)/N}$. At 70% on MMLU's 14,042 questions that is 0.39 points. At 50% on GPQA Diamond's 198 questions it is 3.55 points, so a 95% interval is about $\pm7$ points. HumanEval's 164 problems give about $\pm3.9$ points of standard error at 50%. Miller ("Adding Error Bars to Evals", 2024) argues that every eval should report these, and should use clustered standard errors when questions come in groups, such as several questions about one passage.

$$
\hat\Delta=\bar d=\frac1N\sum_{i=1}^N (B_i-A_i),\qquad \mathrm{SE}_{\text{paired}}=\frac{s_d}{\sqrt N},\qquad s_d^2=\frac{N}{N-1}\Big(\frac{n_{10}+n_{01}}{N}-\hat\Delta^2\Big)
$$

- $A_i,B_i\in\{0,1\}$: whether each model got question i right
- $n_{10},\,n_{01}$: discordant questions: only A right, only B right. Questions both get right or both get wrong carry no information about the difference
- $\mathrm{SE}_{\text{paired}}$: smaller than the unpaired $\sqrt{\mathrm{SE}_A^2+\mathrm{SE}_B^2}$ whenever the two models find the same questions hard

Two models evaluated on the same questions are **paired**. Hard questions are hard for both, so most of the per-question variance cancels in the difference. Treating the two accuracies as independent overstates the uncertainty, and the correct test (McNemar's test, or a paired bootstrap that resamples questions) is often much sharper. The same logic gives the smallest difference worth trying to detect. With 80% power at $\alpha=0.05$ that is about $2.8\cdot\mathrm{SE}_{\text{paired}}$. If that exceeds the gap you care about, you need more questions, not more decimal places.

> Figure: Left toggle: a HumanEval-sized suite of 164 problems with fixed true per-sample pass rates (34 of them unsolvable, so pass@k never exceeds 79.3%). Each problem draws n samples from a fixed seeded stream. At the default n = 10, k = 10, the true pass@10 is 56.3% and the unbiased estimator reads 56.1%. That is simply the 92 of 164 problems with at least one correct sample. The plug-in 1 − (1 − c/n)^k gives 50.4% on this run, and its exact expectation over runs is 50.9%, a bias of −5.5 points. At k = 1 the two estimators coincide (22.2% on this run, true 22.5%). At n = 5, k = 5 the expected plug-in is 41.1% against a true 47.4% (bias −6.4 points). At Chen et al.'s n = 200 the bias shrinks: at k = 100 it is −1.3 points (74.4% vs 75.8%). Right toggle: two models answer the same N questions, A with true accuracy 70% and B with 70% plus the gap. ρ is the chance that B faces the same draw of difficulty as A. At the default N = 200, gap 3 pp and ρ = 0.5, the observed accuracies are 70.5% and 74.5% (Δ = +4.0 pp, 17 vs 25 discordant questions). The unpaired SE is 4.46 pp and the paired SE 3.24 pp. The paired 95% CI is [−2.3, +10.3] and the 500-resample bootstrap gives [−2.0, +10.0], so p = 0.216 and the gap is not resolved. At N = 1,000 the bootstrap interval is [+1.8, +7.4] (p = 0.0019). At MMLU's N = 14,042 the paired SE is 0.39 pp. At N = 50 the observed gap is +16.0 pp although the true one is 3. Setting ρ = 0.9 at N = 200 cuts the paired SE to 1.71 pp while the unpaired SE stays 4.46. With ρ = 0, the two SEs agree (4.54 vs 4.51).

## Worked example: is model B really better?

Two models answer the same 500 multiple-choice questions. A gets 70.0% and B gets 74.0%. Is the 4-point gap real? The per-question outcomes form a 2×2 table:

Paired outcomes on N = 500 questions (counts made up; statistics computed in Python)

| A vs B | B right | B wrong | total |
| --- | --- | --- | --- |
| A right | 320 | 30 | 350 (70.0%) |
| A wrong | 50 | 100 | 150 |
| total | 370 (74.0%) | 130 | 500 |

**Three ways to put an error bar on Δ = +4.0 points**

1. **Unpaired (wrong here).** $\mathrm{SE}=\sqrt{0.70\cdot0.30/500+0.74\cdot0.26/500}=2.84$ pp. The 95% CI is $4.0\pm1.96\cdot2.84=[-1.6,+9.6]$ pp, with $z=1.41$ and $p=0.159$: "not significant".
2. **Paired.** Only the $30+50=80$ discordant questions vary. $s_d^2=\tfrac{500}{499}(80/500-0.04^2)=0.1587$, so $\mathrm{SE}=\sqrt{0.1587/500}=1.78$ pp. The CI is $[+0.5,+7.5]$ pp, with $z=2.25$ and $p=0.025$.
3. **McNemar's test** uses the discordant counts alone: $\chi^2=(50-30)^2/(50+30)=5.0$ with 1 degree of freedom, so $p=0.025$. The exact binomial version (is 30 out of 80 consistent with $\tfrac12$?) gives $p=0.033$.
4. **Paired bootstrap** with 10,000 resamples of the 500 questions (the code below): 95% CI $[+0.6,+7.6]$ pp, in line with the CLT interval.
5. Conclusion: B is better at the 5% level, but the interval is wide. The true gap could plausibly be anywhere from about half a point to about seven and a half points. A claim of "4.0 points better" should come with that interval.

```python
import numpy as np
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

# ---- 1. multiple choice by log-likelihood (the acc / acc_norm protocol) ----
tok = AutoTokenizer.from_pretrained("gpt2")
model = AutoModelForCausalLM.from_pretrained("gpt2").eval()

@torch.no_grad()
def option_logprob(context: str, option: str) -> float:
    """Sum of log p(option tokens | context). Context and option are tokenized
    separately and concatenated, so the option's tokens are exactly its own."""
    ctx = tok(context).input_ids
    opt = tok(option, add_special_tokens=False).input_ids
    ids = torch.tensor([ctx + opt])
    logp = model(ids).logits[0, :-1].log_softmax(-1)      # row t predicts token t+1
    targets = ids[0, 1:]
    lp = logp.gather(-1, targets[:, None])[:, 0]
    return lp[-len(opt):].sum().item()                    # keep only the option's tokens

def score_mc(context: str, options: list[str], gold: int) -> tuple[bool, bool]:
    s = np.array([option_logprob(context, " " + o) for o in options])
    chars = np.array([len(o) for o in options], dtype=float)
    return int(s.argmax()) == gold, int((s / chars).argmax()) == gold   # acc, acc_norm

ctx = "A man is sitting on a roof. He"
opts = ["is using wrap to wrap a pair of skis.", "is ripping level tiles off.",
        "is holding a rubik's cube.", "starts pulling up roofing on a roof."]
print(score_mc(ctx, opts, gold=3))

# ---- 2. pass@k: the unbiased estimator of Chen et al. (2021) ----
def pass_at_k(n: int, c: int, k: int) -> float:
    """1 - C(n-c, k) / C(n, k), computed as a stable product."""
    if n - c < k:
        return 1.0
    return 1.0 - np.prod(1.0 - k / np.arange(n - c + 1, n + 1))

print(f"{pass_at_k(10, 3, 1):.4f} {pass_at_k(10, 3, 5):.4f} {1 - 0.7 ** 5:.4f}")   # 0.3000 0.9167 0.8319 (plug-in)

# ---- 3. paired bootstrap CI for acc(B) - acc(A) on the same questions ----
def paired_bootstrap(a: np.ndarray, b: np.ndarray, reps: int = 10_000, seed: int = 0):
    d = b.astype(float) - a.astype(float)                 # per-question difference in {-1, 0, 1}
    se = d.std(ddof=1) / np.sqrt(len(d))                  # paired CLT standard error
    rng = np.random.default_rng(seed)
    idx = rng.integers(0, len(d), size=(reps, len(d)))    # resample questions, not answers
    boot = d[idx].mean(axis=1)
    return d.mean(), se, np.percentile(boot, [2.5, 97.5])

# the worked example: 320 both right, 30 only A, 50 only B, 100 both wrong
a = np.array([1] * 320 + [1] * 30 + [0] * 50 + [0] * 100)
b = np.array([1] * 320 + [0] * 30 + [1] * 50 + [0] * 100)
delta, se, ci = paired_bootstrap(a, b)
print(f"delta={delta:+.3f}  paired SE={se:.4f}  bootstrap 95% CI={ci.round(3)}")
# delta=+0.040  paired SE=0.0178  bootstrap 95% CI=[0.006 0.076]

# ---- 4. Bradley-Terry ratings from pairwise battles (MM algorithm, Hunter 2004) ----
def bradley_terry(wins: np.ndarray, iters: int = 1000) -> np.ndarray:
    """wins[i, j] = battles i won against j (count a tie as half a win for each side)."""
    games = wins + wins.T
    p = np.ones(len(wins))
    for _ in range(iters):
        p = wins.sum(axis=1) / (games / (p[:, None] + p[None, :])).sum(axis=1)
        p /= np.exp(np.log(p).mean())                     # fix the scale: geometric mean 1
    return 1000 + 400 * np.log10(p)                       # the Elo-like scale the Arena reports

wins = np.array([[0, 60, 75],                             # model 0 beat model 1 60 times, model 2 75 times
                 [40, 0, 55],
                 [25, 45, 0]], dtype=float)
print(bradley_terry(wins).round(1))                       # [1085.3  987.8  926.9]
```

Likelihood scoring of multiple choice with Hugging Face transformers (not run here), plus pass@k, a paired bootstrap and a Bradley–Terry fit in numpy (run; outputs shown)

> **The protocol is part of the number:** "MMLU 68.9" means nothing until you know the shot count, the prompt template, whether letters or full options were scored, whether chain-of-thought was used and how answers were extracted. Changing any of these can move a score by several points, more than the gap between neighbouring models. Compare models only under one harness and one configuration that you ran yourself. Watch for answer-extraction failures too: a chat model that writes "The answer is (B)" instead of "B" can be scored as wrong by a strict regex.

> **Pair everything:** Two models on the same questions share most of their variance, because hard questions are hard for both. The paired standard error uses only the discordant questions and can be several times smaller than the unpaired one. In the figure at $N=200$, raising $\rho$ from 0 to 0.9 cuts it from 4.54 to 1.71 points. The same idea applies to seeds and prompts: compare models on the same seeds and prompt variants, and analyse the differences.

> **Benchmarks saturate and leak:** A benchmark is most informative while the best models are well below its ceiling. Once they reach it, the remaining errors are often label noise: re-annotation studies have found wrong or ambiguous keys in a noticeable share of MMLU questions. Gains near the ceiling may also come from contamination rather than skill. Treat a public benchmark as a shrinking resource. Keep a private, held-out set that resembles your real use, and refresh it.

**Strengths**

- Log-likelihood multiple choice is deterministic, cheap and needs no answer parsing; it gives signal even for small base models
- Generative exact match and unit tests check what users care about, the final answer or a working program
- Unbiased pass@k from n ≥ k samples has far lower variance than drawing exactly k samples
- LLM judges and Arena votes cover open-ended quality that has no answer key
- Bradley–Terry ratings put many models on one scale from sparse pairwise comparisons

**Limitations**

- Scores depend on prompt, shot count and scoring details; harnesses disagree on the same model
- Static test sets leak into web-scale training data and saturate within a few years
- Small benchmarks (164 to a few hundred items) have standard errors of 2 to 4 points
- LLM judges show position, verbosity and self-enhancement biases
- Arena ratings reflect the prompts and tastes of the people who vote, and reward style as well as substance
- Perplexity is not comparable across tokenizers and tracks downstream ability only loosely

- **Pretraining ablations:** Validation loss and bits per byte, plus log-likelihood multiple choice (HellaSwag, ARC), give smooth signals on small models where generative scores are still near zero.
- **Post-training regression tests:** A fixed suite of generative tasks (GSM8K, HumanEval, instruction following), run on every checkpoint with paired comparisons against the previous one.
- **Model selection:** A private set drawn from your own traffic, graded by tests, rubrics or an LLM judge with both answer orders, is what predicts production quality.
- **Public leaderboards:** Chatbot Arena and benchmark tables compare released models; read them with their confidence intervals and protocols.

**Question:** A problem gets n = 10 samples and exactly c = 1 passes. What is the unbiased pass@5 estimate?

- A. 0.10, the fraction of samples that passed
- B. 0.41, from 1 − (1 − 0.1)^5
- C. 0.50, from 1 − C(9, 5)/C(10, 5)
- D. 1.00, because some sample passed

**Answer:** C. 0.50, from 1 − C(9, 5)/C(10, 5) — A random 5-subset of the 10 samples misses the single correct one with probability $\binom95/\binom{10}5=126/252=0.5$, so pass@5 $=0.5$. The plug-in $1-0.9^5=0.410$ is biased low, because $1-(1-p)^k$ is concave in $p$.

**Question:** On GSM8K's 1,319 test problems, model A scores 90.0% and model B scores 91.5% (B gets 20 more right). When is B's lead statistically significant at the 5% level?

- A. Always: 1.5 points on 1,319 problems is large
- B. Never: the unpaired SE is 1.13 points, so z ≈ 1.3
- C. It depends on the discordant questions: with 10 only-A and 30 only-B it is (p ≈ 0.002), with 100 and 120 it is not (p ≈ 0.18)
- D. Only if both models were run at temperature 0

**Answer:** C. It depends on the discordant questions: with 10 only-A and 30 only-B it is (p ≈ 0.002), with 100 and 120 it is not (p ≈ 0.18) — Both scenarios give $\hat\Delta=20/1319=1.52$ points. With 40 discordant questions the paired SE is 0.48 points ($z=3.17$; McNemar $\chi^2=400/40=10$). With 220 discordant questions it is 1.12 points ($z=1.35$, $p=0.18$). The two accuracies alone cannot decide; the per-question outcomes can.

**Key points**

- Perplexity $\exp(\text{mean NLL})$ is per token, so it compares only models with one tokenizer; bits per byte fixes that, but neither predicts downstream ability reliably across model families.
- Multiple choice by log-likelihood picks $\arg\max_j\sum_t\log p(a_{j,t}\mid c,a_{j,<t})$; length normalisation (`acc_norm`) removes the bias toward short options.
- pass@k $=\mathbb E[1-\binom{n-c}{k}/\binom nk]$ from $n\ge k$ samples is unbiased; the plug-in $1-(1-c/n)^k$ is biased low by Jensen's inequality.
- LLM judges agree with humans about as often as humans agree with each other on MT-Bench, but they have position, verbosity and self-enhancement biases. Swap the answer order and control for length.
- Chatbot Arena fits Bradley–Terry, $P(i\succ j)=1/(1+10^{(R_j-R_i)/400})$, to crowd votes and reports bootstrap intervals; 100 rating points means a 64% win rate.
- Contamination, prompt format, shot count and answer extraction can each move scores by more than the gaps being reported.
- Report $N$ and a paired confidence interval: the SE of an accuracy is $\sqrt{p(1-p)/N}$ (3.55 points on 198 questions at 50%), and paired tests on discordant questions are much sharper than unpaired ones.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
