LLM evaluation and benchmarking

LLM evaluation scores a model by perplexity, by log-likelihood ranking of fixed choices (MMLU, HellaSwag), by checking generated answers (GSM8K exact match, unbiased pass@k on HumanEval) or by pairwise judgments fitted with Bradley–Terry; a delta counts only with a paired confidence interval.

16 min read2472 wordscore0 viewsupdated 2026-10-13

Definition

LLM evaluation is the measurement of what a language model can do, under a fixed and fully specified protocol: a dataset, a prompt format, a scoring rule and a statistic with its uncertainty. There are four main protocols. Intrinsic evaluation reports perplexity or bits per byte on held-out text. Likelihood-based multiple choice ranks fixed answer options by their log-probability, as in MMLU (Hendrycks et al., ICLR 2021). Generative evaluation samples an answer and checks it, by exact match on GSM8K (Cobbe et al., 2021) or by unit tests on HumanEval with the pass@k estimator of Chen et al. ("Evaluating Large Language Models Trained on Code", 2021). Preference evaluation compares two models' answers using an LLM judge (MT-Bench) or crowd votes (Chatbot Arena; Zheng et al., NeurIPS 2023; Chiang et al., ICML 2024), fitted with a Bradley–Terry model.

A Language modeling loss tells you how well a model predicts text. It does not tell you whether the model can answer a chemistry question, write a working function or follow an instruction. Before GPT-3 that gap was closed by fine-tuning: GLUE and SuperGLUE scored a pre-trained encoder such as BERT after training it on each task. Brown et al. ("Language Models are Few-Shot Learners", 2020) changed the setup. They evaluated one frozen model on dozens of tasks by putting zero, one or a few solved examples in the prompt. Since then a benchmark score has meant "this checkpoint, this prompt, this scorer", with no task-specific training at all.

That shift created the problems this article is about. Scores depend on prompt wording and answer extraction. Test sets leak into web-scale training data. Benchmarks saturate within a year or two, so new ones keep appearing: MMLU (2020), HumanEval and GSM8K (2021), BIG-bench (2022), GPQA and SWE-bench (2023). Open-ended chat quality has no answer key at all, so it is judged by humans or by other models. Benchmarks are often small, a few hundred items, so a two-point gap can be pure noise.

One model, four evaluation protocolshover a laneLLMp_θ(x)perplexity: held-out text, no task, no promptheld-out corpusx₁ … x_Tper-token NLL−log p(x_t | x<t)PPL = exp(mean)or bits per bytelower is bettersame tokenizer onlymultiple choice by log-likelihood: MMLU, HellaSwag, ARCcontext + choice jj = A, B, C, DΣ log p(choice)÷ length (acc_norm)argmax over j= gold answer?accuracychance = 1 / #choicesgenerative: GSM8K, MATH, HumanEval, SWE-benchprompt, 0/k-shotoptional CoTsample n outputsT = 0 … 0.8parse final answeror run unit testsexact matchpass@k (unbiased)pairwise preference: MT-Bench (LLM judge), Chatbot Arena (humans)one prompt →model A, model Bjudge: human/LLMswap A/B orderwin / tie / lossmany battlesBradley–Terry fitratings ± bootstrapevery number: N items · SE = √(p(1−p)/N) · paired CI on deltas · seeds and temperatureexact prompt, shot count and answer format · contamination check against the training data
fig.The four protocols side by side. Perplexity needs only text. Multiple choice never lets the model generate: it compares the log-probability of each fixed option. Generative evaluation samples, then parses an answer or runs tests. Preference evaluation compares two models on the same prompt and turns many win/tie/loss verdicts into ratings. Whatever the protocol, the reported number needs its sample size, its standard error and the exact prompt format.

Intrinsic evaluation: perplexity and bits per byte

PPL=exp⁡(−1T∑t=1Tlog⁡pθ(xt∣x<t)),BPB=−∑t=1Tlog⁡pθ(xt∣x<t)B ln⁡2\mathrm{PPL}=\exp\Big(-\frac1T\sum_{t=1}^{T}\log p_\theta(x_t\mid x_{<t})\Big),\qquad \mathrm{BPB}=\frac{-\sum_{t=1}^{T}\log p_\theta(x_t\mid x_{<t})}{B\,\ln 2}
TT
number of tokens in the held-out text
BB
number of UTF-8 bytes in the same text
PPL\mathrm{PPL}
effective branching factor: a model with PPL 10 is as unsure as a uniform choice among 10 tokens
BPB\mathrm{BPB}
bits per byte; independent of the tokenizer, used by The Pile (Gao et al. 2020)

Perplexity is cheap, smooth and has no prompt to tune, so it is the standard signal during Pretraining and continued pretraining. It has two limits. It is per token, so two models with different tokenizers are not comparable: a tokenizer with longer tokens spreads the same information over fewer, harder tokens. Bits per byte normalises by the text instead. For example, 1,000 tokens at mean NLL 2.3 nats give PPL e2.3=9.97e^{2.3}=9.97; if the text is 4,200 bytes, that is 1000⋅2.3/(4200ln⁡2)=0.7901000\cdot2.3/(4200\ln2)=0.790 bits per byte. The second limit is that perplexity predicts downstream ability only loosely. Within one model family, lower loss usually means better benchmarks. Across families, data mixtures and post-training it often does not, and post-training such as RLHF often raises perplexity on generic web text while making the model far more useful.

Multiple choice by log-likelihood

For a question with fixed options, the cleanest protocol never lets the model generate. The harness builds the context (instructions, few-shot examples, the question), appends each option in turn, and reads off the log-probability of the option's tokens. The prediction is the option with the highest score. No parsing is needed and the result is deterministic. EleutherAI's lm-evaluation-harness and the early Open LLM Leaderboard used this protocol for ARC, HellaSwag and MMLU.

sj=∑t=1∣aj∣log⁡pθ(aj,t∣c, aj,<t),ȷ^=arg⁡max⁡jsj  (acc),ȷ^=arg⁡max⁡jsjlen(aj)  (acc_norm)s_j=\sum_{t=1}^{|a_j|}\log p_\theta\big(a_{j,t}\mid c,\,a_{j,<t}\big),\qquad \hat\jmath=\arg\max_j s_j\ \ (\texttt{acc}),\qquad \hat\jmath=\arg\max_j \frac{s_j}{\mathrm{len}(a_j)}\ \ (\texttt{acc\_norm})
cc
context: instructions, few-shot examples and the question
aja_j
answer option jj, tokenized as a continuation of cc
sjs_j
total log-probability of the option; a sum, so longer options are penalised
len(aj)\mathrm{len}(a_j)
lm-evaluation-harness divides by the option's character count; GPT-3 divided by token count

The sum is biased toward short options: every extra token adds another negative log-probability. On HellaSwag, where the four endings are full sentences of different lengths, this matters. Say the wrong ending has 2 tokens with total −2.0-2.0 and the right one has 6 tokens with total −4.2-4.2. The raw sum picks the wrong one, while per-token averages (−1.0-1.0 vs −0.7-0.7) pick the right one. Brown et al. used per-token normalisation for most tasks. For ARC, OpenBookQA and RACE they divided by the option's unconditional probability given only "Answer:", p(aj∣c)/p(aj∣"Answer:")p(a_j\mid c)/p(a_j\mid\text{"Answer:"}), which removes the prior that some answers are simply common strings.

MMLU is usually scored differently. The options are shown in the prompt labelled A–D, and the harness compares the probabilities of the single letter tokens " A", " B", " C", " D" after "Answer:". This tests whether the model can map its knowledge onto a letter, which small base models do poorly. The Hugging Face team showed in 2023 that three popular MMLU implementations (the original code, HELM and the Eleuther harness) gave noticeably different scores, and even different rankings, for the same models. A score without its protocol cannot be compared.

Generative evaluation and pass@k

Math and code need the model to produce the answer. GSM8K, 8.5K grade-school word problems needing 2 to 8 steps, is scored by extracting the final number and checking exact match. Chain-of-thought prompting (Wei et al., 2022) uses 8 worked exemplars. MATH (Hendrycks et al., NeurIPS 2021) asks for a final answer in ⋅\boxed{\cdot}, which must be normalised before comparison. For code, matching against a reference is useless because many programs are correct. HumanEval runs the completion against hidden unit tests.

Sampling makes one more question natural: if the model may try kk times, how often does at least one attempt pass? Chen et al. define pass@k as the expected fraction of problems solved by at least one of kk samples. Drawing exactly kk samples per problem gives a high-variance estimate. Instead they draw n≥kn\ge k samples (they used n=200n=200), count the cc that pass, and compute:

pass@k=Eproblems[1−(n−ck)(nk)]=E[1−∏i=n−c+1n(1−ki)]\text{pass@}k=\mathbb E_{\text{problems}}\left[1-\frac{\binom{n-c}{k}}{\binom{n}{k}}\right]=\mathbb E\left[1-\prod_{i=n-c+1}^{n}\Big(1-\frac{k}{i}\Big)\right]
nn
samples drawn per problem; must be at least k
cc
samples that pass every unit test
(n−ck)/(nk)\binom{n-c}{k}/\binom{n}{k}
probability that a random kk-subset of the nn samples contains no correct one
∏(1−k/i)\prod(1-k/i)
the numerically stable form from the paper; binomials overflow for n = 200

Why is it unbiased? The nn samples are i.i.d. with success probability pp. A uniformly random kk-subset of them is itself kk i.i.d. samples, so the probability that the subset has no success, averaged over the draw, is exactly (1−p)k(1-p)^k. The obvious plug-in alternative 1−(1−p^)k1-(1-\hat p)^k with p^=c/n\hat p=c/n is biased low. The function p↦1−(1−p)kp\mapsto1-(1-p)^k is concave, so by Jensen's inequality E[1−(1−p^)k]≤1−(1−p)k\mathbb E[1-(1-\hat p)^k]\le1-(1-p)^k. The bias grows with kk and shrinks with nn. At k=1k=1 both estimators reduce to c/nc/n.

pass@k for one problem with n = 10 samples, c = 3 correct

  1. 1k=1k=1: 1−(71)/(101)=1−7/10=0.3001-\binom{7}{1}/\binom{10}{1}=1-7/10=0.300, the same as c/nc/n.
  2. 2k=5k=5: (75)=21\binom{7}{5}=21 and (105)=252\binom{10}{5}=252, so pass@5 =1−21/252=0.917=1-21/252=0.917. The product form agrees: 1−(1−58)(1−59)(1−510)=1−38⋅49⋅12=0.9171-(1-\tfrac5{8})(1-\tfrac59)(1-\tfrac5{10})=1-\tfrac38\cdot\tfrac49\cdot\tfrac12=0.917.
  3. 3Plug-in estimate: 1−(1−0.3)5=1−0.168=0.8321-(1-0.3)^5=1-0.168=0.832, which is 8.5 points lower.
  4. 4k=8>n−c=7k=8>n-c=7: any 8 of the 10 samples must include a correct one, so pass@8 =1=1 exactly. The plug-in gives 1−0.78=0.9421-0.7^8=0.942.

Temperature matters. Chen et al. found the best temperature rose with kk: about T=0.2T=0.2 for pass@1 and T=0.8T=0.8 for pass@100, since diversity helps when any one success counts. Codex-12B solved 28.8% of HumanEval with one sample. Many current reports give greedy pass@1 instead, which is a different number. SWE-bench (Jimenez et al., ICLR 2024) extends the idea to real repositories: the model must produce a patch that makes the issue's failing tests pass without breaking the others.

Benchmarks most often reported

MMLU
Hendrycks et al., ICLR 2021. 57 subjects from elementary maths to law, 4 options, 14,042 test questions, usually 5-shot. GPT-3 175B scored 43.9% few-shot against 25% chance
HellaSwag
Zellers et al., ACL 2019. Pick the right continuation of a scene among 4; wrong endings chosen by adversarial filtering against BERT-era models. Humans above 95%; 10,042 validation items. Scored by length-normalised log-likelihood
ARC
Clark et al., 2018. 7,787 grade-school science questions; the Challenge set (2,590) keeps those that both a retrieval and a word co-occurrence baseline got wrong
GSM8K
Cobbe et al., 2021. 8.5K grade-school word problems (7,473 train, 1,319 test), 2 to 8 steps, integer answers, exact match
MATH
Hendrycks et al., NeurIPS 2021. 12,500 competition problems (7,500 train, 5,000 test), 7 subjects, difficulty levels 1 to 5, final answer in ⋅\boxed{\cdot}. MATH-500 is the 500-problem test subset of Lightman et al. (2023)
HumanEval
Chen et al., 2021. 164 hand-written Python functions with docstrings, on average 7.7 unit tests each; pass@k with n = 200 samples
TruthfulQA
Lin, Hilton and Evans, ACL 2022. 817 questions in 38 categories built to trigger common misconceptions. Best model then truthful on 58% vs 94% for humans; larger models were often less truthful
BIG-bench
Srivastava et al., 2022. 204 tasks from 450 authors at 132 institutions. BIG-Bench Hard (Suzgun et al., 2022) keeps 23 tasks on which earlier models fell below the average human rater
GPQA
Rein et al., 2023. 448 graduate-level multiple-choice questions in biology, physics and chemistry. Experts 65%, skilled non-experts 34% even with unrestricted web access for 30+ minutes. Diamond subset: 198 questions
SWE-bench
Jimenez et al., ICLR 2024. 2,294 GitHub issues from 12 Python repositories; graded by the repository's own tests. SWE-bench Verified (OpenAI, 2024) is a human-validated subset of 500
MT-Bench
Zheng et al., NeurIPS 2023. 80 two-turn questions in 8 categories (writing, roleplay, extraction, reasoning, math, coding, STEM, humanities), graded 1 to 10 by GPT-4

LLM-as-a-judge and Chatbot Arena

Open-ended answers have no key. Zheng et al. ("Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena", NeurIPS 2023) asked GPT-4 to grade answers, either one at a time on a 1–10 scale or as a pairwise comparison. GPT-4's verdicts agreed with human experts over 80% of the time, about as often as two humans agreed with each other. The same paper catalogued the judge's biases. Position bias: the judge prefers whichever answer comes first (or second), so verdicts can flip when the order is swapped. Verbosity bias: longer answers win even when the extra length adds nothing. Self-enhancement: a judge may favour answers written by its own model family. Judges are also weak at grading maths and reasoning they cannot solve themselves. The standard fixes are to run both orders and call it a tie when they disagree, to give the judge a reference answer, and to control for length. Length-controlled AlpacaEval (Dubois et al., 2024) regresses out the length difference before computing win rates.

Chatbot Arena (Chiang et al., ICML 2024) replaces the judge with crowds. A user types a prompt, two anonymous models answer, and the user votes for one, a tie, or both bad. The model names are revealed only after the vote. The pairwise outcomes are fitted with the Bradley–Terry model, the same model that underlies the reward model in RLHF:

P(i≻j)=11+e−(βi−βj),Ri=1000+400ln⁡10 βi,P(i≻j)=11+10(Rj−Ri)/400P(i\succ j)=\frac{1}{1+e^{-(\beta_i-\beta_j)}},\qquad R_i=1000+\frac{400}{\ln 10}\,\beta_i,\qquad P(i\succ j)=\frac{1}{1+10^{(R_j-R_i)/400}}
βi\beta_i
latent strength of model i, fitted by maximum likelihood (logistic regression on battle outcomes)
RiR_i
the same strength on the Elo scale; 400/ln 10 ≈ 173.7 points per unit of log-odds
Ri−Rj=100R_i-R_j=100
means i wins 64.0% of battles; 200 means 76.0%; 400 means 90.9%

The leaderboard first used online Elo updates, Ri←Ri+K(Si−Ei)R_i\leftarrow R_i+K(S_i-E_i). For example, with K=32K=32 a 1000-rated model that beats a 1100-rated one had expected score 0.3600.360 and gains 32⋅0.640=20.532\cdot0.640=20.5 points. Online Elo depends on the order of battles and assumes players' strengths drift, which static model weights do not. Bradley–Terry maximum likelihood uses all battles at once and does not depend on their order. The Arena reports bootstrap confidence intervals for each rating. Models whose intervals overlap are effectively tied. The Arena later added "style control", which adds answer length and markdown use as covariates so that ratings are not driven by formatting.

Few-shot, zero-shot and prompt sensitivity

Base models are usually evaluated few-shot: kk solved examples in the prompt show the format and the answer space. MMLU is typically 5-shot and GSM8K 8-shot with chain-of-thought. Instruction-tuned and chat models are often evaluated zero-shot through their chat template, because few-shot examples formatted as plain text can confuse them. Neither is "the" score. Reports should state the shot count, the template and whether chain-of-thought was used.

Scores are sensitive to choices that should not matter. Zhao et al. ("Calibrate Before Use", ICML 2021) found that the prompt format, the choice of few-shot examples and even their order could move GPT-3 accuracy from near chance to near state of the art. They traced this to majority-label, recency and common-token biases. Lu et al. (ACL 2022) found the same for example order. Sclar et al. (2023) varied only formatting (separators, casing, spacing) and saw accuracy spreads of up to 76 points for LLaMA-2-13B on some tasks. Multiple-choice answers also depend on which letter holds the right option. Robust evaluations average over several prompts or permute the options, and report the spread as well as the mean.

Contamination

A benchmark measures generalisation only if its test items were not in training. Web-scale corpora make that hard to guarantee: benchmarks are posted on GitHub, discussed on forums and copied into other datasets. Brown et al. checked GPT-3 with 13-gram overlap between benchmarks and training data, and reported results on "clean" subsets. Overlap checks miss paraphrases and translations. Other signals include unusually high likelihood of the exact test text (Min-K% Prob, Shi et al., 2023), accuracy that drops on freshly written look-alike problems (GSM1k, Zhang et al., 2024), and accuracy that falls for problems published after the training cutoff (LiveCodeBench, Jain et al., 2024). BIG-bench embeds a canary string in its files so that training pipelines can filter them out. Fine-tuning on benchmark-like data is a related, softer problem: the model gets better at the format, not at the skill.

How big is a benchmark difference?

An accuracy is a mean of NN Bernoulli outcomes. Its standard error is p(1−p)/N\sqrt{p(1-p)/N}. At 70% on MMLU's 14,042 questions that is 0.39 points. At 50% on GPQA Diamond's 198 questions it is 3.55 points, so a 95% interval is about ±7\pm7 points. HumanEval's 164 problems give about ±3.9\pm3.9 points of standard error at 50%. Miller ("Adding Error Bars to Evals", 2024) argues that every eval should report these, and should use clustered standard errors when questions come in groups, such as several questions about one passage.

Δ^=dˉ=1N∑i=1N(Bi−Ai),SEpaired=sdN,sd2=NN−1(n10+n01N−Δ^2)\hat\Delta=\bar d=\frac1N\sum_{i=1}^N (B_i-A_i),\qquad \mathrm{SE}_{\text{paired}}=\frac{s_d}{\sqrt N},\qquad s_d^2=\frac{N}{N-1}\Big(\frac{n_{10}+n_{01}}{N}-\hat\Delta^2\Big)
Ai,Bi∈{0,1}A_i,B_i\in\{0,1\}
whether each model got question i right
n10, n01n_{10},\,n_{01}
discordant questions: only A right, only B right. Questions both get right or both get wrong carry no information about the difference
SEpaired\mathrm{SE}_{\text{paired}}
smaller than the unpaired SEA2+SEB2\sqrt{\mathrm{SE}_A^2+\mathrm{SE}_B^2} whenever the two models find the same questions hard

Two models evaluated on the same questions are paired. Hard questions are hard for both, so most of the per-question variance cancels in the difference. Treating the two accuracies as independent overstates the uncertainty, and the correct test (McNemar's test, or a paired bootstrap that resamples questions) is often much sharper. The same logic gives the smallest difference worth trying to detect. With 80% power at α=0.05\alpha=0.05 that is about 2.8⋅SEpaired2.8\cdot\mathrm{SE}_{\text{paired}}. If that exceeds the gap you care about, you need more questions, not more decimal places.

164 problems · n = 10 samples each · pass@k = P(at least 1 of k samples passes)0.000.250.500.751.00125102050100200k (log scale)k > ntrue pass@1056.3%unbiased, this run56.1%naive 1−(1−c/n)^k, this run50.4%E[naive] over runs50.9%bias of naive−5.5 ppproblems with c > 092 / 164ceiling (true pass@∞)79.3%— true · — unbiased 1 − C(n−c,k)/C(n,k) · — naive, this run · - - naive, expectedunbiased: E[1 − C(n−c,k)/C(n,k)] = 1 − (1−p)^k exactly, for every k ≤ nnaive: 1 − (1−p)^k is concave in p, so plugging in p̂ = c/n underestimates (Jensen)k = n: unbiased = fraction of problems with any correct sample
panel
fig.Left toggle: a HumanEval-sized suite of 164 problems with fixed true per-sample pass rates (34 of them unsolvable, so pass@k never exceeds 79.3%). Each problem draws n samples from a fixed seeded stream. At the default n = 10, k = 10, the true pass@10 is 56.3% and the unbiased estimator reads 56.1%. That is simply the 92 of 164 problems with at least one correct sample. The plug-in 1 − (1 − c/n)^k gives 50.4% on this run, and its exact expectation over runs is 50.9%, a bias of −5.5 points. At k = 1 the two estimators coincide (22.2% on this run, true 22.5%). At n = 5, k = 5 the expected plug-in is 41.1% against a true 47.4% (bias −6.4 points). At Chen et al.'s n = 200 the bias shrinks: at k = 100 it is −1.3 points (74.4% vs 75.8%). Right toggle: two models answer the same N questions, A with true accuracy 70% and B with 70% plus the gap. ρ is the chance that B faces the same draw of difficulty as A. At the default N = 200, gap 3 pp and ρ = 0.5, the observed accuracies are 70.5% and 74.5% (Δ = +4.0 pp, 17 vs 25 discordant questions). The unpaired SE is 4.46 pp and the paired SE 3.24 pp. The paired 95% CI is [−2.3, +10.3] and the 500-resample bootstrap gives [−2.0, +10.0], so p = 0.216 and the gap is not resolved. At N = 1,000 the bootstrap interval is [+1.8, +7.4] (p = 0.0019). At MMLU's N = 14,042 the paired SE is 0.39 pp. At N = 50 the observed gap is +16.0 pp although the true one is 3. Setting ρ = 0.9 at N = 200 cuts the paired SE to 1.71 pp while the unpaired SE stays 4.46. With ρ = 0, the two SEs agree (4.54 vs 4.51).

Worked example: is model B really better?

Two models answer the same 500 multiple-choice questions. A gets 70.0% and B gets 74.0%. Is the 4-point gap real? The per-question outcomes form a 2×2 table:

A vs BB rightB wrongtotal
A right32030350 (70.0%)
A wrong50100150
total370 (74.0%)130500
Paired outcomes on N = 500 questions (counts made up; statistics computed in Python)

Three ways to put an error bar on Δ = +4.0 points

  1. 1Unpaired (wrong here). SE=0.70⋅0.30/500+0.74⋅0.26/500=2.84\mathrm{SE}=\sqrt{0.70\cdot0.30/500+0.74\cdot0.26/500}=2.84 pp. The 95% CI is 4.0±1.96⋅2.84=[−1.6,+9.6]4.0\pm1.96\cdot2.84=[-1.6,+9.6] pp, with z=1.41z=1.41 and p=0.159p=0.159: "not significant".
  2. 2Paired. Only the 30+50=8030+50=80 discordant questions vary. sd2=500499(80/500−0.042)=0.1587s_d^2=\tfrac{500}{499}(80/500-0.04^2)=0.1587, so SE=0.1587/500=1.78\mathrm{SE}=\sqrt{0.1587/500}=1.78 pp. The CI is [+0.5,+7.5][+0.5,+7.5] pp, with z=2.25z=2.25 and p=0.025p=0.025.
  3. 3McNemar's test uses the discordant counts alone: χ2=(50−30)2/(50+30)=5.0\chi^2=(50-30)^2/(50+30)=5.0 with 1 degree of freedom, so p=0.025p=0.025. The exact binomial version (is 30 out of 80 consistent with 12\tfrac12?) gives p=0.033p=0.033.
  4. 4Paired bootstrap with 10,000 resamples of the 500 questions (the code below): 95% CI [+0.6,+7.6][+0.6,+7.6] pp, in line with the CLT interval.
  5. 5Conclusion: B is better at the 5% level, but the interval is wide. The true gap could plausibly be anywhere from about half a point to about seven and a half points. A claim of "4.0 points better" should come with that interval.
python
1import numpy as np
2import torch
3from transformers import AutoModelForCausalLM, AutoTokenizer
4
5# ---- 1. multiple choice by log-likelihood (the acc / acc_norm protocol) ----
6tok = AutoTokenizer.from_pretrained("gpt2")
7model = AutoModelForCausalLM.from_pretrained("gpt2").eval()
8
9@torch.no_grad()
10def option_logprob(context: str, option: str) -> float:
11 """Sum of log p(option tokens | context). Context and option are tokenized
12 separately and concatenated, so the option's tokens are exactly its own."""
13 ctx = tok(context).input_ids
14 opt = tok(option, add_special_tokens=False).input_ids
15 ids = torch.tensor([ctx + opt])
16 logp = model(ids).logits[0, :-1].log_softmax(-1) # row t predicts token t+1
17 targets = ids[0, 1:]
18 lp = logp.gather(-1, targets[:, None])[:, 0]
19 return lp[-len(opt):].sum().item() # keep only the option's tokens
20
21def score_mc(context: str, options: list[str], gold: int) -> tuple[bool, bool]:
22 s = np.array([option_logprob(context, " " + o) for o in options])
23 chars = np.array([len(o) for o in options], dtype=float)
24 return int(s.argmax()) == gold, int((s / chars).argmax()) == gold # acc, acc_norm
25
26ctx = "A man is sitting on a roof. He"
27opts = ["is using wrap to wrap a pair of skis.", "is ripping level tiles off.",
28 "is holding a rubik's cube.", "starts pulling up roofing on a roof."]
29print(score_mc(ctx, opts, gold=3))
30
31# ---- 2. pass@k: the unbiased estimator of Chen et al. (2021) ----
32def pass_at_k(n: int, c: int, k: int) -> float:
33 """1 - C(n-c, k) / C(n, k), computed as a stable product."""
34 if n - c < k:
35 return 1.0
36 return 1.0 - np.prod(1.0 - k / np.arange(n - c + 1, n + 1))
37
38print(f"{pass_at_k(10, 3, 1):.4f} {pass_at_k(10, 3, 5):.4f} {1 - 0.7 ** 5:.4f}") # 0.3000 0.9167 0.8319 (plug-in)
39
40# ---- 3. paired bootstrap CI for acc(B) - acc(A) on the same questions ----
41def paired_bootstrap(a: np.ndarray, b: np.ndarray, reps: int = 10_000, seed: int = 0):
42 d = b.astype(float) - a.astype(float) # per-question difference in {-1, 0, 1}
43 se = d.std(ddof=1) / np.sqrt(len(d)) # paired CLT standard error
44 rng = np.random.default_rng(seed)
45 idx = rng.integers(0, len(d), size=(reps, len(d))) # resample questions, not answers
46 boot = d[idx].mean(axis=1)
47 return d.mean(), se, np.percentile(boot, [2.5, 97.5])
48
49# the worked example: 320 both right, 30 only A, 50 only B, 100 both wrong
50a = np.array([1] * 320 + [1] * 30 + [0] * 50 + [0] * 100)
51b = np.array([1] * 320 + [0] * 30 + [1] * 50 + [0] * 100)
52delta, se, ci = paired_bootstrap(a, b)
53print(f"delta={delta:+.3f} paired SE={se:.4f} bootstrap 95% CI={ci.round(3)}")
54# delta=+0.040 paired SE=0.0178 bootstrap 95% CI=[0.006 0.076]
55
56# ---- 4. Bradley-Terry ratings from pairwise battles (MM algorithm, Hunter 2004) ----
57def bradley_terry(wins: np.ndarray, iters: int = 1000) -> np.ndarray:
58 """wins[i, j] = battles i won against j (count a tie as half a win for each side)."""
59 games = wins + wins.T
60 p = np.ones(len(wins))
61 for _ in range(iters):
62 p = wins.sum(axis=1) / (games / (p[:, None] + p[None, :])).sum(axis=1)
63 p /= np.exp(np.log(p).mean()) # fix the scale: geometric mean 1
64 return 1000 + 400 * np.log10(p) # the Elo-like scale the Arena reports
65
66wins = np.array([[0, 60, 75], # model 0 beat model 1 60 times, model 2 75 times
67 [40, 0, 55],
68 [25, 45, 0]], dtype=float)
69print(bradley_terry(wins).round(1)) # [1085.3 987.8 926.9]
Likelihood scoring of multiple choice with Hugging Face transformers (not run here), plus pass@k, a paired bootstrap and a Bradley–Terry fit in numpy (run; outputs shown)

Use when

  • +Log-likelihood multiple choice is deterministic, cheap and needs no answer parsing; it gives signal even for small base models
  • +Generative exact match and unit tests check what users care about, the final answer or a working program
  • +Unbiased pass@k from n ≥ k samples has far lower variance than drawing exactly k samples
  • +LLM judges and Arena votes cover open-ended quality that has no answer key
  • +Bradley–Terry ratings put many models on one scale from sparse pairwise comparisons

Avoid when

  • −Scores depend on prompt, shot count and scoring details; harnesses disagree on the same model
  • −Static test sets leak into web-scale training data and saturate within a few years
  • −Small benchmarks (164 to a few hundred items) have standard errors of 2 to 4 points
  • −LLM judges show position, verbosity and self-enhancement biases
  • −Arena ratings reflect the prompts and tastes of the people who vote, and reward style as well as substance
  • −Perplexity is not comparable across tokenizers and tracks downstream ability only loosely

Where it's used

Pretraining ablations

Validation loss and bits per byte, plus log-likelihood multiple choice (HellaSwag, ARC), give smooth signals on small models where generative scores are still near zero.

Post-training regression tests

A fixed suite of generative tasks (GSM8K, HumanEval, instruction following), run on every checkpoint with paired comparisons against the previous one.

Model selection

A private set drawn from your own traffic, graded by tests, rubrics or an LLM judge with both answer orders, is what predicts production quality.

Public leaderboards

Chatbot Arena and benchmark tables compare released models; read them with their confidence intervals and protocols.

Check yourself

A problem gets n = 10 samples and exactly c = 1 passes. What is the unbiased pass@5 estimate?

Check yourself

On GSM8K's 1,319 test problems, model A scores 90.0% and model B scores 91.5% (B gets 20 more right). When is B's lead statistically significant at the 5% level?

Remember

  • ✓Perplexity exp⁡(mean NLL)\exp(\text{mean NLL}) is per token, so it compares only models with one tokenizer; bits per byte fixes that, but neither predicts downstream ability reliably across model families.
  • ✓Multiple choice by log-likelihood picks arg⁡max⁡j∑tlog⁡p(aj,t∣c,aj,<t)\arg\max_j\sum_t\log p(a_{j,t}\mid c,a_{j,<t}); length normalisation (acc_norm) removes the bias toward short options.
  • ✓pass@k =E[1−(n−ck)/(nk)]=\mathbb E[1-\binom{n-c}{k}/\binom nk] from n≥kn\ge k samples is unbiased; the plug-in 1−(1−c/n)k1-(1-c/n)^k is biased low by Jensen's inequality.
  • ✓LLM judges agree with humans about as often as humans agree with each other on MT-Bench, but they have position, verbosity and self-enhancement biases. Swap the answer order and control for length.
  • ✓Chatbot Arena fits Bradley–Terry, P(i≻j)=1/(1+10(Rj−Ri)/400)P(i\succ j)=1/(1+10^{(R_j-R_i)/400}), to crowd votes and reports bootstrap intervals; 100 rating points means a 64% win rate.
  • ✓Contamination, prompt format, shot count and answer extraction can each move scores by more than the gaps being reported.
  • ✓Report NN and a paired confidence interval: the SE of an accuracy is p(1−p)/N\sqrt{p(1-p)/N} (3.55 points on 198 questions at 50%), and paired tests on discordant questions are much sharper than unpaired ones.
← back to Deep Learning roadmap