Emergent abilities

An ability is emergent if it is absent in smaller models and present in larger ones (Wei et al., 2022). Many such jumps come from all-or-nothing metrics: exact match on L tokens is pᴸ, which looks sudden even when per-token loss falls smoothly.

8 min read863 wordsadvanced0 viewsupdated 2026-10-21

Definition

Emergent ability "An ability is emergent if it is not present in smaller models but is present in larger models" (Wei et al., "Emergent Abilities of Large Language Models", TMLR 2022). Such an ability cannot be predicted by extrapolating the performance of smaller models: the score sits near chance across several sizes, then rises sharply. The word comes from Anderson's essay "More Is Different" (1972), where quantitative change produces qualitative change.

The question matters to anyone who picks, trains or regulates models. If abilities appear suddenly, you cannot test a small model and forecast what a large one will do. A capability that is harmless at 10B parameters may appear at 100B with no warning. Ganguli et al. ("Predictability and Surprise in Large Generative Models", 2022) put it as two facts that coexist: overall loss is predictable, specific capabilities are not.

Wei et al. collected examples from GPT-3, LaMDA, Gopher, Chinchilla and PaLM. Few-shot tasks such as multi-digit arithmetic, transliteration into the International Phonetic Alphabet, word unscrambling, MMLU and TruthfulQA stayed near chance until a model-size threshold, then improved. They also listed prompting methods that help only above a certain scale. Chain-of-thought prompting is the best known: Wei et al.'s chain-of-thought paper (2022) found it helped only at around 100B parameters; smaller models wrote fluent but illogical chains and gained nothing or lost accuracy.

BIG-bench (Srivastava et al., 2022; 204 tasks) called this "breakthrough" behaviour. It noted that breakthrough tasks often involve several steps or components, or use brittle metrics. The next section shows why those two features produce jumps.

Where a jump can come from

Suppose the probability of each correct answer token improves smoothly with scale. Score a task by exact match on an answer of LL tokens, and treat the tokens as independent for simplicity:

p(N)=e−ℓ(N),EM(N)=p(N)L=e−L ℓ(N),E[token edit distance]=L (1−p(N))p(N)=e^{-\ell(N)},\qquad \text{EM}(N)=p(N)^{L}=e^{-L\,\ell(N)},\qquad \mathbb E[\text{token edit distance}]=L\,\big(1-p(N)\big)
ℓ(N)\ell(N)
per-token cross-entropy on the answer; a power law in model size NN
p(N)p(N)
probability that a given answer token is correct
LL
answer length in tokens (or number of reasoning steps that must all be right)
EM\text{EM}
exact match: 1 only if every token is right; a nonlinear metric

The edit distance is linear in pp and changes as smoothly as pp does. Exact match raises pp to the power LL. It stays near zero while pp is moderate and climbs fast once pp approaches 1. Multiple-choice grading behaves the same way: it scores 1 only when the right option's probability overtakes all the others, a threshold on a smooth quantity.

This is the argument of Schaeffer, Miranda and Koyejo, "Are Emergent Abilities of Large Language Models a Mirage?" (NeurIPS 2023). On GPT-3/InstructGPT arithmetic, the jumps seen under exact match became smooth, predictable curves under token edit distance. In their survey of BIG-bench, the large majority of emergent abilities (over 92% in their count) appeared under just two metrics, multiple-choice grade and exact string match. They also produced "emergence" in vision models on purpose, just by choosing a sharp metric.

same 7 models, same outputs, four ways to score themper-token loss ℓ2·(N/10⁸)^−0.510⁸10¹¹ Nlog axis: a linetoken right pp = e^−ℓ10⁸10¹¹ Nsmooth S-curveexact matchall 5 right: p⁵10⁸10¹¹ Nflat, then a jumpitems correctround(100·p⁵)10⁸10¹¹ N0, 0, 4, … 73Nℓpp⁵of 10010^82.0000.135<0.001010^90.6320.5310.042410^100.2000.8190.3683710^110.0630.9390.72973nothing jumps inside themodel: ℓ improves by the samefactor √10 per half-decade.The jump is made by the metric.Wei et al. (2022): emergent = absent in smaller models, present in larger ones,so not predictable by extrapolating the smaller models' scores.Schaeffer et al. (2023): many jumps vanish under continuous metrics (edit distance, Brier).
fig.Seven toy models from 10⁸ to 10¹¹ parameters with per-token loss ℓ = 2(N/10⁸)^−0.5, a straight line on log axes. The probability of a correct token rises smoothly, 0.135 → 0.531 → 0.819 → 0.939 at 10⁸, 10⁹, 10¹⁰, 10¹¹. Exact match on a 5-token answer, p⁵, is < 0.001, 0.042, 0.368, 0.729. Counted on 100 items it reads 0, 0, 4, 17, 37, 57, 73: flat, then a jump, though nothing changed abruptly inside the models.

Small test sets hide the early rise

Pr⁡[observed score=0]=(1−s)n,smallest non-zero score=1/n\Pr[\text{observed score}=0]=(1-s)^{n},\qquad \text{smallest non-zero score}=1/n
ss
the model's true per-item success rate on the task
nn
number of test items

A model with true exact-match rate s=0.0036s=0.0036 (the 108.510^{8.5} model above) scores exactly 0 on a 200-item test with probability (1−0.0036)200=0.486(1-0.0036)^{200}=0.486. With 2,000 items that falls to 0.0007. Many BIG-bench tasks have only a few hundred items, so "absent in smaller models" can mean "below the test's resolution".

The main sources

Definition
Wei, Tay, Bommasani, Raffel, Zoph et al., "Emergent Abilities of Large Language Models", TMLR 2022; examples plotted against training FLOPs and parameter count
Scale-gated prompting
Wei et al., "Chain-of-Thought Prompting Elicits Reasoning in Large Language Models", NeurIPS 2022: gains appear only at around 100B parameters
Benchmark
BIG-bench (Srivastava et al., 2022): 204 tasks; describes "breakthrough" tasks, often multi-step or with brittle metrics
Metric critique
Schaeffer, Miranda & Koyejo, NeurIPS 2023: nonlinear or discontinuous metrics plus small test sets can produce apparent emergence
Policy view
Ganguli et al., FAccT 2022: smooth loss scaling alongside abrupt, hard-to-predict specific capabilities
The reverse
McKenzie et al., "Inverse Scaling: When Bigger Isn't Better" (2023): tasks where larger models do worse
13 models · answer = 5 tokens · n = 200 test items · equal difficultyper-token loss ℓ = 2(N/10⁸)^−0.5 falls smoothly; p(token right) = e^−ℓ; exact match needs all 5fit on these 6025507510010^810^910^1010^11parameters N (log scale) · score in %largest model, 10^11.5 exact match88.5% token accuracy97.5%mid model, 10^9.5 exact match18.0% token accuracy72.5%first EM above 010^8.5forecast 10^11.5 EMfrom 6 smallest: line on EM13.3% via token loss85.0% actual88.5%exact match (all tokens right)token accuracyline fit to EM · ◇ forecast via lossdots: sampled scores on the same 200 items (resolution 0.5%) · thin lines: expected scores
item difficulty
fig.13 synthetic models with smoothly falling per-token loss, each scored on the same n items. Defaults (L = 5 answer tokens, n = 200, equal difficulty): the six smallest models score 0–7.5% exact match (first non-zero at 10^8.5), the 10^9.5 model scores 18.0% exact match against 72.5% token accuracy, and the largest reaches 88.5%. A straight line through the six smallest exact-match scores forecasts 13.3% for the largest; fitting the per-token loss on log–log axes and converting forecasts 85.0%. At L = 10 the first six score exactly 0 (line forecast 0.0%; loss forecast 73.1% vs actual 72.5%); at L = 20, 47.5% vs 47.0%.

The lesson for practice is in the two forecasts. A sharp metric throws away the information that small models are improving. A continuous metric keeps it, and a scaling-law fit on it predicts the large model well. With mixed item difficulty the exact-match curve is less sharp and the loss-based forecast is only approximate (71.8% vs 83.0% at the defaults), because averaging over items breaks the simple pLp^L relation.

Worked example: one curve, three metrics

Nℓp = e^−ℓEM, L = 5EM, L = 10edit distance, L = 5
10⁸2.0000.135< 0.001< 0.0014.32
10⁹0.6320.5310.0420.0022.34
10¹⁰0.2000.8190.3680.1350.91
10¹¹0.0630.9390.7290.5310.31
ℓ = 2(N/10⁸)^−0.5. Exact match for 5- and 10-token answers, expected token edit distance for 5 tokens.

Each decade of NN divides the loss by 10≈3.16\sqrt{10}\approx3.16. The edit distance falls by a factor of 1.8–2.9 per decade throughout. Exact match for L=10L=10 goes from 0.002 to 0.135 to 0.531: about a 76-fold rise across one decade, then 3.9-fold across the next. The same compounding applies to multi-step tasks. A model that gets each of 20 steps right with probability 0.95 completes the whole task 0.9520=35.8%0.95^{20}=35.8\% of the time; at 0.99 per step it completes it 81.8% of the time.

python
1import numpy as np
2
3def answer_scores(gold_logprobs):
4 """Teacher-forced log p(gold token) at each answer position -> three metrics."""
5 lp = np.asarray(gold_logprobs, dtype=float)
6 return {
7 "exact_match_prob": float(np.exp(lp.sum())), # p(all tokens right) = prod p_t
8 "mean_token_prob": float(np.exp(lp).mean()), # smooth, per-token
9 "answer_logprob": float(lp.sum()), # continuous, never stuck at 0
10 }
11
12# three small models scored on the same 5-token answer (per-token loss 2.0, 0.632, 0.2)
13sizes = np.array([1e8, 1e9, 1e10])
14losses = np.array([2.0, 0.632, 0.2])
15for n, l in zip(sizes, losses):
16 print(f"{n:.0e}", {k: round(v, 4) for k, v in answer_scores([-l] * 5).items()})
17# 1e+08 {'exact_match_prob': 0.0, 'mean_token_prob': 0.1353, 'answer_logprob': -10.0}
18# 1e+09 {'exact_match_prob': 0.0424, 'mean_token_prob': 0.5315, 'answer_logprob': -3.16}
19# 1e+10 {'exact_match_prob': 0.3679, 'mean_token_prob': 0.8187, 'answer_logprob': -1.0}
20
21# forecast a 1e11 model: fit the per-token loss on log-log axes, then convert
22b, a = np.polyfit(np.log10(sizes), np.log(losses), 1)
23l_hat = np.exp(a + b * 11)
24print(round(b / np.log(10), 3), round(l_hat, 4), round(np.exp(-5 * l_hat), 3))
25# -0.5 0.0632 0.729 (exponent, predicted loss, predicted exact match)
Score answers with continuous metrics and forecast a larger model from small ones (numpy; runs as is). The log-probs would come from teacher-forced scoring of the gold answer.

Where it's used

Model selection

Compare candidate models on per-token log-likelihood or partial credit as well as pass/fail, to see how far a smaller, cheaper model is from the threshold (see Model families and model selection).

Evaluation design

Use enough items that 1/n1/n is well below the scores you care about, and report continuous metrics next to exact match (see LLM evaluation and benchmarking).

Training forecasts

Fit scaling curves on small runs using loss or log-likelihood of reference answers, then convert to the task metric, as in the figure.

Risk assessment

Test dangerous capabilities on every new model with strong elicitation (good prompts, chain of thought, tools), since they may appear between sizes.

Check yourself

Per-token accuracy on a 10-token answer improves from 0.5 to 0.9 across model sizes. What happens to exact match?

Check yourself

Why does a straight line through small models' exact-match scores forecast the large model badly in the figure?

Remember

  • ✓Wei et al. (2022): an ability is emergent if it is absent in smaller models and present in larger ones, so it cannot be extrapolated.
  • ✓Examples: multi-digit arithmetic, IPA transliteration, MMLU and gains from chain-of-thought prompting at about 100B parameters.
  • ✓Schaeffer et al. (2023): all-or-nothing metrics (exact match, multiple-choice grade) turn smooth gains into jumps; EM = pᴸ.
  • ✓Small test sets add a floor: a 200-item test cannot show scores below 0.5%.
  • ✓Forecast with continuous metrics (log-likelihood, edit distance), decide with the task metric, and test each new model directly.
← back to Generative AI roadmap