# Emergent abilities

> An ability is emergent if it is absent in smaller models and present in larger ones (Wei et al., 2022). Many such jumps come from all-or-nothing metrics: exact match on L tokens is pᴸ, which looks sudden even when per-token loss falls smoothly.

Source: https://www.mlfoundry.in/learn/emergent-abilities  
Section: Generative AI › Foundations of Generative AI  
Level: advanced · 8 min read · updated 2026-10-21

**Emergent ability:** "An ability is emergent if it is not present in smaller models but is present in larger models" (Wei et al., "Emergent Abilities of Large Language Models", TMLR 2022). Such an ability cannot be predicted by extrapolating the performance of smaller models: the score sits near chance across several sizes, then rises sharply. The word comes from Anderson's essay "More Is Different" (1972), where quantitative change produces qualitative change.

The question matters to anyone who picks, trains or regulates models. If abilities appear suddenly, you cannot test a small model and forecast what a large one will do. A capability that is harmless at 10B parameters may appear at 100B with no warning. Ganguli et al. ("Predictability and Surprise in Large Generative Models", 2022) put it as two facts that coexist: overall loss is predictable, specific capabilities are not.

Wei et al. collected examples from GPT-3, LaMDA, Gopher, Chinchilla and PaLM. Few-shot tasks such as multi-digit arithmetic, transliteration into the International Phonetic Alphabet, word unscrambling, MMLU and TruthfulQA stayed near chance until a model-size threshold, then improved. They also listed prompting methods that help only above a certain scale. **Chain-of-thought prompting** is the best known: Wei et al.'s chain-of-thought paper (2022) found it helped only at around 100B parameters; smaller models wrote fluent but illogical chains and gained nothing or lost accuracy.

BIG-bench (Srivastava et al., 2022; 204 tasks) called this "breakthrough" behaviour. It noted that breakthrough tasks often involve several steps or components, or use brittle metrics. The next section shows why those two features produce jumps.

## Where a jump can come from

Suppose the probability of each correct answer token improves smoothly with scale. Score a task by exact match on an answer of $L$ tokens, and treat the tokens as independent for simplicity:

$$
p(N)=e^{-\ell(N)},\qquad \text{EM}(N)=p(N)^{L}=e^{-L\,\ell(N)},\qquad \mathbb E[\text{token edit distance}]=L\,\big(1-p(N)\big)
$$

- $\ell(N)$: per-token cross-entropy on the answer; a power law in model size $N$
- $p(N)$: probability that a given answer token is correct
- $L$: answer length in tokens (or number of reasoning steps that must all be right)
- $\text{EM}$: exact match: 1 only if every token is right; a **nonlinear** metric

The edit distance is linear in $p$ and changes as smoothly as $p$ does. Exact match raises $p$ to the power $L$. It stays near zero while $p$ is moderate and climbs fast once $p$ approaches 1. Multiple-choice grading behaves the same way: it scores 1 only when the right option's probability overtakes all the others, a threshold on a smooth quantity.

This is the argument of Schaeffer, Miranda and Koyejo, "Are Emergent Abilities of Large Language Models a Mirage?" (NeurIPS 2023). On GPT-3/InstructGPT arithmetic, the jumps seen under exact match became smooth, predictable curves under token edit distance. In their survey of BIG-bench, the large majority of emergent abilities (over 92% in their count) appeared under just two metrics, multiple-choice grade and exact string match. They also produced "emergence" in vision models on purpose, just by choosing a sharp metric.

> Figure: Seven toy models from 10⁸ to 10¹¹ parameters with per-token loss ℓ = 2(N/10⁸)^−0.5, a straight line on log axes. The probability of a correct token rises smoothly, 0.135 → 0.531 → 0.819 → 0.939 at 10⁸, 10⁹, 10¹⁰, 10¹¹. Exact match on a 5-token answer, p⁵, is < 0.001, 0.042, 0.368, 0.729. Counted on 100 items it reads 0, 0, 4, 17, 37, 57, 73: flat, then a jump, though nothing changed abruptly inside the models.

## Small test sets hide the early rise

$$
\Pr[\text{observed score}=0]=(1-s)^{n},\qquad \text{smallest non-zero score}=1/n
$$

- $s$: the model's true per-item success rate on the task
- $n$: number of test items

A model with true exact-match rate $s=0.0036$ (the $10^{8.5}$ model above) scores exactly 0 on a 200-item test with probability $(1-0.0036)^{200}=0.486$. With 2,000 items that falls to 0.0007. Many BIG-bench tasks have only a few hundred items, so "absent in smaller models" can mean "below the test's resolution".

**The main sources**

- **Definition:** Wei, Tay, Bommasani, Raffel, Zoph et al., "Emergent Abilities of Large Language Models", TMLR 2022; examples plotted against training FLOPs and parameter count
- **Scale-gated prompting:** Wei et al., "Chain-of-Thought Prompting Elicits Reasoning in Large Language Models", NeurIPS 2022: gains appear only at around 100B parameters
- **Benchmark:** BIG-bench (Srivastava et al., 2022): 204 tasks; describes "breakthrough" tasks, often multi-step or with brittle metrics
- **Metric critique:** Schaeffer, Miranda & Koyejo, NeurIPS 2023: nonlinear or discontinuous metrics plus small test sets can produce apparent emergence
- **Policy view:** Ganguli et al., FAccT 2022: smooth loss scaling alongside abrupt, hard-to-predict specific capabilities
- **The reverse:** McKenzie et al., "Inverse Scaling: When Bigger Isn't Better" (2023): tasks where larger models do worse

> Figure: 13 synthetic models with smoothly falling per-token loss, each scored on the same n items. Defaults (L = 5 answer tokens, n = 200, equal difficulty): the six smallest models score 0–7.5% exact match (first non-zero at 10^8.5), the 10^9.5 model scores 18.0% exact match against 72.5% token accuracy, and the largest reaches 88.5%. A straight line through the six smallest exact-match scores forecasts 13.3% for the largest; fitting the per-token loss on log–log axes and converting forecasts 85.0%. At L = 10 the first six score exactly 0 (line forecast 0.0%; loss forecast 73.1% vs actual 72.5%); at L = 20, 47.5% vs 47.0%.

The lesson for practice is in the two forecasts. A sharp metric throws away the information that small models are improving. A continuous metric keeps it, and a scaling-law fit on it predicts the large model well. With mixed item difficulty the exact-match curve is less sharp and the loss-based forecast is only approximate (71.8% vs 83.0% at the defaults), because averaging over items breaks the simple $p^L$ relation.

## Worked example: one curve, three metrics

ℓ = 2(N/10⁸)^−0.5. Exact match for 5- and 10-token answers, expected token edit distance for 5 tokens.

| N | ℓ | p = e^−ℓ | EM, L = 5 | EM, L = 10 | edit distance, L = 5 |
| --- | --- | --- | --- | --- | --- |
| 10⁸ | 2.000 | 0.135 | < 0.001 | < 0.001 | 4.32 |
| 10⁹ | 0.632 | 0.531 | 0.042 | 0.002 | 2.34 |
| 10¹⁰ | 0.200 | 0.819 | 0.368 | 0.135 | 0.91 |
| 10¹¹ | 0.063 | 0.939 | 0.729 | 0.531 | 0.31 |

Each decade of $N$ divides the loss by $\sqrt{10}\approx3.16$. The edit distance falls by a factor of 1.8–2.9 per decade throughout. Exact match for $L=10$ goes from 0.002 to 0.135 to 0.531: about a 76-fold rise across one decade, then 3.9-fold across the next. The same compounding applies to multi-step tasks. A model that gets each of 20 steps right with probability 0.95 completes the whole task $0.95^{20}=35.8\%$ of the time; at 0.99 per step it completes it 81.8% of the time.

```python
import numpy as np

def answer_scores(gold_logprobs):
    """Teacher-forced log p(gold token) at each answer position -> three metrics."""
    lp = np.asarray(gold_logprobs, dtype=float)
    return {
        "exact_match_prob": float(np.exp(lp.sum())),   # p(all tokens right) = prod p_t
        "mean_token_prob": float(np.exp(lp).mean()),   # smooth, per-token
        "answer_logprob": float(lp.sum()),             # continuous, never stuck at 0
    }

# three small models scored on the same 5-token answer (per-token loss 2.0, 0.632, 0.2)
sizes = np.array([1e8, 1e9, 1e10])
losses = np.array([2.0, 0.632, 0.2])
for n, l in zip(sizes, losses):
    print(f"{n:.0e}", {k: round(v, 4) for k, v in answer_scores([-l] * 5).items()})
# 1e+08 {'exact_match_prob': 0.0, 'mean_token_prob': 0.1353, 'answer_logprob': -10.0}
# 1e+09 {'exact_match_prob': 0.0424, 'mean_token_prob': 0.5315, 'answer_logprob': -3.16}
# 1e+10 {'exact_match_prob': 0.3679, 'mean_token_prob': 0.8187, 'answer_logprob': -1.0}

# forecast a 1e11 model: fit the per-token loss on log-log axes, then convert
b, a = np.polyfit(np.log10(sizes), np.log(losses), 1)
l_hat = np.exp(a + b * 11)
print(round(b / np.log(10), 3), round(l_hat, 4), round(np.exp(-5 * l_hat), 3))
# -0.5 0.0632 0.729   (exponent, predicted loss, predicted exact match)
```

Score answers with continuous metrics and forecast a larger model from small ones (numpy; runs as is). The log-probs would come from teacher-forced scoring of the gold answer.

> **Mirage or not, the product sees the jump:** The metric argument explains why a curve looks sudden. It does not make the jump irrelevant. If your application needs a 20-step plan to be fully right, exact success is what you ship, and it really does go from useless to useful over a narrow range of model quality. Use continuous metrics to **forecast**; use the task metric to **decide**.

> **Not every jump is an artefact:** A smooth per-token loss is an assumption, not a law. Abilities can also depend on the prompt (an ability may show up only with a few-shot or chain-of-thought prompt) and on training data, not just size, and some curves stay sharp under continuous metrics. Evaluate each new model directly on the capabilities you care about, especially in safety testing (see **Red teaming**), instead of assuming it behaves like a bigger copy of the last one.

- **Model selection:** Compare candidate models on per-token log-likelihood or partial credit as well as pass/fail, to see how far a smaller, cheaper model is from the threshold (see **Model families and model selection**).
- **Evaluation design:** Use enough items that $1/n$ is well below the scores you care about, and report continuous metrics next to exact match (see **LLM evaluation and benchmarking**).
- **Training forecasts:** Fit scaling curves on small runs using loss or log-likelihood of reference answers, then convert to the task metric, as in the figure.
- **Risk assessment:** Test dangerous capabilities on every new model with strong elicitation (good prompts, chain of thought, tools), since they may appear between sizes.

**Question:** Per-token accuracy on a 10-token answer improves from 0.5 to 0.9 across model sizes. What happens to exact match?

- A. It improves from 0.5 to 0.9
- B. It improves from 0.001 to 0.349
- C. It stays at 0
- D. It improves from 0.05 to 0.09

**Answer:** B. It improves from 0.001 to 0.349 — Exact match is $p^{10}$: $0.5^{10}=0.00098$ and $0.9^{10}=0.349$. A 1.8× gain in per-token accuracy becomes a 357× gain in exact match, which looks like an ability switching on.

**Question:** Why does a straight line through small models' exact-match scores forecast the large model badly in the figure?

- A. The large model was trained differently
- B. Exact match is a nonlinear function of a smoothly improving quantity, so small models' scores carry almost no signal
- C. The test set is too large
- D. Token accuracy is noisy

**Answer:** B. Exact match is a nonlinear function of a smoothly improving quantity, so small models' scores carry almost no signal — The six smallest models score 0–7.5% exact match, a nearly flat line. Their per-token loss, by contrast, falls on a clean log–log line, and extrapolating that and converting gives 85.0% against an actual 88.5%.

**Key points**

- Wei et al. (2022): an ability is emergent if it is absent in smaller models and present in larger ones, so it cannot be extrapolated.
- Examples: multi-digit arithmetic, IPA transliteration, MMLU and gains from chain-of-thought prompting at about 100B parameters.
- Schaeffer et al. (2023): all-or-nothing metrics (exact match, multiple-choice grade) turn smooth gains into jumps; EM = pᴸ.
- Small test sets add a floor: a 200-item test cannot show scores below 0.5%.
- Forecast with continuous metrics (log-likelihood, edit distance), decide with the task metric, and test each new model directly.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
