# Language modeling

> A language model assigns p(x₁…x_T) = ∏ p(xₜ | x<ₜ) by the chain rule and is trained by minimising the cross-entropy −(1/T)Σ log p(xₜ | x<ₜ). Perplexity is e^loss; bits per byte rescales the loss by tokens/bytes so tokenizers compare. Causal LMs predict every next token, masked LMs only masked ones.

Source: https://www.mlfoundry.in/learn/language-modeling  
Section: Deep Learning › Large Language Models (LLMs)  
Level: core · 15 min read · updated 2026-10-13

**Language model:** is a probability distribution over sequences of tokens, $p(x_1,\dots,x_T)$. By the chain rule it factorizes, with no loss of generality, into next-token conditionals $\prod_t p(x_t\mid x_{<t})$. A model of those conditionals can score text (higher probability means more plausible) and generate it, by sampling one token at a time. The idea goes back to Claude Shannon ("A Mathematical Theory of Communication", 1948), who generated English from letter and word $n$-gram statistics. The neural version starts with Bengio, Ducharme, Vincent and Jauvin ("A Neural Probabilistic Language Model", JMLR 2003). Every model in this module, from **BERT** to **Llama**, is trained with a language-modeling objective.

For decades language models were a component, not a product. A speech recognizer or a translation system proposes candidate sentences, and the LM ranks them by how likely they are as English. "Recognize speech" should beat "wreck a nice beach". Those systems used **n-gram** models: count how often each word follows each short history in a large corpus, and smooth the counts so unseen combinations do not get probability zero. Katz back-off (1987) and Kneser–Ney smoothing (Kneser and Ney, 1995) were the standard. Chen and Goodman's 1998 comparison found a modified Kneser–Ney the best of the methods they tested.

Two limits of n-grams drove the move to neural models. The first is **sparsity**. The number of possible histories grows as $V^{n-1}$, so most 4-grams never occur even in billions of words. The second is **no generalization**. Seeing "the cat sat" teaches an n-gram nothing about "the dog sat", because *cat* and *dog* are unrelated symbols. Bengio et al. mapped each word to a learned vector and predicted the next word with a neural network over the concatenated vectors of the previous $n-1$ words. Similar words get similar vectors, so evidence is shared between them (see **Word embeddings**). Mikolov et al. ("Recurrent neural network based language model", Interspeech 2010) removed the fixed window with an RNN. Transformers (see **Decoder-only transformer**) then made it possible to condition on thousands of tokens and to train on all positions in parallel.

## The chain rule and the training objective

$$
p_\theta(x_{1:T})=\prod_{t=1}^{T}p_\theta(x_t\mid x_{<t}),\qquad p_\theta(x_t=v\mid x_{<t})=\frac{\exp\big(z_v(x_{<t})\big)}{\sum_{u=1}^{V}\exp\big(z_u(x_{<t})\big)}
$$

- $x_{<t}$: all earlier tokens; a beginning-of-sequence token stands in for the empty history at $t=1$
- $z(x_{<t})\in\mathbb R^V$: logits from the network: an n-gram table, an RNN, or the LM head of a transformer
- $V$: vocabulary size, set by the tokenizer (50,257 for GPT-2, 32,000 for Llama 2, 128,256 for Llama 3; see **Tokenization algorithms**)
- $\prod_t$: the factorization is exact for *any* distribution. Models differ only in how they approximate each conditional

The chain rule makes no independence assumption. An n-gram model adds one, the **Markov assumption** $p(x_t\mid x_{<t})\approx p(x_t\mid x_{t-n+1:t-1})$. Neural LMs keep the full history, up to the context window. Training maximises the likelihood of a corpus. Equivalently, it minimises the average negative log-probability of each actual next token:

$$
\mathcal L(\theta)=-\frac1T\sum_{t=1}^{T}\log p_\theta(x_t\mid x_{<t})\;\approx\;H(p_{\text{data}},p_\theta)=H(p_{\text{data}})+D_{\mathrm{KL}}\big(p_{\text{data}}\,\|\,p_\theta\big)
$$

- $\mathcal L$: the token-level cross-entropy, in nats with $\ln$ or bits with $\log_2$; it is exactly the softmax cross-entropy classification loss, applied at every position
- $H(p_{\text{data}})$: the entropy of the language itself, a floor no model can beat. Shannon ("Prediction and Entropy of Printed English", 1951) estimated it at roughly 0.6–1.3 bits per character for English
- $D_{\mathrm{KL}}$: the part a better model can remove; cross-entropy minus the entropy floor

> Figure: Language modeling at a glance. The chain rule turns the probability of a sentence into a product of next-token predictions, and an n-gram truncates each history while a neural LM keeps all of it. A causal LM (left) uses a lower-triangular mask, and every position predicts the token after it, so a sequence of T tokens gives T training targets. A masked LM (right) attends to both sides but is trained only on the roughly 15% of positions that were masked. The summed negative log-likelihood gives cross-entropy, its exponential is perplexity, and normalizing by UTF-8 bytes instead of tokens gives bits per byte, which compares models with different tokenizers.

## Perplexity and bits per byte

$$
\mathrm{PPL}=\exp(\mathcal L_{\text{nats}})=2^{\mathcal L_{\text{bits}}}=p(x_{1:T})^{-1/T},\qquad \mathrm{BPB}=\frac{T}{B}\cdot\frac{\mathcal L_{\text{nats}}}{\ln 2}=\frac{-\log_2 p(x_{1:T})}{B}
$$

- $\mathrm{PPL}$: the inverse geometric-mean probability per token: the model is as uncertain as a uniform choice among PPL tokens. A uniform model has $\mathrm{PPL}=V$
- $T,\ B$: number of tokens and number of UTF-8 bytes in the evaluation text
- $\mathrm{BPB}$: total bits divided by bytes. It does not depend on the tokenizer, so models with different vocabularies can be compared (used by the Pile benchmark, enwik8, and many scaling studies)

Perplexity is only comparable when the **tokenization and the test text** are the same. A model with a bigger vocabulary uses fewer, harder-to-predict tokens, so its per-token perplexity is higher even when it compresses the text better. Bits per byte (or bits per character) fixes this by charging the total code length to the bytes of the text. Per token, the conversion is $\mathrm{BPB}=\mathcal L_{\text{bits}}/(\text{bytes per token})$. With a loss of 3.0 nats per token and 4.3 bytes per token, that is $3.0/\ln2/4.3=1.007$ bits per byte.

**Numbers worth knowing**

- **Shannon 1951:** human prediction experiments put the entropy of printed English at about 0.6–1.3 bits per letter, with long contexts
- **Jozefowicz et al. 2016:** "Exploring the Limits of Language Modeling": on the One Billion Word benchmark, large LSTMs with character-CNN inputs cut single-model perplexity from 51.3 to 30.0; an ensemble reached 23.7
- **GPT-2 1.5B (2019):** zero-shot, with no training on the benchmarks: perplexity 35.76 on Penn Treebank, 18.34 on WikiText-2, 17.48 on WikiText-103; 0.93 bits per byte on enwik8 and 0.98 bits per character on text8
- **BERT (2019):** masked LM: 15% of tokens are chosen; of those, 80% become `[MASK]`, 10% a random token and 10% stay unchanged
- **T5 (2020):** span corruption: 15% of tokens are dropped in spans of mean length 3, and the decoder generates the missing spans
- **Llama 2 (2023):** causal LM, 2 trillion training tokens, 4,096-token context, 32,000-token vocabulary

## Causal versus masked objectives

A **causal LM** (autoregressive LM) predicts $x_t$ from $x_{<t}$ at every position, under a lower-triangular attention mask (see **Causal attention**). It defines a proper distribution over sequences, so it can score text and generate it. GPT, Llama, Qwen and Mistral are causal LMs. A sequence of $T$ tokens gives $T$ training targets in one forward pass.

A **masked LM** (Devlin et al., **BERT**, NAACL 2019) corrupts the input, replacing 15% of positions as described above, and predicts the original tokens there from context on **both** sides. Bidirectional context makes better features for classification and extraction. But only the masked 15% of positions give a loss, and the model does not define $p(x)$, so it cannot generate fluently. **RoBERTa** keeps the objective and changes the masks dynamically. ELECTRA (Clark et al., 2020) replaces masking with a discriminator that detects replaced tokens, so that every position contributes to the loss.

Between the two sit **prefix LMs**, which attend bidirectionally over a prefix and causally over the rest, and **span corruption** in **T5** and **BART**: an encoder reads text with spans removed, and a decoder generates them autoregressively. XLNet (Yang et al., 2019) trains a causal LM over random factorization orders, so it gets bidirectional context while keeping a proper likelihood. Today's general-purpose LLMs are almost all causal, because one objective covers both pretraining and generation (see **Decoder-only models** and **Pretraining and continued pretraining**).

$$
\mathcal L_{\text{CLM}}=-\sum_{t=1}^{T}\log p(x_t\mid x_{<t}),\qquad \mathcal L_{\text{MLM}}=-\sum_{t\in\mathcal M}\log p(x_t\mid \tilde x),\quad |\mathcal M|\approx0.15\,T
$$

- $\tilde x$: the corrupted input: masked, randomly replaced or unchanged tokens at the positions $\mathcal M$
- $\mathcal M$: the chosen positions; the loss ignores every other position

> Figure: An n-gram LM trained on six sentences (40 tokens, V = 11 words + </s>) scores a held-out sentence with the chain rule. Each bar is one token's surprisal −log₂ p, and the lower chart is the full next-token distribution at the selected position. The default is a bigram with add-k smoothing, k = 0.1. On 'the cat sat on the rug', p(rug | the) = (1 + 0.1)/(9 + 1.2) = 0.108, or 3.21 bits. The sentence costs 9.43 bits over 7 tokens: H = 1.347 bits/token, perplexity 2.544, 0.410 bits per byte, against a train perplexity of 2.649. Switch to MLE and that test sentence drops to perplexity 1.985, but 'a dog sat on the mat' becomes ∞, because 'a dog' never occurred and gets p = 0. Laplace (k = 1) gives every word at least 1/21 after 'the' and raises perplexity to 5.279 on test sentence 1 and 5.409 on train. A trigram fits training text better (train perplexity 1.632 with MLE), but with add-0.01 the odd sentence 'the mat saw a dog' reaches 32.272. Interpolating trigram, bigram and unigram (λ = 0.6, 0.3, 0.1) brings it to 19.040 while keeping test sentence 1 at 1.814. The unigram model ignores context and sits near perplexity 10 on everything.

## Worked example: a bigram by hand

Train on the six sentences in the figure. The word *the* occurs 9 times as a history, followed by *cat* 3 times, *dog* 4, *mat* 1 and *rug* 1. Score "the cat sat on the rug" with a bigram, first by maximum likelihood (MLE), then with add-one (Laplace) smoothing over $V=12$ outcomes.

Bigram probabilities for one test sentence (computed in Python)

| history → token | c(h,w) | c(h) | p MLE | −log₂ p MLE | p add-1 | −log₂ p add-1 |
| --- | --- | --- | --- | --- | --- | --- |
| <s> → the | 5 | 6 | 0.8333 | 0.263 | 0.3333 | 1.585 |
| the → cat | 3 | 9 | 0.3333 | 1.585 | 0.1905 | 2.392 |
| cat → sat | 2 | 5 | 0.4000 | 1.322 | 0.1765 | 2.503 |
| sat → on | 3 | 3 | 1.0000 | 0.000 | 0.2667 | 1.907 |
| on → the | 2 | 3 | 0.6667 | 0.585 | 0.2000 | 2.322 |
| the → rug | 1 | 9 | 0.1111 | 3.170 | 0.0952 | 3.392 |
| rug → </s> | 1 | 1 | 1.0000 | 0.000 | 0.1538 | 2.700 |
| total |  |  |  | 6.925 |  | 16.801 |

**From probabilities to the metrics**

1. Chain rule, MLE: $p=0.8333\cdot0.3333\cdot0.4\cdot1\cdot0.6667\cdot0.1111\cdot1=2^{-6.925}$. Cross-entropy $H=6.925/7=0.989$ bits/token, so perplexity $2^{0.989}=1.985$.
2. Add-one: each count gains 1 and each history total gains $V=12$, so $p(\text{rug}\mid\text{the})=(1+1)/(9+12)=0.0952$. The total is 16.801 bits, $H=2.400$ and perplexity $5.279$. Smoothing costs probability on seen text.
3. Bits per byte: the sentence is 22 bytes plus one for the end of line, $B=23$. MLE gives $6.925/23=0.301$ BPB and add-one gives $16.801/23=0.730$.
4. Now score "a dog sat on the mat". *a* is followed only by *cat* (2) and *mat* (1) in training, so $p_{\text{MLE}}(\text{dog}\mid\text{a})=0/3=0$, the log-probability is $-\infty$ and perplexity is infinite. Add-one gives $1/15=0.067$ and perplexity 7.189.
5. A uniform model over 12 outcomes would have perplexity exactly 12. The unigram MLE sits at 9.596 on this sentence: knowing word frequencies helps a little, and knowing the previous word helps far more.

```python
import math, torch, torch.nn.functional as F
from transformers import AutoModelForCausalLM, AutoTokenizer

tok = AutoTokenizer.from_pretrained("gpt2")
model = AutoModelForCausalLM.from_pretrained("gpt2").eval()

# 1) the causal-LM loss is cross-entropy between logits at t and the token at t + 1
ids = tok("The cat sat on the mat.", return_tensors="pt").input_ids          # (1, T)
with torch.no_grad():
    logits = model(ids).logits                                              # (1, T, 50257)
loss = F.cross_entropy(logits[0, :-1], ids[0, 1:])                          # shift by one
assert torch.allclose(loss, model(ids, labels=ids).loss)                   # HF shifts internally

# 2) perplexity and bits per byte of a long text, with a sliding window
text = open("test.txt", encoding="utf-8").read()
ids = tok(text, return_tensors="pt").input_ids[0]
ctx, stride = 1024, 512
nll, n_tok, prev_end = 0.0, 0, 0
for begin in range(0, len(ids), stride):
    end = min(begin + ctx, len(ids))
    x = ids[begin:end][None]
    y = x.clone()
    y[:, : -(end - prev_end)] = -100        # tokens scored by an earlier window are ignored
    with torch.no_grad():
        out = model(x, labels=y)            # mean NLL (nats) over the scored targets
    n = int((y[:, 1:] != -100).sum())       # labels are shifted inside the model
    nll += out.loss.item() * n
    n_tok += n
    prev_end = end
    if end == len(ids):
        break

loss = nll / n_tok                          # nats per token
print(f"perplexity     {math.exp(loss):.2f}")
print(f"bits per token {loss / math.log(2):.3f}")
print(f"bits per byte  {nll / math.log(2) / len(text.encode('utf-8')):.3f}")  # tokenizer-independent
```

Causal-LM loss, perplexity and bits per byte of a text file with GPT-2 (PyTorch + Hugging Face transformers; not run here)

> **Perplexities across tokenizers are not comparable:** A perplexity of 10 with a 32k vocabulary and 12 with a 128k vocabulary says nothing about which model is better. The larger vocabulary predicts fewer, longer tokens. Compare on the **same text** in bits per byte (or total nats) instead. The same caution applies to context length and evaluation protocol. A sliding window with overlap gives each token more context than disjoint chunks, so it reports lower perplexity for the same model.

> **Language modeling is compression:** An arithmetic coder driven by a model uses $-\log_2 p(x_t\mid x_{<t})$ bits per token, so the cross-entropy is the achievable compressed size. Lower loss means a shorter code. This is why bits per byte is natural, and why enwik8 is both a compression benchmark (the Hutter Prize) and a language-modeling benchmark. The figure's 0.410 bits per byte would compress that sentence about 20 times, but only because the model has memorised a 40-token corpus.

> **One zero is fatal:** Log-likelihood sums over tokens, so a single $p=0$ makes the whole corpus $-\infty$ and perplexity infinite. Count-based models need smoothing. Neural LMs avoid zeros because a softmax is never exactly zero. A tokenizer that maps unseen strings to `<unk>` hides the problem instead, and that is one reason for byte-level vocabularies (see **Byte-pair encoding (BPE)**).

**Strengths**

- Self-supervised: every token of raw text is a training target, so data is effectively unlimited
- Chain-rule factorization is exact; one model both scores and generates
- Cross-entropy is a strictly proper scoring rule: it is minimised only by the true conditional distribution
- Bits per byte gives a tokenizer-independent, compression-grounded metric

**Limitations**

- Perplexity measures fit to text, not usefulness; low perplexity does not guarantee instruction following or truthfulness
- Per-token perplexity depends on the tokenizer, the test set and the context protocol
- Causal LMs see only left context during training; masked LMs see both sides but cannot generate
- Left-to-right generation is sequential: T tokens need T forward steps (see KV cache)

**Question:** Model A has perplexity 9.0 per token with a 32k vocabulary; model B has 11.0 per token with a 128k vocabulary, on the same text. What can you conclude?

- A. A is better because its perplexity is lower
- B. B is better because its vocabulary is larger
- C. Nothing yet: convert both to bits per byte (or total bits) on the same text before comparing
- D. They are equal because perplexity is normalised by vocabulary size

**Answer:** C. Nothing yet: convert both to bits per byte (or total bits) on the same text before comparing — Per-token perplexity depends on how many tokens the text is split into. If B uses 25% fewer tokens, its total code length $T_B\log_2 11$ can be smaller than $T_A\log_2 9$. Bits per byte, $-\log_2 p(\text{text})/B$, charges both to the same bytes.

**Question:** Why does a masked LM like BERT get fewer training signals per sequence than a causal LM?

- A. It uses a smaller vocabulary
- B. The loss is computed only at the ~15% masked positions, while a causal LM has a target at every position
- C. It attends in both directions, which halves the gradient
- D. It must predict the [MASK] token itself

**Answer:** B. The loss is computed only at the ~15% masked positions, while a causal LM has a target at every position — A causal LM's triangular mask lets every position predict its successor, so it has $T$ targets. BERT corrupts and predicts about $0.15\,T$ positions. ELECTRA's replaced-token detection was designed to recover a loss at every position.

**Key points**

- Chain rule: $p(x_{1:T})=\prod_t p(x_t\mid x_{<t})$ is exact; n-grams truncate the history to $n-1$ tokens, neural LMs keep it up to the context window.
- Training loss: cross-entropy $-\tfrac1T\sum_t\log p(x_t\mid x_{<t})=H(p_{\text{data}})+D_{\mathrm{KL}}$; the entropy of the text is a floor.
- Perplexity $=e^{\mathcal L}$, the effective branching factor; bits per byte $=-\log_2 p(\text{text})/\text{bytes}$ removes the dependence on the tokenizer.
- Count-based LMs need smoothing (add-k, interpolation, Kneser–Ney) because one unseen n-gram makes perplexity infinite.
- Causal LM: T targets per sequence, generates. Masked LM: about 15% targets, bidirectional, does not define p(x). Span corruption and prefix LMs sit between.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
