Language modeling

A language model assigns p(x₁…x_T) = ∏ p(xₜ | x<ₜ) by the chain rule and is trained by minimising the cross-entropy −(1/T)Σ log p(xₜ | x<ₜ). Perplexity is e^loss; bits per byte rescales the loss by tokens/bytes so tokenizers compare. Causal LMs predict every next token, masked LMs only masked ones.

15 min read1185 wordscore0 viewsupdated 2026-10-13

Definition

Language model is a probability distribution over sequences of tokens, p(x1,…,xT)p(x_1,\dots,x_T). By the chain rule it factorizes, with no loss of generality, into next-token conditionals ∏tp(xt∣x<t)\prod_t p(x_t\mid x_{<t}). A model of those conditionals can score text (higher probability means more plausible) and generate it, by sampling one token at a time. The idea goes back to Claude Shannon ("A Mathematical Theory of Communication", 1948), who generated English from letter and word nn-gram statistics. The neural version starts with Bengio, Ducharme, Vincent and Jauvin ("A Neural Probabilistic Language Model", JMLR 2003). Every model in this module, from BERT to Llama, is trained with a language-modeling objective.

For decades language models were a component, not a product. A speech recognizer or a translation system proposes candidate sentences, and the LM ranks them by how likely they are as English. "Recognize speech" should beat "wreck a nice beach". Those systems used n-gram models: count how often each word follows each short history in a large corpus, and smooth the counts so unseen combinations do not get probability zero. Katz back-off (1987) and Kneser–Ney smoothing (Kneser and Ney, 1995) were the standard. Chen and Goodman's 1998 comparison found a modified Kneser–Ney the best of the methods they tested.

Two limits of n-grams drove the move to neural models. The first is sparsity. The number of possible histories grows as Vn−1V^{n-1}, so most 4-grams never occur even in billions of words. The second is no generalization. Seeing "the cat sat" teaches an n-gram nothing about "the dog sat", because cat and dog are unrelated symbols. Bengio et al. mapped each word to a learned vector and predicted the next word with a neural network over the concatenated vectors of the previous n−1n-1 words. Similar words get similar vectors, so evidence is shared between them (see Word embeddings). Mikolov et al. ("Recurrent neural network based language model", Interspeech 2010) removed the fixed window with an RNN. Transformers (see Decoder-only transformer) then made it possible to condition on thousands of tokens and to train on all positions in parallel.

The chain rule and the training objective

pθ(x1:T)=∏t=1Tpθ(xt∣x<t),pθ(xt=v∣x<t)=exp⁡(zv(x<t))∑u=1Vexp⁡(zu(x<t))p_\theta(x_{1:T})=\prod_{t=1}^{T}p_\theta(x_t\mid x_{<t}),\qquad p_\theta(x_t=v\mid x_{<t})=\frac{\exp\big(z_v(x_{<t})\big)}{\sum_{u=1}^{V}\exp\big(z_u(x_{<t})\big)}
x<tx_{<t}
all earlier tokens; a beginning-of-sequence token stands in for the empty history at t=1t=1
z(x<t)∈RVz(x_{<t})\in\mathbb R^V
logits from the network: an n-gram table, an RNN, or the LM head of a transformer
VV
vocabulary size, set by the tokenizer (50,257 for GPT-2, 32,000 for Llama 2, 128,256 for Llama 3; see Tokenization algorithms)
∏t\prod_t
the factorization is exact for any distribution. Models differ only in how they approximate each conditional

The chain rule makes no independence assumption. An n-gram model adds one, the Markov assumption p(xt∣x<t)≈p(xt∣xt−n+1:t−1)p(x_t\mid x_{<t})\approx p(x_t\mid x_{t-n+1:t-1}). Neural LMs keep the full history, up to the context window. Training maximises the likelihood of a corpus. Equivalently, it minimises the average negative log-probability of each actual next token:

L(θ)=−1T∑t=1Tlog⁡pθ(xt∣x<t)  ≈  H(pdata,pθ)=H(pdata)+DKL(pdata ∥ pθ)\mathcal L(\theta)=-\frac1T\sum_{t=1}^{T}\log p_\theta(x_t\mid x_{<t})\;\approx\;H(p_{\text{data}},p_\theta)=H(p_{\text{data}})+D_{\mathrm{KL}}\big(p_{\text{data}}\,\|\,p_\theta\big)
L\mathcal L
the token-level cross-entropy, in nats with ln⁡\ln or bits with log⁡2\log_2; it is exactly the softmax cross-entropy classification loss, applied at every position
H(pdata)H(p_{\text{data}})
the entropy of the language itself, a floor no model can beat. Shannon ("Prediction and Entropy of Printed English", 1951) estimated it at roughly 0.6–1.3 bits per character for English
DKLD_{\mathrm{KL}}
the part a better model can remove; cross-entropy minus the entropy floor
chain rule: p(x₁…x_T) = ∏ₜ p(xₜ | x₁…xₜ₋₁), exact for any distributionp(the cat sat)=p(the | <s>)·p(cat | the)·p(sat | the cat)·p(</s> | the cat sat)n-gram LM: truncate the history to n−1 words · neural LM: condition on all of itcausal LM (GPT, Llama)target = input shifted by onequery ikey j ≤ i<s>thethecatcatsatsatonontheloss at every position: T targetsgenerates left to right; scores p(x)exactly by the chain rulemasked LM (BERT)mask 15% of tokens, see both sidesevery keythe[M]catsaton[M]matloss only at masked positionsbidirectional features for understanding;no direct p(x), no generationcross-entropy L = −(1/T) Σₜ ln p(xₜ | x<t) nats/token → perplexity = e^Lbits per byte = L · T / (B · ln 2): T tokens, B UTF-8 bytes; independent of the tokenizern-gram + smoothing (Katz 1987, Kneser–Ney 1995) → feed-forward neural LM (Bengio et al. 2003)→ RNN LM (Mikolov et al. 2010) → LSTM LMs → Transformer LMs (GPT 2018, BERT 2018)
fig.Language modeling at a glance. The chain rule turns the probability of a sentence into a product of next-token predictions, and an n-gram truncates each history while a neural LM keeps all of it. A causal LM (left) uses a lower-triangular mask, and every position predicts the token after it, so a sequence of T tokens gives T training targets. A masked LM (right) attends to both sides but is trained only on the roughly 15% of positions that were masked. The summed negative log-likelihood gives cross-entropy, its exponential is perplexity, and normalizing by UTF-8 bytes instead of tokens gives bits per byte, which compares models with different tokenizers.

Perplexity and bits per byte

PPL=exp⁡(Lnats)=2Lbits=p(x1:T)−1/T,BPB=TB⋅Lnatsln⁡2=−log⁡2p(x1:T)B\mathrm{PPL}=\exp(\mathcal L_{\text{nats}})=2^{\mathcal L_{\text{bits}}}=p(x_{1:T})^{-1/T},\qquad \mathrm{BPB}=\frac{T}{B}\cdot\frac{\mathcal L_{\text{nats}}}{\ln 2}=\frac{-\log_2 p(x_{1:T})}{B}
PPL\mathrm{PPL}
the inverse geometric-mean probability per token: the model is as uncertain as a uniform choice among PPL tokens. A uniform model has PPL=V\mathrm{PPL}=V
T, BT,\ B
number of tokens and number of UTF-8 bytes in the evaluation text
BPB\mathrm{BPB}
total bits divided by bytes. It does not depend on the tokenizer, so models with different vocabularies can be compared (used by the Pile benchmark, enwik8, and many scaling studies)

Perplexity is only comparable when the tokenization and the test text are the same. A model with a bigger vocabulary uses fewer, harder-to-predict tokens, so its per-token perplexity is higher even when it compresses the text better. Bits per byte (or bits per character) fixes this by charging the total code length to the bytes of the text. Per token, the conversion is BPB=Lbits/(bytes per token)\mathrm{BPB}=\mathcal L_{\text{bits}}/(\text{bytes per token}). With a loss of 3.0 nats per token and 4.3 bytes per token, that is 3.0/ln⁡2/4.3=1.0073.0/\ln2/4.3=1.007 bits per byte.

Numbers worth knowing

Shannon 1951
human prediction experiments put the entropy of printed English at about 0.6–1.3 bits per letter, with long contexts
Jozefowicz et al. 2016
"Exploring the Limits of Language Modeling": on the One Billion Word benchmark, large LSTMs with character-CNN inputs cut single-model perplexity from 51.3 to 30.0; an ensemble reached 23.7
GPT-2 1.5B (2019)
zero-shot, with no training on the benchmarks: perplexity 35.76 on Penn Treebank, 18.34 on WikiText-2, 17.48 on WikiText-103; 0.93 bits per byte on enwik8 and 0.98 bits per character on text8
BERT (2019)
masked LM: 15% of tokens are chosen; of those, 80% become [MASK], 10% a random token and 10% stay unchanged
T5 (2020)
span corruption: 15% of tokens are dropped in spans of mean length 3, and the decoder generates the missing spans
Llama 2 (2023)
causal LM, 2 trillion training tokens, 4,096-token context, 32,000-token vocabulary

Causal versus masked objectives

A causal LM (autoregressive LM) predicts xtx_t from x<tx_{<t} at every position, under a lower-triangular attention mask (see Causal attention). It defines a proper distribution over sequences, so it can score text and generate it. GPT, Llama, Qwen and Mistral are causal LMs. A sequence of TT tokens gives TT training targets in one forward pass.

A masked LM (Devlin et al., BERT, NAACL 2019) corrupts the input, replacing 15% of positions as described above, and predicts the original tokens there from context on both sides. Bidirectional context makes better features for classification and extraction. But only the masked 15% of positions give a loss, and the model does not define p(x)p(x), so it cannot generate fluently. RoBERTa keeps the objective and changes the masks dynamically. ELECTRA (Clark et al., 2020) replaces masking with a discriminator that detects replaced tokens, so that every position contributes to the loss.

Between the two sit prefix LMs, which attend bidirectionally over a prefix and causally over the rest, and span corruption in T5 and BART: an encoder reads text with spans removed, and a decoder generates them autoregressively. XLNet (Yang et al., 2019) trains a causal LM over random factorization orders, so it gets bidirectional context while keeping a proper likelihood. Today's general-purpose LLMs are almost all causal, because one objective covers both pretraining and generation (see Decoder-only models and Pretraining and continued pretraining).

LCLM=−∑t=1Tlog⁡p(xt∣x<t),LMLM=−∑t∈Mlog⁡p(xt∣x~),∣M∣≈0.15 T\mathcal L_{\text{CLM}}=-\sum_{t=1}^{T}\log p(x_t\mid x_{<t}),\qquad \mathcal L_{\text{MLM}}=-\sum_{t\in\mathcal M}\log p(x_t\mid \tilde x),\quad |\mathcal M|\approx0.15\,T
x~\tilde x
the corrupted input: masked, randomly replaced or unchanged tokens at the positions M\mathcal M
M\mathcal M
the chosen positions; the loss ignores every other position
p(w | h) = (c(h, w) + k) / (c(h) + k·V) · k = 0.1, V = 12test: −log₂ p(wₜ | h) per token (click a bar) · total 9.429 bits over 7 tokens0.50the0.7081.72cat0.3041.56sat0.3390.44on0.7381.00the0.5003.21rug0.1081.00</s>0.500p( · | the ) over V = 12 · sums to 1.000 · actual next: ruga0.010cat0.304dog0.402mat0.108on0.010ran0.010rug0.108sat0.010saw0.010the0.010to0.010</s>0.010cross-entropy H1.347 bits/tokenperplexity 2^H2.544bits per byte0.410 (23 bytes)train perplexity2.649perplexity per test sentence1. 2.544 the cat sat on the rug2. 4.688 a dog sat on the mat3. 7.894 the mat saw a dogselected: p(rug | the)(c(h,w)+k)/(c(h)+kV) =(1+0.1)/(9+1.2) = 0.1078−log₂ p = 3.213 bitssmoothing moves mass to unseenwords: train PPL ↑, test PPL ↓train: 6 sentences, 40 tokens
order
smoothing
test sentence
fig.An n-gram LM trained on six sentences (40 tokens, V = 11 words + </s>) scores a held-out sentence with the chain rule. Each bar is one token's surprisal −log₂ p, and the lower chart is the full next-token distribution at the selected position. The default is a bigram with add-k smoothing, k = 0.1. On 'the cat sat on the rug', p(rug | the) = (1 + 0.1)/(9 + 1.2) = 0.108, or 3.21 bits. The sentence costs 9.43 bits over 7 tokens: H = 1.347 bits/token, perplexity 2.544, 0.410 bits per byte, against a train perplexity of 2.649. Switch to MLE and that test sentence drops to perplexity 1.985, but 'a dog sat on the mat' becomes ∞, because 'a dog' never occurred and gets p = 0. Laplace (k = 1) gives every word at least 1/21 after 'the' and raises perplexity to 5.279 on test sentence 1 and 5.409 on train. A trigram fits training text better (train perplexity 1.632 with MLE), but with add-0.01 the odd sentence 'the mat saw a dog' reaches 32.272. Interpolating trigram, bigram and unigram (λ = 0.6, 0.3, 0.1) brings it to 19.040 while keeping test sentence 1 at 1.814. The unigram model ignores context and sits near perplexity 10 on everything.

Worked example: a bigram by hand

Train on the six sentences in the figure. The word the occurs 9 times as a history, followed by cat 3 times, dog 4, mat 1 and rug 1. Score "the cat sat on the rug" with a bigram, first by maximum likelihood (MLE), then with add-one (Laplace) smoothing over V=12V=12 outcomes.

history → tokenc(h,w)c(h)p MLE−log₂ p MLEp add-1−log₂ p add-1
<s> → the560.83330.2630.33331.585
the → cat390.33331.5850.19052.392
cat → sat250.40001.3220.17652.503
sat → on331.00000.0000.26671.907
on → the230.66670.5850.20002.322
the → rug190.11113.1700.09523.392
rug → </s>111.00000.0000.15382.700
total6.92516.801
Bigram probabilities for one test sentence (computed in Python)

From probabilities to the metrics

  1. 1Chain rule, MLE: p=0.8333⋅0.3333⋅0.4⋅1⋅0.6667⋅0.1111⋅1=2−6.925p=0.8333\cdot0.3333\cdot0.4\cdot1\cdot0.6667\cdot0.1111\cdot1=2^{-6.925}. Cross-entropy H=6.925/7=0.989H=6.925/7=0.989 bits/token, so perplexity 20.989=1.9852^{0.989}=1.985.
  2. 2Add-one: each count gains 1 and each history total gains V=12V=12, so p(rug∣the)=(1+1)/(9+12)=0.0952p(\text{rug}\mid\text{the})=(1+1)/(9+12)=0.0952. The total is 16.801 bits, H=2.400H=2.400 and perplexity 5.2795.279. Smoothing costs probability on seen text.
  3. 3Bits per byte: the sentence is 22 bytes plus one for the end of line, B=23B=23. MLE gives 6.925/23=0.3016.925/23=0.301 BPB and add-one gives 16.801/23=0.73016.801/23=0.730.
  4. 4Now score "a dog sat on the mat". a is followed only by cat (2) and mat (1) in training, so pMLE(dog∣a)=0/3=0p_{\text{MLE}}(\text{dog}\mid\text{a})=0/3=0, the log-probability is −∞-\infty and perplexity is infinite. Add-one gives 1/15=0.0671/15=0.067 and perplexity 7.189.
  5. 5A uniform model over 12 outcomes would have perplexity exactly 12. The unigram MLE sits at 9.596 on this sentence: knowing word frequencies helps a little, and knowing the previous word helps far more.
python
1import math, torch, torch.nn.functional as F
2from transformers import AutoModelForCausalLM, AutoTokenizer
3
4tok = AutoTokenizer.from_pretrained("gpt2")
5model = AutoModelForCausalLM.from_pretrained("gpt2").eval()
6
7# 1) the causal-LM loss is cross-entropy between logits at t and the token at t + 1
8ids = tok("The cat sat on the mat.", return_tensors="pt").input_ids # (1, T)
9with torch.no_grad():
10 logits = model(ids).logits # (1, T, 50257)
11loss = F.cross_entropy(logits[0, :-1], ids[0, 1:]) # shift by one
12assert torch.allclose(loss, model(ids, labels=ids).loss) # HF shifts internally
13
14# 2) perplexity and bits per byte of a long text, with a sliding window
15text = open("test.txt", encoding="utf-8").read()
16ids = tok(text, return_tensors="pt").input_ids[0]
17ctx, stride = 1024, 512
18nll, n_tok, prev_end = 0.0, 0, 0
19for begin in range(0, len(ids), stride):
20 end = min(begin + ctx, len(ids))
21 x = ids[begin:end][None]
22 y = x.clone()
23 y[:, : -(end - prev_end)] = -100 # tokens scored by an earlier window are ignored
24 with torch.no_grad():
25 out = model(x, labels=y) # mean NLL (nats) over the scored targets
26 n = int((y[:, 1:] != -100).sum()) # labels are shifted inside the model
27 nll += out.loss.item() * n
28 n_tok += n
29 prev_end = end
30 if end == len(ids):
31 break
32
33loss = nll / n_tok # nats per token
34print(f"perplexity {math.exp(loss):.2f}")
35print(f"bits per token {loss / math.log(2):.3f}")
36print(f"bits per byte {nll / math.log(2) / len(text.encode('utf-8')):.3f}") # tokenizer-independent
Causal-LM loss, perplexity and bits per byte of a text file with GPT-2 (PyTorch + Hugging Face transformers; not run here)

Use when

  • +Self-supervised: every token of raw text is a training target, so data is effectively unlimited
  • +Chain-rule factorization is exact; one model both scores and generates
  • +Cross-entropy is a strictly proper scoring rule: it is minimised only by the true conditional distribution
  • +Bits per byte gives a tokenizer-independent, compression-grounded metric

Avoid when

  • −Perplexity measures fit to text, not usefulness; low perplexity does not guarantee instruction following or truthfulness
  • −Per-token perplexity depends on the tokenizer, the test set and the context protocol
  • −Causal LMs see only left context during training; masked LMs see both sides but cannot generate
  • −Left-to-right generation is sequential: T tokens need T forward steps (see KV cache)

Check yourself

Model A has perplexity 9.0 per token with a 32k vocabulary; model B has 11.0 per token with a 128k vocabulary, on the same text. What can you conclude?

Check yourself

Why does a masked LM like BERT get fewer training signals per sequence than a causal LM?

Remember

  • ✓Chain rule: p(x1:T)=∏tp(xt∣x<t)p(x_{1:T})=\prod_t p(x_t\mid x_{<t}) is exact; n-grams truncate the history to n−1n-1 tokens, neural LMs keep it up to the context window.
  • ✓Training loss: cross-entropy −1T∑tlog⁡p(xt∣x<t)=H(pdata)+DKL-\tfrac1T\sum_t\log p(x_t\mid x_{<t})=H(p_{\text{data}})+D_{\mathrm{KL}}; the entropy of the text is a floor.
  • ✓Perplexity =eL=e^{\mathcal L}, the effective branching factor; bits per byte =−log⁡2p(text)/bytes=-\log_2 p(\text{text})/\text{bytes} removes the dependence on the tokenizer.
  • ✓Count-based LMs need smoothing (add-k, interpolation, Kneser–Ney) because one unseen n-gram makes perplexity infinite.
  • ✓Causal LM: T targets per sequence, generates. Masked LM: about 15% targets, bidirectional, does not define p(x). Span corruption and prefix LMs sit between.
← back to Deep Learning roadmap