Definition
Language model is a probability distribution over sequences of tokens, . By the chain rule it factorizes, with no loss of generality, into next-token conditionals . A model of those conditionals can score text (higher probability means more plausible) and generate it, by sampling one token at a time. The idea goes back to Claude Shannon ("A Mathematical Theory of Communication", 1948), who generated English from letter and word -gram statistics. The neural version starts with Bengio, Ducharme, Vincent and Jauvin ("A Neural Probabilistic Language Model", JMLR 2003). Every model in this module, from BERT to Llama, is trained with a language-modeling objective.
For decades language models were a component, not a product. A speech recognizer or a translation system proposes candidate sentences, and the LM ranks them by how likely they are as English. "Recognize speech" should beat "wreck a nice beach". Those systems used n-gram models: count how often each word follows each short history in a large corpus, and smooth the counts so unseen combinations do not get probability zero. Katz back-off (1987) and Kneser–Ney smoothing (Kneser and Ney, 1995) were the standard. Chen and Goodman's 1998 comparison found a modified Kneser–Ney the best of the methods they tested.
Two limits of n-grams drove the move to neural models. The first is sparsity. The number of possible histories grows as , so most 4-grams never occur even in billions of words. The second is no generalization. Seeing "the cat sat" teaches an n-gram nothing about "the dog sat", because cat and dog are unrelated symbols. Bengio et al. mapped each word to a learned vector and predicted the next word with a neural network over the concatenated vectors of the previous words. Similar words get similar vectors, so evidence is shared between them (see Word embeddings). Mikolov et al. ("Recurrent neural network based language model", Interspeech 2010) removed the fixed window with an RNN. Transformers (see Decoder-only transformer) then made it possible to condition on thousands of tokens and to train on all positions in parallel.
The chain rule and the training objective
- all earlier tokens; a beginning-of-sequence token stands in for the empty history at
- logits from the network: an n-gram table, an RNN, or the LM head of a transformer
- vocabulary size, set by the tokenizer (50,257 for GPT-2, 32,000 for Llama 2, 128,256 for Llama 3; see Tokenization algorithms)
- the factorization is exact for any distribution. Models differ only in how they approximate each conditional
The chain rule makes no independence assumption. An n-gram model adds one, the Markov assumption . Neural LMs keep the full history, up to the context window. Training maximises the likelihood of a corpus. Equivalently, it minimises the average negative log-probability of each actual next token:
- the token-level cross-entropy, in nats with or bits with ; it is exactly the softmax cross-entropy classification loss, applied at every position
- the entropy of the language itself, a floor no model can beat. Shannon ("Prediction and Entropy of Printed English", 1951) estimated it at roughly 0.6–1.3 bits per character for English
- the part a better model can remove; cross-entropy minus the entropy floor
Perplexity and bits per byte
- the inverse geometric-mean probability per token: the model is as uncertain as a uniform choice among PPL tokens. A uniform model has
- number of tokens and number of UTF-8 bytes in the evaluation text
- total bits divided by bytes. It does not depend on the tokenizer, so models with different vocabularies can be compared (used by the Pile benchmark, enwik8, and many scaling studies)
Perplexity is only comparable when the tokenization and the test text are the same. A model with a bigger vocabulary uses fewer, harder-to-predict tokens, so its per-token perplexity is higher even when it compresses the text better. Bits per byte (or bits per character) fixes this by charging the total code length to the bytes of the text. Per token, the conversion is . With a loss of 3.0 nats per token and 4.3 bytes per token, that is bits per byte.
Numbers worth knowing
- Shannon 1951
- human prediction experiments put the entropy of printed English at about 0.6–1.3 bits per letter, with long contexts
- Jozefowicz et al. 2016
- "Exploring the Limits of Language Modeling": on the One Billion Word benchmark, large LSTMs with character-CNN inputs cut single-model perplexity from 51.3 to 30.0; an ensemble reached 23.7
- GPT-2 1.5B (2019)
- zero-shot, with no training on the benchmarks: perplexity 35.76 on Penn Treebank, 18.34 on WikiText-2, 17.48 on WikiText-103; 0.93 bits per byte on enwik8 and 0.98 bits per character on text8
- BERT (2019)
- masked LM: 15% of tokens are chosen; of those, 80% become
[MASK], 10% a random token and 10% stay unchanged - T5 (2020)
- span corruption: 15% of tokens are dropped in spans of mean length 3, and the decoder generates the missing spans
- Llama 2 (2023)
- causal LM, 2 trillion training tokens, 4,096-token context, 32,000-token vocabulary
Causal versus masked objectives
A causal LM (autoregressive LM) predicts from at every position, under a lower-triangular attention mask (see Causal attention). It defines a proper distribution over sequences, so it can score text and generate it. GPT, Llama, Qwen and Mistral are causal LMs. A sequence of tokens gives training targets in one forward pass.
A masked LM (Devlin et al., BERT, NAACL 2019) corrupts the input, replacing 15% of positions as described above, and predicts the original tokens there from context on both sides. Bidirectional context makes better features for classification and extraction. But only the masked 15% of positions give a loss, and the model does not define , so it cannot generate fluently. RoBERTa keeps the objective and changes the masks dynamically. ELECTRA (Clark et al., 2020) replaces masking with a discriminator that detects replaced tokens, so that every position contributes to the loss.
Between the two sit prefix LMs, which attend bidirectionally over a prefix and causally over the rest, and span corruption in T5 and BART: an encoder reads text with spans removed, and a decoder generates them autoregressively. XLNet (Yang et al., 2019) trains a causal LM over random factorization orders, so it gets bidirectional context while keeping a proper likelihood. Today's general-purpose LLMs are almost all causal, because one objective covers both pretraining and generation (see Decoder-only models and Pretraining and continued pretraining).
- the corrupted input: masked, randomly replaced or unchanged tokens at the positions
- the chosen positions; the loss ignores every other position
Worked example: a bigram by hand
Train on the six sentences in the figure. The word the occurs 9 times as a history, followed by cat 3 times, dog 4, mat 1 and rug 1. Score "the cat sat on the rug" with a bigram, first by maximum likelihood (MLE), then with add-one (Laplace) smoothing over outcomes.
| history → token | c(h,w) | c(h) | p MLE | −log₂ p MLE | p add-1 | −log₂ p add-1 |
|---|---|---|---|---|---|---|
| <s> → the | 5 | 6 | 0.8333 | 0.263 | 0.3333 | 1.585 |
| the → cat | 3 | 9 | 0.3333 | 1.585 | 0.1905 | 2.392 |
| cat → sat | 2 | 5 | 0.4000 | 1.322 | 0.1765 | 2.503 |
| sat → on | 3 | 3 | 1.0000 | 0.000 | 0.2667 | 1.907 |
| on → the | 2 | 3 | 0.6667 | 0.585 | 0.2000 | 2.322 |
| the → rug | 1 | 9 | 0.1111 | 3.170 | 0.0952 | 3.392 |
| rug → </s> | 1 | 1 | 1.0000 | 0.000 | 0.1538 | 2.700 |
| total | 6.925 | 16.801 |
From probabilities to the metrics
- 1Chain rule, MLE: . Cross-entropy bits/token, so perplexity .
- 2Add-one: each count gains 1 and each history total gains , so . The total is 16.801 bits, and perplexity . Smoothing costs probability on seen text.
- 3Bits per byte: the sentence is 22 bytes plus one for the end of line, . MLE gives BPB and add-one gives .
- 4Now score "a dog sat on the mat". a is followed only by cat (2) and mat (1) in training, so , the log-probability is and perplexity is infinite. Add-one gives and perplexity 7.189.
- 5A uniform model over 12 outcomes would have perplexity exactly 12. The unigram MLE sits at 9.596 on this sentence: knowing word frequencies helps a little, and knowing the previous word helps far more.
1import math, torch, torch.nn.functional as F2from transformers import AutoModelForCausalLM, AutoTokenizer34tok = AutoTokenizer.from_pretrained("gpt2")5model = AutoModelForCausalLM.from_pretrained("gpt2").eval()67# 1) the causal-LM loss is cross-entropy between logits at t and the token at t + 18ids = tok("The cat sat on the mat.", return_tensors="pt").input_ids # (1, T)9with torch.no_grad():10 logits = model(ids).logits # (1, T, 50257)11loss = F.cross_entropy(logits[0, :-1], ids[0, 1:]) # shift by one12assert torch.allclose(loss, model(ids, labels=ids).loss) # HF shifts internally1314# 2) perplexity and bits per byte of a long text, with a sliding window15text = open("test.txt", encoding="utf-8").read()16ids = tok(text, return_tensors="pt").input_ids[0]17ctx, stride = 1024, 51218nll, n_tok, prev_end = 0.0, 0, 019for begin in range(0, len(ids), stride):20 end = min(begin + ctx, len(ids))21 x = ids[begin:end][None]22 y = x.clone()23 y[:, : -(end - prev_end)] = -100 # tokens scored by an earlier window are ignored24 with torch.no_grad():25 out = model(x, labels=y) # mean NLL (nats) over the scored targets26 n = int((y[:, 1:] != -100).sum()) # labels are shifted inside the model27 nll += out.loss.item() * n28 n_tok += n29 prev_end = end30 if end == len(ids):31 break3233loss = nll / n_tok # nats per token34print(f"perplexity {math.exp(loss):.2f}")35print(f"bits per token {loss / math.log(2):.3f}")36print(f"bits per byte {nll / math.log(2) / len(text.encode('utf-8')):.3f}") # tokenizer-independentUse when
- +Self-supervised: every token of raw text is a training target, so data is effectively unlimited
- +Chain-rule factorization is exact; one model both scores and generates
- +Cross-entropy is a strictly proper scoring rule: it is minimised only by the true conditional distribution
- +Bits per byte gives a tokenizer-independent, compression-grounded metric
Avoid when
- −Perplexity measures fit to text, not usefulness; low perplexity does not guarantee instruction following or truthfulness
- −Per-token perplexity depends on the tokenizer, the test set and the context protocol
- −Causal LMs see only left context during training; masked LMs see both sides but cannot generate
- −Left-to-right generation is sequential: T tokens need T forward steps (see KV cache)
Check yourself
Model A has perplexity 9.0 per token with a 32k vocabulary; model B has 11.0 per token with a 128k vocabulary, on the same text. What can you conclude?
Check yourself
Why does a masked LM like BERT get fewer training signals per sequence than a causal LM?
Remember
- ✓Chain rule: is exact; n-grams truncate the history to tokens, neural LMs keep it up to the context window.
- ✓Training loss: cross-entropy ; the entropy of the text is a floor.
- ✓Perplexity , the effective branching factor; bits per byte removes the dependence on the tokenizer.
- ✓Count-based LMs need smoothing (add-k, interpolation, Kneser–Ney) because one unseen n-gram makes perplexity infinite.
- ✓Causal LM: T targets per sequence, generates. Masked LM: about 15% targets, bidirectional, does not define p(x). Span corruption and prefix LMs sit between.