# Transformer-based language models

> Transformer LMs form three families set by the attention mask: encoder-only (bidirectional, MLM: BERT), decoder-only (causal, next-token: GPT, Llama) and encoder–decoder (span corruption or denoising: T5, BART). Loss is a power law in N and D; Chinchilla’s optimum is ~20 tokens per parameter.

Source: https://www.mlfoundry.in/learn/transformer-based-language-models  
Section: Deep Learning › Large Language Models (LLMs)  
Level: core · 16 min read · updated 2026-10-13

**Transformer-based language model:** is a neural network built from Transformer blocks (Vaswani et al., "Attention Is All You Need", NeurIPS 2017) and pretrained on raw text to model the probability of tokens given other tokens. Models differ in two choices. The **attention mask** decides which positions each token may see. The **pretraining objective** decides which tokens are predicted from which. These two choices define three families: **encoder-only** models such as BERT, **decoder-only** models such as GPT and Llama, and **encoder–decoder** models such as T5 and BART. Almost every large language model since 2018 belongs to one of them.

## From recurrent LMs to pretrained transformers

Before 2018, language models were mostly LSTMs (see **Language modeling**). They were trained left to right, one step at a time, and transferred to tasks mainly through their hidden states (see **Contextual embeddings**). The Transformer removed the recurrence. Every position attends to every allowed position in one matrix product, so training parallelises across the sequence and long-range dependencies are one step away instead of $T$ steps. The original model was an encoder–decoder for translation (see **Encoder–decoder transformer**). Within 18 months, each half had been turned into a pretrained general-purpose model.

In June 2018 OpenAI's **GPT** (Radford et al., "Improving Language Understanding by Generative Pre-Training") took the decoder alone, trained it to predict the next token on BooksCorpus and fine-tuned it per task. In October 2018 Google's **BERT** (Devlin, Chang, Lee and Toutanova, NAACL 2019) took the encoder alone and trained it to fill in masked tokens, so every layer sees both sides. BERT set new results on GLUE, SQuAD and other benchmarks and made encoder fine-tuning the default for language understanding for several years. In October 2019 **T5** (Raffel et al., JMLR 2020) and **BART** (Lewis et al., ACL 2020) kept both halves and trained them to reconstruct corrupted text. From 2020, GPT-3 showed what a very large decoder-only model can do with no fine-tuning at all, and that family has dominated since.

> Figure: The three families. Encoder-only: bidirectional self-attention (full mask) and a head that predicts only the masked tokens. Decoder-only: causal self-attention (lower-triangular mask) and a head that predicts the next token at every position. Encoder–decoder: a bidirectional encoder reads the corrupted input, and a causal decoder with cross-attention writes the target; its combined mask is a full encoder block, a full cross block and a causal decoder block. Below: a timeline of the models discussed here and the growth in size and training tokens.

## Three masks, one block

All three families use the same block: self-attention and an MLP with residual connections and layer normalisation. The difference is a mask $M$ added to the attention logits before the softmax:

$$
\mathrm{Attn}(Q,K,V)=\mathrm{softmax}\!\Big(\frac{QK^\top}{\sqrt{d_h}}+M\Big)V,\qquad M_{ij}=\begin{cases}0 & \text{if } i \text{ may attend to } j\\ -\infty & \text{otherwise}\end{cases}
$$

- $M^{\text{bi}}_{ij}=0\ \forall i,j$: encoder-only: every token sees the whole sequence, $T^2$ allowed entries
- $M^{\text{causal}}_{ij}=0 \iff j\le i$: decoder-only: each token sees itself and the past, $T(T+1)/2$ entries (see **Causal attention**)
- $M^{\text{prefix}}_{ij}=0 \iff j<P \ \lor\ j\le i$: prefix LM: bidirectional over the first $P$ tokens, causal after (UniLM; a T5 variant)
- $\text{enc–dec}$: encoder rows bidirectional; decoder rows see all encoder states through cross-attention and earlier decoder states causally

The mask determines what the model can be trained to predict. A token that can see itself cannot be asked to predict itself, or the task is trivial. A bidirectional encoder must therefore hide the targets in the *input*, by masking or corrupting them. A causal decoder hides the future through the mask itself, so every position can be a target at once. An encoder–decoder splits the sequence: the encoder reads one part bidirectionally and the decoder predicts the other part causally.

## The objectives

$$
\mathcal L_{\text{CLM}}=-\sum_{t=1}^{T}\log p_\theta(x_t\mid x_{<t}),\qquad \mathcal L_{\text{MLM}}=-\sum_{i\in\mathcal M}\log p_\theta(x_i\mid \tilde x),\qquad \mathcal L_{\text{seq2seq}}=-\sum_{t=1}^{|y|}\log p_\theta\big(y_t\mid y_{<t},\,\tilde x\big)
$$

- $\mathcal L_{\text{CLM}}$: causal LM, the GPT objective: every token is a target, so a sequence of $T$ tokens gives $T$ predictions from one forward pass
- $\mathcal M,\ \tilde x$: MLM, the BERT objective: choose 15% of positions; replace 80% of them with [MASK], 10% with a random token, keep 10%; predict the originals from the corrupted $\tilde x$
- $y,\ \tilde x$: encoder–decoder denoising: the encoder reads a corrupted $\tilde x$, the decoder generates the target $y$

The 80/10/10 rule exists because [MASK] never appears at fine-tuning time. If the model only ever had to predict at [MASK] positions, it could ignore the representations of ordinary tokens. Random and unchanged tokens force it to keep a good representation of every position. The cost of MLM is efficiency: only 15% of tokens produce a loss. ELECTRA (Clark et al., ICLR 2020) recovered the rest with **replaced token detection**: a small generator fills the masks and the main model classifies every token as original or replaced.

**Span corruption** (T5) drops contiguous spans: 15% of tokens with a mean span length of 3. Each span is replaced by a unique sentinel in the input, and the target lists each sentinel followed by the tokens it hid. The paper's example turns "Thank you for inviting me to your party last week." into the input "Thank you <X> me to your party <Y> week." and the target "<X> for inviting <Y> last <Z>". Targets are short, so the decoder is cheap. **Text infilling** (BART) replaces spans whose lengths are drawn from Poisson($\lambda=3$) with a *single* [MASK] each, about 30% of tokens in total, and also permutes sentences. The decoder regenerates the whole original text, so the model must also work out how many tokens each mask hides.

**Landmark configurations**

- **GPT (2018):** decoder-only, 12 layers, $d=768$, 12 heads, 512-token context, about 117M parameters; BooksCorpus; BPE with 40,000 merges
- **BERT (2018):** encoder-only; Base 12 layers, $d=768$, 12 heads, 110M; Large 24 layers, $d=1024$, 16 heads, 340M; BooksCorpus (800M words) + English Wikipedia (2,500M words); MLM + next-sentence prediction; 512 tokens; WordPiece vocabulary of 30,000
- **GPT-2 (2019):** up to 1.5B parameters, 48 layers, $d=1600$; WebText (about 40 GB, 8M documents); 1024-token context; byte-level BPE, 50,257 tokens
- **RoBERTa (2019):** BERT-Large architecture retrained on 160 GB of text with dynamic masking, no next-sentence prediction and much larger batches
- **T5 (2019):** encoder–decoder, sizes from 60M to 11B; C4, about 750 GB of cleaned Common Crawl; span corruption at 15%, mean span 3; SentencePiece vocabulary of 32,000; the final models were pre-trained on about 1 trillion tokens
- **BART (2019):** encoder–decoder; Large has 12 encoder and 12 decoder layers, $d=1024$; text infilling (30% of tokens, Poisson(3) spans) + sentence permutation; the same 160 GB as RoBERTa
- **GPT-3 (2020):** decoder-only, 175B parameters, 96 layers, $d=12288$, 96 heads, 2048-token context, 300B training tokens; few-shot in-context learning without fine-tuning
- **After 2020:** PaLM (2022): 540B parameters, 780B tokens. Chinchilla (2022): 70B, 1.4T tokens. LLaMA (2023): 7B–65B, 1.0–1.4T tokens. Llama 2 (2023): 2T tokens. Llama 3 (2024): over 15T tokens. All decoder-only

> Figure: One 11-token sentence prepared for five objectives, with the input, the target positions that carry a loss and the attention mask (rows are queries; for encoder–decoders, the rows and columns are [encoder | decoder]). At the default, decoder-only CLM, all 11 positions are targets (the last predicts </s>), and the causal mask allows 66 of 121 entries (54.5%). Encoder-only MLM at 15% masks round(1.65) = 2 tokens (with seed 1, "last week" → [MASK] [MASK]). Only 2 of 11 positions (18.2%) have a loss, but the mask is full: 121 of 121. A prefix LM with a 50% prefix sees the first 6 tokens bidirectionally. Its loss is on the last 6 predictions, and 81 of 121 entries (66.9%) are allowed. T5 span corruption at its paper setting (15%, mean span 3) gives 2 noise tokens in one span. T5's segmentation always ends on a noise span, so the input is "… party last <X>" (10 tokens) and the target is "<X> week . <Y>" (4 tokens, 2 of them sentinels). The combined mask allows 150 of 196 entries (76.5%). At 50% and mean span 1 the 6 noise tokens split into 5 spans, and the target grows to 12 tokens. BART infilling at 30% hides 3 tokens behind 2 [MASK]s (10 encoder tokens). The decoder rebuilds all 11, and the mask allows 276 of 441 entries (62.6%). Resample to draw new positions.

## Which family for which job

**Encoder-only** models give the best representation per parameter for understanding tasks: classification, token tagging, extractive question answering, retrieval and reranking. Every token sees the full context. They cannot generate text naturally, because nothing in their training predicts a next token (see **BERT**, **RoBERTa**, **ALBERT**, **DistilBERT**, **Encoder-only models**).

**Decoder-only** models are the simplest and the most general. One stack, one mask and one objective cover every task framed as text continuation. Training uses every token, and generation reuses cached keys and values (see **KV cache**). With enough scale they learn tasks from a few examples in the prompt. Since GPT-3, almost all frontier LLMs have been decoder-only (see **GPT architecture**, **Llama**, **Decoder-only models**).

**Encoder–decoder** models fit tasks with a clear input and output, such as translation, summarisation and structured generation. The encoder reads the input bidirectionally, once, and the decoder attends to it at every step. T5 cast every task as text-to-text, and its controlled comparison found the encoder–decoder with span corruption the best of the architectures it tested at a matched cost (see **T5**, **BART**, **Encoder–decoder models**). Wang et al. (ICML 2022) trained many architecture and objective combinations at 5B scale. A causal decoder trained on plain language modelling generalised best zero-shot straight after pretraining. Models with bidirectional input attention, trained with masked objectives and then multitask fine-tuning, did best after fine-tuning.

## Scaling

$$
C\approx 6ND,\qquad L(N,D)=E+\frac{A}{N^{\alpha}}+\frac{B}{D^{\beta}},\qquad N_{\text{opt}}\propto C^{0.5},\ D_{\text{opt}}\propto C^{0.5}
$$

- $C\approx6ND$: training FLOPs for $N$ parameters and $D$ tokens: 2 per parameter per token forward, 4 backward
- $L(N,D)$: the parametric fit of Hoffmann et al. ("Training Compute-Optimal Large Language Models", 2022): $E=1.69$, $A=406.4$, $B=410.7$, $\alpha=0.34$, $\beta=0.28$
- $N_{\text{opt}},D_{\text{opt}}$: minimising $L$ subject to $6ND=C$ gives parameters and tokens growing equally with compute, about 20 tokens per parameter

Kaplan et al. ("Scaling Laws for Neural Language Models", 2020) first showed smooth power laws for decoder-only transformers. In their fits, loss falls as $L\propto N^{-0.076}$ with parameters and $L\propto D^{-0.095}$ with data when the other is not a bottleneck. Architecture details such as depth versus width barely mattered at fixed $N$. They concluded that compute should go mostly into larger models, roughly $N\propto C^{0.73}$, which is how GPT-3 came to be trained on only 300B tokens. Hoffmann et al. redid the analysis with learning-rate schedules matched to each run's length and found $N$ and $D$ should grow equally. Their 70B **Chinchilla**, trained on 1.4T tokens, outperformed the 280B Gopher, trained on 300B tokens with about the same compute.

Practice has since moved past "compute-optimal". The 20-tokens-per-parameter rule minimises *training* compute for a given loss. A model that will serve billions of requests is cheaper overall if it is smaller and trained far longer. Llama 3 8B saw over 15T tokens, nearly 2,000 per parameter. The scaling laws still hold; the objective being optimised has changed.

## Worked example: masks, targets and a compute budget

The 11-token sentence under each objective (seed 1 in the figure; counts checked in Python)

| setup | input tokens | target tokens | tokens with a loss | mask entries allowed |
| --- | --- | --- | --- | --- |
| decoder-only, CLM | 11 | 11 | 11 | 66 / 121 |
| encoder-only, MLM 15% | 11 | 11 (2 scored) | 2 | 121 / 121 |
| prefix LM, P = 6 | 11 | 11 (6 scored) | 6 | 81 / 121 |
| T5 span corruption 15% | 10 | 4 | 4 | 150 / 196 |
| BART infilling 30% | 10 | 11 | 11 | 276 / 441 |

**Counting entries and choosing N and D**

1. Causal: row $i$ sees $i$ keys, so $\sum_{i=1}^{11} i=66$. Prefix with $P=6$: the first 6 rows see 6 keys (36), and rows 7–11 see $6+1,\dots,6+5$ (45), 81 in total.
2. Encoder–decoder with $n_e$ encoder and $n_d$ decoder tokens: $n_e^2+n_dn_e+n_d(n_d+1)/2$. For T5's $10+4$: $100+40+10=150$. For BART's $10+11$: $100+110+66=276$.
3. Compute budget: GPT-3 used $C=6\cdot175\cdot10^9\cdot300\cdot10^9=3.15\cdot10^{23}$ FLOPs.
4. At 20 tokens per parameter, $C=6N\cdot20N=120N^2$, so $N=\sqrt{3.15\cdot10^{23}/120}=51$B parameters and $D=20N=1.02$T tokens: a model a third the size, trained on more than three times the data.
5. The Chinchilla fit predicts loss $1.69+406.4/(70\cdot10^9)^{0.34}+410.7/(1.4\cdot10^{12})^{0.28}=1.937$ for Chinchilla and $1.993$ for Gopher (280B, 300B tokens), whose budgets by $6ND$ are $5.88\cdot10^{23}$ and $5.04\cdot10^{23}$.

```python
import torch
from transformers import (AutoTokenizer, AutoModelForMaskedLM,
                          AutoModelForCausalLM, AutoModelForSeq2SeqLM)

text = "Thank you for inviting me to your party last week."

# 1) encoder-only, masked LM: loss only where we masked
tok = AutoTokenizer.from_pretrained("bert-base-uncased")
bert = AutoModelForMaskedLM.from_pretrained("bert-base-uncased")
enc = tok(text, return_tensors="pt")
labels = enc.input_ids.clone()
hide = torch.tensor([w in (2, 3) for w in enc.word_ids(0)])[None]   # the words "for inviting"
enc.input_ids[hide] = tok.mask_token_id
labels[~hide] = -100                                   # -100 = ignored by the cross-entropy
print("MLM loss", bert(**enc, labels=labels).loss.item())

# 2) decoder-only, causal LM: every token is a target (the model shifts labels internally)
tok = AutoTokenizer.from_pretrained("gpt2")
gpt2 = AutoModelForCausalLM.from_pretrained("gpt2")
ids = tok(text, return_tensors="pt").input_ids
print("CLM loss", gpt2(input_ids=ids, labels=ids).loss.item())

# 3) encoder–decoder, T5 span corruption: sentinels in the input, spans in the target
tok = AutoTokenizer.from_pretrained("t5-small")
t5 = AutoModelForSeq2SeqLM.from_pretrained("t5-small")
src = tok("Thank you <extra_id_0> me to your party <extra_id_1> week.", return_tensors="pt").input_ids
tgt = tok("<extra_id_0> for inviting <extra_id_1> last <extra_id_2>", return_tensors="pt").input_ids
print("span-corruption loss", t5(input_ids=src, labels=tgt).loss.item())

# The masks themselves, for T = 11 tokens and a 6-token prefix
T, P = 11, 6
bidirectional = torch.ones(T, T, dtype=torch.bool)
causal = torch.tril(torch.ones(T, T, dtype=torch.bool))
prefix = causal.clone()
prefix[:, :P] = True                                   # everyone sees the whole prefix
print(bidirectional.sum().item(), causal.sum().item(), prefix.sum().item())   # 121 66 81

# as an additive bias for attention logits
bias = torch.zeros(T, T).masked_fill(~causal, float("-inf"))
```

The three objectives on one sentence with Hugging Face transformers, and the three masks in torch (not run here: written for torch 2 and transformers 4; the mask counts were checked separately in numpy)

> **A BERT is not a small GPT:** An MLM-trained encoder has never predicted a next token. Asking it to generate, by repeatedly appending [MASK] and taking the argmax, gives poor text. Conversely, the hidden state of a causal decoder at token $t$ has seen only the left context. Using it as a token embedding for tagging loses the right half of the sentence. Choose the family by what the task needs to see and what it must produce.

> **The mask is the architecture:** An encoder–decoder is, mask-wise, a prefix LM whose prefix and continuation use separate weights. A prefix LM is a decoder whose mask opens up over the prompt. Much of the "architecture debate" is a debate over the mask and the objective, which is why UniLM could train one network as an encoder, a decoder and a seq2seq model just by switching masks, and why UL2 mixes several denoising objectives in one model.

> **Loss-bearing tokens per FLOP:** Per forward pass, CLM scores every token, MLM about 15%, and span corruption about 15% of the input plus the sentinels. That is one reason decoder-only pretraining is efficient at scale. Encoder–decoders compensate with short targets: the decoder in the figure's T5 example runs over 4 tokens instead of 11.

**Strengths**

- Encoder-only: full bidirectional context; best per-parameter for classification, tagging and retrieval
- Decoder-only: one objective covers every task; every token is a target; simple KV-cached generation; scales best
- Encoder–decoder: a bidirectional read of the input plus fluent generation; short targets make denoising cheap
- All three: the same block and the same scaling behaviour, so lessons transfer

**Limitations**

- Encoder-only: no natural generation; only 15% of tokens supervised per pass; [MASK] never appears at test time
- Decoder-only: no bidirectional view of the input unless a prefix mask is added
- Encoder–decoder: two stacks and cross-attention to manage, and an input/output split that is awkward for chat
- All three: quality is bought with compute; the compute-optimal and the inference-optimal size differ

**Question:** Why can a decoder-only model use every token in a sequence as a training target, while BERT uses only about 15%?

- A. Decoders have more parameters
- B. The causal mask hides each token's future, so position t can predict token t+1 without seeing it; a bidirectional encoder sees every token, so targets must be hidden in the input
- C. BERT's vocabulary is smaller
- D. BERT uses next-sentence prediction for the other 85%

**Answer:** B. The causal mask hides each token's future, so position t can predict token t+1 without seeing it; a bidirectional encoder sees every token, so targets must be hidden in the input — With $M_{ij}=0\iff j\le i$, the prediction at $t$ never depends on $x_{t+1}$, so all $T$ predictions are valid in one pass. In a bidirectional encoder every position sees every other, so only corrupted positions can be scored. Masking more than about 15% removes too much context.

**Question:** You have compute C = 3.15·10²³ FLOPs. Following the Chinchilla rule of about 20 tokens per parameter with C ≈ 6ND, roughly what model should you train?

- A. 175B parameters on 300B tokens
- B. About 51B parameters on about 1T tokens
- C. About 10B parameters on about 5T tokens
- D. About 500B parameters on about 100B tokens

**Answer:** B. About 51B parameters on about 1T tokens — $C=6N\cdot20N=120N^2$, so $N=\sqrt{3.15\cdot10^{23}/120}\approx5.1\cdot10^{10}$ and $D=20N\approx1.0\cdot10^{12}$. GPT-3 spent the same budget on 175B parameters and 300B tokens, far from this optimum.

**Key points**

- Three families, one block: encoder-only (bidirectional mask, MLM), decoder-only (causal mask $j\le i$, next-token), encoder–decoder (bidirectional encoder, causal decoder with cross-attention, span corruption or denoising).
- MLM scores about 15% of tokens with 80/10/10 replacement; CLM scores all of them; T5 replaces 15% in mean-3 spans with sentinels; BART infills Poisson(3) spans covering 30% and rebuilds the whole text.
- Timeline: Transformer 2017, GPT and BERT 2018, GPT-2 2019, T5 and BART 2019, GPT-3 2020, Chinchilla and PaLM 2022, LLaMA 2023; decoder-only models dominate today.
- Training compute is $C\approx6ND$; loss follows power laws in $N$ and $D$, and the compute-optimal split is about 20 tokens per parameter (Chinchilla 70B on 1.4T).
- Inference cost pushes modern models to train far beyond 20 tokens per parameter (Llama 3 8B: over 15T tokens).

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
