Definition
Transformer-based language model is a neural network built from Transformer blocks (Vaswani et al., "Attention Is All You Need", NeurIPS 2017) and pretrained on raw text to model the probability of tokens given other tokens. Models differ in two choices. The attention mask decides which positions each token may see. The pretraining objective decides which tokens are predicted from which. These two choices define three families: encoder-only models such as BERT, decoder-only models such as GPT and Llama, and encoder–decoder models such as T5 and BART. Almost every large language model since 2018 belongs to one of them.
From recurrent LMs to pretrained transformers
Before 2018, language models were mostly LSTMs (see Language modeling). They were trained left to right, one step at a time, and transferred to tasks mainly through their hidden states (see Contextual embeddings). The Transformer removed the recurrence. Every position attends to every allowed position in one matrix product, so training parallelises across the sequence and long-range dependencies are one step away instead of steps. The original model was an encoder–decoder for translation (see Encoder–decoder transformer). Within 18 months, each half had been turned into a pretrained general-purpose model.
In June 2018 OpenAI's GPT (Radford et al., "Improving Language Understanding by Generative Pre-Training") took the decoder alone, trained it to predict the next token on BooksCorpus and fine-tuned it per task. In October 2018 Google's BERT (Devlin, Chang, Lee and Toutanova, NAACL 2019) took the encoder alone and trained it to fill in masked tokens, so every layer sees both sides. BERT set new results on GLUE, SQuAD and other benchmarks and made encoder fine-tuning the default for language understanding for several years. In October 2019 T5 (Raffel et al., JMLR 2020) and BART (Lewis et al., ACL 2020) kept both halves and trained them to reconstruct corrupted text. From 2020, GPT-3 showed what a very large decoder-only model can do with no fine-tuning at all, and that family has dominated since.
Three masks, one block
All three families use the same block: self-attention and an MLP with residual connections and layer normalisation. The difference is a mask added to the attention logits before the softmax:
- encoder-only: every token sees the whole sequence, allowed entries
- decoder-only: each token sees itself and the past, entries (see Causal attention)
- prefix LM: bidirectional over the first tokens, causal after (UniLM; a T5 variant)
- encoder rows bidirectional; decoder rows see all encoder states through cross-attention and earlier decoder states causally
The mask determines what the model can be trained to predict. A token that can see itself cannot be asked to predict itself, or the task is trivial. A bidirectional encoder must therefore hide the targets in the input, by masking or corrupting them. A causal decoder hides the future through the mask itself, so every position can be a target at once. An encoder–decoder splits the sequence: the encoder reads one part bidirectionally and the decoder predicts the other part causally.
The objectives
- causal LM, the GPT objective: every token is a target, so a sequence of tokens gives predictions from one forward pass
- MLM, the BERT objective: choose 15% of positions; replace 80% of them with [MASK], 10% with a random token, keep 10%; predict the originals from the corrupted
- encoder–decoder denoising: the encoder reads a corrupted , the decoder generates the target
The 80/10/10 rule exists because [MASK] never appears at fine-tuning time. If the model only ever had to predict at [MASK] positions, it could ignore the representations of ordinary tokens. Random and unchanged tokens force it to keep a good representation of every position. The cost of MLM is efficiency: only 15% of tokens produce a loss. ELECTRA (Clark et al., ICLR 2020) recovered the rest with replaced token detection: a small generator fills the masks and the main model classifies every token as original or replaced.
Span corruption (T5) drops contiguous spans: 15% of tokens with a mean span length of 3. Each span is replaced by a unique sentinel in the input, and the target lists each sentinel followed by the tokens it hid. The paper's example turns "Thank you for inviting me to your party last week." into the input "Thank you <X> me to your party <Y> week." and the target "<X> for inviting <Y> last <Z>". Targets are short, so the decoder is cheap. Text infilling (BART) replaces spans whose lengths are drawn from Poisson() with a single [MASK] each, about 30% of tokens in total, and also permutes sentences. The decoder regenerates the whole original text, so the model must also work out how many tokens each mask hides.
Landmark configurations
- GPT (2018)
- decoder-only, 12 layers, , 12 heads, 512-token context, about 117M parameters; BooksCorpus; BPE with 40,000 merges
- BERT (2018)
- encoder-only; Base 12 layers, , 12 heads, 110M; Large 24 layers, , 16 heads, 340M; BooksCorpus (800M words) + English Wikipedia (2,500M words); MLM + next-sentence prediction; 512 tokens; WordPiece vocabulary of 30,000
- GPT-2 (2019)
- up to 1.5B parameters, 48 layers, ; WebText (about 40 GB, 8M documents); 1024-token context; byte-level BPE, 50,257 tokens
- RoBERTa (2019)
- BERT-Large architecture retrained on 160 GB of text with dynamic masking, no next-sentence prediction and much larger batches
- T5 (2019)
- encoder–decoder, sizes from 60M to 11B; C4, about 750 GB of cleaned Common Crawl; span corruption at 15%, mean span 3; SentencePiece vocabulary of 32,000; the final models were pre-trained on about 1 trillion tokens
- BART (2019)
- encoder–decoder; Large has 12 encoder and 12 decoder layers, ; text infilling (30% of tokens, Poisson(3) spans) + sentence permutation; the same 160 GB as RoBERTa
- GPT-3 (2020)
- decoder-only, 175B parameters, 96 layers, , 96 heads, 2048-token context, 300B training tokens; few-shot in-context learning without fine-tuning
- After 2020
- PaLM (2022): 540B parameters, 780B tokens. Chinchilla (2022): 70B, 1.4T tokens. LLaMA (2023): 7B–65B, 1.0–1.4T tokens. Llama 2 (2023): 2T tokens. Llama 3 (2024): over 15T tokens. All decoder-only
Which family for which job
Encoder-only models give the best representation per parameter for understanding tasks: classification, token tagging, extractive question answering, retrieval and reranking. Every token sees the full context. They cannot generate text naturally, because nothing in their training predicts a next token (see BERT, RoBERTa, ALBERT, DistilBERT, Encoder-only models).
Decoder-only models are the simplest and the most general. One stack, one mask and one objective cover every task framed as text continuation. Training uses every token, and generation reuses cached keys and values (see KV cache). With enough scale they learn tasks from a few examples in the prompt. Since GPT-3, almost all frontier LLMs have been decoder-only (see GPT architecture, Llama, Decoder-only models).
Encoder–decoder models fit tasks with a clear input and output, such as translation, summarisation and structured generation. The encoder reads the input bidirectionally, once, and the decoder attends to it at every step. T5 cast every task as text-to-text, and its controlled comparison found the encoder–decoder with span corruption the best of the architectures it tested at a matched cost (see T5, BART, Encoder–decoder models). Wang et al. (ICML 2022) trained many architecture and objective combinations at 5B scale. A causal decoder trained on plain language modelling generalised best zero-shot straight after pretraining. Models with bidirectional input attention, trained with masked objectives and then multitask fine-tuning, did best after fine-tuning.
Scaling
- training FLOPs for parameters and tokens: 2 per parameter per token forward, 4 backward
- the parametric fit of Hoffmann et al. ("Training Compute-Optimal Large Language Models", 2022): , , , ,
- minimising subject to gives parameters and tokens growing equally with compute, about 20 tokens per parameter
Kaplan et al. ("Scaling Laws for Neural Language Models", 2020) first showed smooth power laws for decoder-only transformers. In their fits, loss falls as with parameters and with data when the other is not a bottleneck. Architecture details such as depth versus width barely mattered at fixed . They concluded that compute should go mostly into larger models, roughly , which is how GPT-3 came to be trained on only 300B tokens. Hoffmann et al. redid the analysis with learning-rate schedules matched to each run's length and found and should grow equally. Their 70B Chinchilla, trained on 1.4T tokens, outperformed the 280B Gopher, trained on 300B tokens with about the same compute.
Practice has since moved past "compute-optimal". The 20-tokens-per-parameter rule minimises training compute for a given loss. A model that will serve billions of requests is cheaper overall if it is smaller and trained far longer. Llama 3 8B saw over 15T tokens, nearly 2,000 per parameter. The scaling laws still hold; the objective being optimised has changed.
Worked example: masks, targets and a compute budget
| setup | input tokens | target tokens | tokens with a loss | mask entries allowed |
|---|---|---|---|---|
| decoder-only, CLM | 11 | 11 | 11 | 66 / 121 |
| encoder-only, MLM 15% | 11 | 11 (2 scored) | 2 | 121 / 121 |
| prefix LM, P = 6 | 11 | 11 (6 scored) | 6 | 81 / 121 |
| T5 span corruption 15% | 10 | 4 | 4 | 150 / 196 |
| BART infilling 30% | 10 | 11 | 11 | 276 / 441 |
Counting entries and choosing N and D
- 1Causal: row sees keys, so . Prefix with : the first 6 rows see 6 keys (36), and rows 7–11 see (45), 81 in total.
- 2Encoder–decoder with encoder and decoder tokens: . For T5's : . For BART's : .
- 3Compute budget: GPT-3 used FLOPs.
- 4At 20 tokens per parameter, , so B parameters and T tokens: a model a third the size, trained on more than three times the data.
- 5The Chinchilla fit predicts loss for Chinchilla and for Gopher (280B, 300B tokens), whose budgets by are and .
1import torch2from transformers import (AutoTokenizer, AutoModelForMaskedLM,3 AutoModelForCausalLM, AutoModelForSeq2SeqLM)45text = "Thank you for inviting me to your party last week."67# 1) encoder-only, masked LM: loss only where we masked8tok = AutoTokenizer.from_pretrained("bert-base-uncased")9bert = AutoModelForMaskedLM.from_pretrained("bert-base-uncased")10enc = tok(text, return_tensors="pt")11labels = enc.input_ids.clone()12hide = torch.tensor([w in (2, 3) for w in enc.word_ids(0)])[None] # the words "for inviting"13enc.input_ids[hide] = tok.mask_token_id14labels[~hide] = -100 # -100 = ignored by the cross-entropy15print("MLM loss", bert(**enc, labels=labels).loss.item())1617# 2) decoder-only, causal LM: every token is a target (the model shifts labels internally)18tok = AutoTokenizer.from_pretrained("gpt2")19gpt2 = AutoModelForCausalLM.from_pretrained("gpt2")20ids = tok(text, return_tensors="pt").input_ids21print("CLM loss", gpt2(input_ids=ids, labels=ids).loss.item())2223# 3) encoder–decoder, T5 span corruption: sentinels in the input, spans in the target24tok = AutoTokenizer.from_pretrained("t5-small")25t5 = AutoModelForSeq2SeqLM.from_pretrained("t5-small")26src = tok("Thank you <extra_id_0> me to your party <extra_id_1> week.", return_tensors="pt").input_ids27tgt = tok("<extra_id_0> for inviting <extra_id_1> last <extra_id_2>", return_tensors="pt").input_ids28print("span-corruption loss", t5(input_ids=src, labels=tgt).loss.item())2930# The masks themselves, for T = 11 tokens and a 6-token prefix31T, P = 11, 632bidirectional = torch.ones(T, T, dtype=torch.bool)33causal = torch.tril(torch.ones(T, T, dtype=torch.bool))34prefix = causal.clone()35prefix[:, :P] = True # everyone sees the whole prefix36print(bidirectional.sum().item(), causal.sum().item(), prefix.sum().item()) # 121 66 813738# as an additive bias for attention logits39bias = torch.zeros(T, T).masked_fill(~causal, float("-inf"))Use when
- +Encoder-only: full bidirectional context; best per-parameter for classification, tagging and retrieval
- +Decoder-only: one objective covers every task; every token is a target; simple KV-cached generation; scales best
- +Encoder–decoder: a bidirectional read of the input plus fluent generation; short targets make denoising cheap
- +All three: the same block and the same scaling behaviour, so lessons transfer
Avoid when
- −Encoder-only: no natural generation; only 15% of tokens supervised per pass; [MASK] never appears at test time
- −Decoder-only: no bidirectional view of the input unless a prefix mask is added
- −Encoder–decoder: two stacks and cross-attention to manage, and an input/output split that is awkward for chat
- −All three: quality is bought with compute; the compute-optimal and the inference-optimal size differ
Check yourself
Why can a decoder-only model use every token in a sequence as a training target, while BERT uses only about 15%?
Check yourself
You have compute C = 3.15·10²³ FLOPs. Following the Chinchilla rule of about 20 tokens per parameter with C ≈ 6ND, roughly what model should you train?
Remember
- ✓Three families, one block: encoder-only (bidirectional mask, MLM), decoder-only (causal mask , next-token), encoder–decoder (bidirectional encoder, causal decoder with cross-attention, span corruption or denoising).
- ✓MLM scores about 15% of tokens with 80/10/10 replacement; CLM scores all of them; T5 replaces 15% in mean-3 spans with sentinels; BART infills Poisson(3) spans covering 30% and rebuilds the whole text.
- ✓Timeline: Transformer 2017, GPT and BERT 2018, GPT-2 2019, T5 and BART 2019, GPT-3 2020, Chinchilla and PaLM 2022, LLaMA 2023; decoder-only models dominate today.
- ✓Training compute is ; loss follows power laws in and , and the compute-optimal split is about 20 tokens per parameter (Chinchilla 70B on 1.4T).
- ✓Inference cost pushes modern models to train far beyond 20 tokens per parameter (Llama 3 8B: over 15T tokens).