Definition
Decoder-only model is a language model family in which a single stack of causally masked transformer blocks is pre-trained on raw text to predict each next token, and every task is then posed as continuing a text prompt. The line starts with OpenAI's GPT (Radford, Narasimhan, Salimans and Sutskever, "Improving Language Understanding by Generative Pre-Training", 2018). GPT-3 (Brown et al., "Language Models are Few-Shot Learners", NeurIPS 2020) made it dominant by showing that one large model can do new tasks from a few examples in the prompt. PaLM, Llama, Mistral, Qwen, Gemma and DeepSeek all follow it. The block-level maths (masked attention, parameter and FLOP counts) is in Decoder-only transformer. This article covers the family: why it won, its variants, and how the main models differ.
In 2018–2020 three designs competed for pre-training. Encoder-only models such as BERT read text bidirectionally and are trained to fill in masked tokens. They are strong at classification and retrieval but cannot generate text natively. Encoder–decoder models such as T5 and BART read an input with one stack and write an output with another. Decoder-only models read and write with the same causal stack. The T5 paper's controlled comparison (Raffel et al., 2020) favoured encoder–decoders with a denoising objective for fine-tuned transfer. Yet almost every frontier model since 2020 is decoder-only. The reasons are mostly about scale and use, not about any single benchmark.
Three objectives, one sequence
- causal LM (GPT, Llama): position sees ; all positions are targets
- prefix LM: the first tokens attend to each other in both directions; only the continuation tokens are targets
- masked LM (BERT): full attention; only the masked set , about 15% of positions, is predicted
- prefix length: the conditioning part of the sequence (the input, in encoder–decoder terms)
- 1 if query position may attend to key position
- allowed pairs; the prefix LM adds the upper triangle of the block
The three models can share the same weights and differ only in the mask and the set of loss positions. That makes the comparison clean. The causal mask is the only one under which every position is both an input and a target, and under which no position's representation depends on anything after it.
Why decoder-only won
Every token is a training signal. Under the causal mask one forward pass over tokens yields next-token losses. BERT's masked LM predicts only the 15% of positions it hides, and ELECTRA (Clark, Luong, Le and Manning, ICLR 2020) was motivated by this waste. A prefix LM gives up the targets inside its prefix. With a fixed budget of tokens and FLOPs, the causal objective extracts the most supervision. It also needs no corruption hyperparameters: no mask rate, no span lengths, no sentinels.
Simplicity. There is one stack, one loss and one data format. Any document is a training example as it stands, with no input/output split. Training and inference run the same computation: inference is training's forward pass with sampling appended. The same code, kernels and parallelism serve every size from 125M to 405B. Scaling-law work (Kaplan et al., 2020; Hoffmann et al., 2022) was done on decoder-only models, so their behaviour at scale is the best mapped.
In-context learning. A decoder-only model can be given a task by writing it into the prefix: an instruction, a few input–output examples, and then the query. GPT-3 showed that this few-shot prompting improves sharply with scale, with no gradient updates and no task-specific head. Because the model is trained to continue arbitrary text, any task that can be phrased as text fits the same interface. Instruction tuning and RLHF later turned this interface into chat assistants (see Supervised fine-tuning (SFT), Instruction tuning).
- worked examples written into the prompt; the weights stay fixed
- is zero-shot, one-shot; GPT-3 used up to as many as fit in its 2,048-token window
KV caching is exact. With a causal mask, appending a token never changes the keys and values of earlier tokens. A server can therefore compute a prompt once, keep its KV cache, and pay one forward step per new token. A system prompt shared by many users can be cached once, and a chat history can be extended turn by turn without recomputation. Under a bidirectional or prefix-LM mask, a longer prefix changes every earlier representation, so the prefix must be re-encoded. Over a -turn conversation this costs about times more:
- user and assistant tokens per turn
- number of turns; the ratio grows linearly in K
Wang, Roberts, Hesslow, Le Scao, Chung, Beltagy, Launay and Raffel tested the architecture question directly ("What Language Model Architecture and Pretraining Objective Work Best for Zero-Shot Generalization?", ICML 2022). They trained causal decoders, non-causal (prefix) decoders and encoder–decoders with over 5B parameters on more than 170B tokens. A causal decoder trained with the plain LM objective had the best zero-shot performance straight after pre-training. A non-causal model trained with masked-LM-style denoising and then multitask-finetuned was best overall. Pre-trained causal decoders could also be adapted cheaply into non-causal ones. So the decoder-only choice is not a free win on every metric. It is the best default when the model must be used as-is, by prompting, across open-ended tasks.
Prefix-LM and other variants
The prefix LM is the main hybrid: bidirectional attention over the input, causal over the output, all in one stack. UniLM (Dong et al., "Unified Language Model Pre-training for Natural Language Understanding and Generation", NeurIPS 2019) trained one transformer under bidirectional, left-to-right and sequence-to-sequence masks. The T5 paper included a prefix LM in its architecture comparison. UL2 (Tay et al., "UL2: Unifying Language Learning Paradigms", ICLR 2023) mixed several denoisers, including a prefix-LM "S-denoiser", and trained a 20B model. GLM (Du et al., ACL 2022) used autoregressive blank infilling: spans are cut out of a bidirectionally encoded context and regenerated left to right. Vision-language models such as PaliGemma (Beyer et al., 2024) use a prefix-LM mask over the image tokens and the prompt.
In practice the prefix idea mostly survives as a loss mask, not an attention mask. Supervised fine-tuning of a chat model keeps causal attention but drops the prompt tokens from the loss (label in Hugging Face). This keeps exact KV caching and trains only on the response. Two other variants change the block rather than the mask. Mixture-of-experts decoders (Mixtral, DeepSeek-V3) route each token through a few expert MLPs, which decouples the parameter count from the per-token compute (see Mixture of experts (MoE)). Sliding-window attention (Mistral 7B) bounds the cache size.
The families
| model | year | sizes | tokens | context | attention | vocab |
|---|---|---|---|---|---|---|
| GPT-3 | 2020 | 125M–175B | 300B | 2,048 | MHA, dense + banded sparse layers | 50,257 |
| PaLM | 2022 | 8B, 62B, 540B | 780B | 2,048 | multi-query, parallel blocks | 256K |
| Llama | 2023 | 7B–65B | 1.0T–1.4T | 2,048 | MHA | 32K |
| Llama 2 | 2023 | 7B, 13B, 70B | 2T | 4,096 | GQA in 70B | 32K |
| Llama 3.1 | 2024 | 8B, 70B, 405B | ~15T | 128K | GQA, 8 KV heads | 128K |
| Mistral 7B | 2023 | 7.3B | not disclosed | 8,192 | GQA + sliding window 4,096 | 32K |
| Mixtral 8×7B | 2024 | 47B total, 13B active | not disclosed | 32K | GQA, top-2 of 8 expert MLPs | 32K |
| Qwen2.5 | 2024 | 0.5B–72B | 18T | 32K–128K | GQA, biases on q, k, v | ~152K |
| DeepSeek-V3 | 2024 | 671B total, 37B active | 14.8T | 128K | multi-head latent attention + MoE | 128K |
Read the table by columns. Token counts grew 60-fold in four years (300B to 18T), much faster than parameters. A 7B or 8B model trained on 2T–15T tokens is far past the compute-optimal ratio of about 20 tokens per parameter (see Pretraining and continued pretraining). This is deliberate: a smaller, longer-trained model is cheaper to serve. Vocabularies grew from 32K to 128K–256K to cover more languages and code with fewer tokens. Attention moved from full multi-head to grouped-query and latent variants, all to shrink the KV cache. The families themselves are covered in GPT architecture, Llama, Mistral, Qwen and Phi.
Configurations from papers and model cards
- GPT-3 175B
- 96 layers, , 96 heads of 128, context 2,048, 300B tokens, batch 3.2M tokens, peak lr
- PaLM 540B
- 118 layers, , 48 heads, multi-query attention, SwiGLU, attention and MLP in parallel, RoPE, no biases, 256K SentencePiece vocabulary, 780B tokens on 6,144 TPU v4 chips
- Llama 3 8B
- 32 layers, , 32 query heads, 8 KV heads, , vocabulary 128,256, RoPE base 500,000, 15T+ tokens
- Llama 3.1 405B
- 126 layers, , 128 query heads, 8 KV heads, , context 128K, trained with about FLOPs
- Mistral 7B
- 32 layers, , 32 query heads, 8 KV heads, , sliding window 4,096, rolling-buffer KV cache, vocabulary 32,000
- Qwen2.5 7B
- 28 layers, , 28 query heads, 4 KV heads, , RoPE base , biases on the q, k, v projections
Emergent abilities and how to read them
Wei et al. ("Emergent Abilities of Large Language Models", TMLR 2022) collected tasks on which decoder-only models score near chance up to some scale and then jump: multi-step arithmetic, some BIG-Bench tasks, chain-of-thought gains. Schaeffer, Miranda and Koyejo ("Are Emergent Abilities of Large Language Models a Mirage?", NeurIPS 2023) argued that many jumps come from the metric. If per-token accuracy improves smoothly, exact match on a -token answer scores , which stays near 0 and then rises steeply. With per-token or log-likelihood metrics the same models improve smoothly. Both views agree that capability grows predictably with scale in the underlying loss. They disagree on how sudden the useful behaviour is.
Worked example: one sequence, three objectives, one chat
Take a sequence of tokens. The prefix LM uses a prefix of , and the masked LM masks positions.
| objective | targets | share of tokens | allowed pairs | earlier K, V fixed? |
|---|---|---|---|---|
| causal LM | 11 | 91.7% | 78 | yes |
| prefix LM, P = 4 | 8 | 66.7% | 84 | only after the prefix |
| masked LM, 15% | 2 | 16.7% | 144 | no |
Counting by hand
- 1Causal: positions 1–11 each predict the next token, so 11 targets. Allowed pairs .
- 2Prefix LM: positions 4–11 predict , so targets. Pairs . The 6 extra pairs over causal are the upper triangle of the prefix block.
- 3Masked LM: 2 targets, and every token sees every token: pairs. For a 4,096-token sequence the gap is 4,095 targets against .
- 4Chat with turns of user and reply tokens. Causal with a KV cache processes each token once: tokens.
- 5Prefix LM: turn re-encodes prefix tokens and generates 200, which is per turn. In total tokens, the causal model, matching .
1import torch, torch.nn.functional as F2from transformers import AutoModelForCausalLM, AutoTokenizer34name = "Qwen/Qwen2.5-0.5B" # any decoder-only checkpoint5tok = AutoTokenizer.from_pretrained(name)6model = AutoModelForCausalLM.from_pretrained(name, torch_dtype=torch.float32).eval()78# 1) Pre-training loss. labels = inputs; the model shifts by one internally,9# so a sequence of T tokens contributes T - 1 targets.10ids = tok("Decoder-only models predict the next token.", return_tensors="pt").input_ids11with torch.no_grad():12 out = model(input_ids=ids, labels=ids)13 logits = out.logits[:, :-1] # position t predicts token t + 114 manual = F.cross_entropy(logits.reshape(-1, logits.size(-1)), ids[:, 1:].reshape(-1))15print(ids.shape[1] - 1, out.loss.item(), manual.item()) # T - 1 targets, the same loss twice1617# 2) Answer-only loss (how SFT is usually done): attention stays causal,18# prompt positions get label -100 and drop out of the cross-entropy.19p_ids = tok("Q: What is 7 * 6?\nA:", return_tensors="pt").input_ids20a_ids = tok(" 42", return_tensors="pt").input_ids21ids2 = torch.cat([p_ids, a_ids], dim=1)22labels = ids2.clone()23labels[:, : p_ids.shape[1]] = -10024with torch.no_grad():25 answer_loss = model(input_ids=ids2, labels=labels).loss2627# 3) The causal mask makes caching exact: encode the prefix once, then step one token.28with torch.no_grad():29 full = model(input_ids=ids2).logits[:, -1]30 pre = model(input_ids=ids2[:, :-1], use_cache=True)31 step = model(input_ids=ids2[:, -1:], past_key_values=pre.past_key_values).logits[:, -1]32print(torch.allclose(full, step, atol=1e-4)) # should be True: earlier keys/values never change3334# 4) A prefix-LM mask (UniLM / UL2 style): bidirectional inside the first P tokens.35def prefix_lm_mask(T: int, P: int) -> torch.Tensor:36 m = torch.tril(torch.ones(T, T, dtype=torch.bool))37 m[:P, :P] = True # True = may attend38 return m3940print(prefix_lm_mask(6, 3).int())41# [[1,1,1,0,0,0], [1,1,1,0,0,0], [1,1,1,0,0,0], [1,1,1,1,0,0], [1,1,1,1,1,0], [1,1,1,1,1,1]]42q = k = v = torch.randn(1, 4, 6, 16) # (batch, heads, T, d_head)43y = F.scaled_dot_product_attention(q, k, v, attn_mask=prefix_lm_mask(6, 3))Use when
- +Every token of every document is a target: maximum supervision per FLOP
- +One stack, one loss, one data format; training and inference share the same computation
- +Any task can be posed as a prompt: zero-shot, few-shot and chat use the same interface
- +Exact, reusable KV caching across tokens, turns and requests
- +The best-studied scaling behaviour and the largest ecosystem of kernels, serving systems and checkpoints
Avoid when
- −The input is read only left to right, which is weaker than bidirectional encoding for some understanding tasks at equal size
- −Prompt and output share one KV cache, so long inputs make the cache large
- −Pure next-token pre-training gives a base model that continues text rather than following instructions; SFT and preference tuning are needed
- −Generation is sequential, one forward step per output token
Where it's used
Chat assistants
Instruction-tuned decoder-only models behind most assistants; the conversation is the prefix.
Code models
Code completion and repair, where left-to-right generation matches how code is written; fill-in-the-middle is added as a data transformation.
Multimodal LLMs
Image or audio encoders project into the token stream of a decoder-only backbone, sometimes with a prefix-LM mask over the non-text tokens.
Check yourself
A chatbot keeps the full conversation as context. Why can a causal decoder reuse its KV cache from turn to turn while a prefix LM with a bidirectional prefix cannot?
Check yourself
Over one 4,096-token sequence, how many loss terms do a causal LM and BERT-style 15% masked LM get?
Remember
- ✓Decoder-only models train one causal stack on : targets per sequence, against about for masked LM and for a prefix LM.
- ✓They won through simplicity (one stack, one loss, raw text), maximum training signal, prompting and in-context learning as a universal interface, and exact KV caching.
- ✓Causal masking keeps earlier keys and values fixed. A prefix LM must re-encode its prefix as it grows: about times more work over chat turns.
- ✓Wang et al. (2022): a causal decoder is best zero-shot after pre-training alone; non-causal plus denoising plus multitask finetuning is best after adaptation.
- ✓The families (GPT, PaLM, Llama, Mistral, Qwen, DeepSeek) share the stack; they differ in tokens (300B to 18T), vocabulary (32K to 256K), attention (MHA, MQA, GQA, MLA) and MoE.