Decoder-only models

Decoder-only models (GPT, PaLM, Llama, Mistral, Qwen, DeepSeek) train one causal stack on next-token prediction. Every token is a target (T − 1 per sequence vs ~15% for masked LM), prompts double as task specifications, and the causal mask makes the KV cache exact and reusable.

14 min read1612 wordscore0 viewsupdated 2026-10-13

Definition

Decoder-only model is a language model family in which a single stack of causally masked transformer blocks is pre-trained on raw text to predict each next token, and every task is then posed as continuing a text prompt. The line starts with OpenAI's GPT (Radford, Narasimhan, Salimans and Sutskever, "Improving Language Understanding by Generative Pre-Training", 2018). GPT-3 (Brown et al., "Language Models are Few-Shot Learners", NeurIPS 2020) made it dominant by showing that one large model can do new tasks from a few examples in the prompt. PaLM, Llama, Mistral, Qwen, Gemma and DeepSeek all follow it. The block-level maths (masked attention, parameter and FLOP counts) is in Decoder-only transformer. This article covers the family: why it won, its variants, and how the main models differ.

In 2018–2020 three designs competed for pre-training. Encoder-only models such as BERT read text bidirectionally and are trained to fill in masked tokens. They are strong at classification and retrieval but cannot generate text natively. Encoder–decoder models such as T5 and BART read an input with one stack and write an output with another. Decoder-only models read and write with the same causal stack. The T5 paper's controlled comparison (Raffel et al., 2020) favoured encoder–decoders with a denoising objective for fine-tuned transfer. Yet almost every frontier model since 2020 is decoder-only. The reasons are mostly about scale and use, not about any single benchmark.

Three objectives, one sequence

LCLM=−∑t=1T−1log⁡p(xt+1∣x≤t),LPLM=−∑t=PT−1log⁡p(xt+1∣x≤t),LMLM=−∑t∈Mlog⁡p(xt∣x∖M)\mathcal L_{\text{CLM}}=-\sum_{t=1}^{T-1}\log p(x_{t+1}\mid x_{\le t}),\qquad \mathcal L_{\text{PLM}}=-\sum_{t=P}^{T-1}\log p(x_{t+1}\mid x_{\le t}),\qquad \mathcal L_{\text{MLM}}=-\sum_{t\in\mathcal M}\log p(x_t\mid x_{\setminus\mathcal M})
CLM\text{CLM}
causal LM (GPT, Llama): position tt sees x≤tx_{\le t}; all T−1T-1 positions are targets
PLM\text{PLM}
prefix LM: the first PP tokens attend to each other in both directions; only the T−PT-P continuation tokens are targets
MLM\text{MLM}
masked LM (BERT): full attention; only the masked set M\mathcal M, about 15% of positions, is predicted
PP
prefix length: the conditioning part of the sequence (the input, in encoder–decoder terms)
MijCLM=1[j≤i],MijPLM=1[j≤i ∨ (i≤P∧j≤P)],MijMLM=1M^{\text{CLM}}_{ij}=\mathbb 1[j\le i],\qquad M^{\text{PLM}}_{ij}=\mathbb 1[j\le i\ \lor\ (i\le P\land j\le P)],\qquad M^{\text{MLM}}_{ij}=1
MijM_{ij}
1 if query position ii may attend to key position jj
∑ijMijCLM\sum_{ij}M^{\text{CLM}}_{ij}
T(T+1)/2T(T+1)/2 allowed pairs; the prefix LM adds the upper triangle of the P×PP\times P block

The three models can share the same weights and differ only in the mask and the set of loss positions. That makes the comparison clean. The causal mask is the only one under which every position is both an input and a target, and under which no position's representation depends on anything after it.

one causal stack, next-token loss, scaled up: the decoder-only familiesOpenAIGPT2018 · 117MGPT-22019 · 1.5BGPT-32020 · 175B175B, 300B tokens:few-shot promptingthe recipe spreads (open weights from 2022)GooglePaLM2022 · 540BGemma2024 · 2B, 7BGemma 22024 · 2B–27BMetaOPT2022 · 125M–175BLlama2023 · 7B–65BLlama 22023 · 7B–70BLlama 3.12024 · 8B–405BMistralMistral 7B2023 · 7.3B, SWAMixtral 8×7B2024 · 13B of 47BAlibabaQwen2023 · 7B, 14BQwen22024 · 0.5B–72BQwen2.52024 · 18T tokensDeepSeekDeepSeek LLM2024 · 7B, 67BDeepSeek-V32024 · 37B of 671Bwhat changed inside the block (the stack itself did not)GPT-2 / GPT-3 erapre-LayerNorm, learned positionsGELU MLP, d_ff = 4dfull multi-head attention50,257-token byte-level BPEcontext 1,024 → 2,048Llama era (2023 →)pre-RMSNorm, RoPE, mostly no biasesSwiGLU MLP, d_ff ≈ 8d/3GQA (8 KV heads), some MoE32K → 128K–256K vocabulariescontext 4K → 128K2023
fig.The decoder-only lineage. GPT, GPT-2 and GPT-3 established the recipe between 2018 and 2020: scale one causal stack and prompt it. From 2022 Google, Meta, Mistral AI, Alibaba and DeepSeek built their own families on it, with sizes shown as published. The bottom row shows what changed inside the block around 2023. The stack, the mask and the loss stayed the same.

Why decoder-only won

Every token is a training signal. Under the causal mask one forward pass over TT tokens yields T−1T-1 next-token losses. BERT's masked LM predicts only the 15% of positions it hides, and ELECTRA (Clark, Luong, Le and Manning, ICLR 2020) was motivated by this waste. A prefix LM gives up the targets inside its prefix. With a fixed budget of tokens and FLOPs, the causal objective extracts the most supervision. It also needs no corruption hyperparameters: no mask rate, no span lengths, no sentinels.

Simplicity. There is one stack, one loss and one data format. Any document is a training example as it stands, with no input/output split. Training and inference run the same computation: inference is training's forward pass with sampling appended. The same code, kernels and parallelism serve every size from 125M to 405B. Scaling-law work (Kaplan et al., 2020; Hoffmann et al., 2022) was done on decoder-only models, so their behaviour at scale is the best mapped.

In-context learning. A decoder-only model can be given a task by writing it into the prefix: an instruction, a few input–output examples, and then the query. GPT-3 showed that this few-shot prompting improves sharply with scale, with no gradient updates and no task-specific head. Because the model is trained to continue arbitrary text, any task that can be phrased as text fits the same interface. Instruction tuning and RLHF later turned this interface into chat assistants (see Supervised fine-tuning (SFT), Instruction tuning).

p(y∣(x1,y1),…,(xk,yk)⏟demonstrations, x)=∏tp(yt∣demos, x, y<t)p\big(y\mid \underbrace{(x_1,y_1),\dots,(x_k,y_k)}_{\text{demonstrations}},\,x\big)=\prod_{t}p\big(y_t\mid \text{demos},\,x,\,y_{<t}\big)
(xi,yi)(x_i,y_i)
worked examples written into the prompt; the weights stay fixed
kk
k=0k=0 is zero-shot, k=1k=1 one-shot; GPT-3 used up to as many as fit in its 2,048-token window

KV caching is exact. With a causal mask, appending a token never changes the keys and values of earlier tokens. A server can therefore compute a prompt once, keep its KV cache, and pay one forward step per new token. A system prompt shared by many users can be cached once, and a chat history can be extended turn by turn without recomputation. Under a bidirectional or prefix-LM mask, a longer prefix changes every earlier representation, so the prefix must be re-encoded. Over a KK-turn conversation this costs about (K+1)/2(K+1)/2 times more:

K (u+a)⏟causal + KV cachevs∑k=1K[(k−1)(u+a)+u+a]=(u+a) K(K+1)2⏟prefix LM, history re-encoded each turn\underbrace{K\,(u+a)}_{\text{causal + KV cache}}\qquad\text{vs}\qquad \underbrace{\sum_{k=1}^{K}\big[(k-1)(u+a)+u+a\big]=(u+a)\,\frac{K(K+1)}{2}}_{\text{prefix LM, history re-encoded each turn}}
u, au,\ a
user and assistant tokens per turn
KK
number of turns; the ratio grows linearly in K

Wang, Roberts, Hesslow, Le Scao, Chung, Beltagy, Launay and Raffel tested the architecture question directly ("What Language Model Architecture and Pretraining Objective Work Best for Zero-Shot Generalization?", ICML 2022). They trained causal decoders, non-causal (prefix) decoders and encoder–decoders with over 5B parameters on more than 170B tokens. A causal decoder trained with the plain LM objective had the best zero-shot performance straight after pre-training. A non-causal model trained with masked-LM-style denoising and then multitask-finetuned was best overall. Pre-trained causal decoders could also be adapted cheaply into non-causal ones. So the decoder-only choice is not a free win on every metric. It is the best default when the model must be used as-is, by prompting, across open-ended tasks.

causal LM (GPT, Llama) · T = 16key j →i↓lossposition i sees j ≤ i and predicts x(i+1)■ causal ■ bidirectional ● targetone 4,096-token sequence givesCLM 4,095 · PLM (P = 2,048) 2,048 · MLM 614targets for the same forward passtargets per sequenceCLM15/16 = 93.8%PLM10/16 = 62.5%MLM2/16 = 12.5%allowed query–key pairs: 136 of 256per 1B training tokens: 938M targetsappend a token: do earlier K, V change?no: cache them, one step per new tokengenerates left to right nativelychat, 8 turns of 50 user + 200 reply tokenscumulative tokens through the stack9,0000turn 8causal + KV cache 2,000 · prefix LM 9,000(K + 1)/2 = 4.5× more work for the prefix LM
objective
fig.One training sequence under each objective. The grid is the attention mask (row i = query, column j = key), the green dots mark positions that carry a loss term, and the chart counts the work over a chat. At the default T = 16 the causal LM supervises 15 of 16 positions (93.8%) with 136 of 256 query–key pairs. The prefix LM with P = 6 supervises 10 (62.5%) and allows 151 pairs, because the 6 × 6 prefix block is full. The masked LM sees all 256 pairs but predicts only 2 masked tokens (12.5%). At T = 32 it masks 5 (15.6%) against the causal LM's 31 (96.9%). The chat panel assumes 50 user and 200 reply tokens per turn. With a KV cache, a causal model processes 2,000 tokens over 8 turns. A prefix LM that re-encodes its growing bidirectional history processes 9,000 (4.5×). At 20 turns the totals are 5,000 and 52,500 (10.5×).

Prefix-LM and other variants

The prefix LM is the main hybrid: bidirectional attention over the input, causal over the output, all in one stack. UniLM (Dong et al., "Unified Language Model Pre-training for Natural Language Understanding and Generation", NeurIPS 2019) trained one transformer under bidirectional, left-to-right and sequence-to-sequence masks. The T5 paper included a prefix LM in its architecture comparison. UL2 (Tay et al., "UL2: Unifying Language Learning Paradigms", ICLR 2023) mixed several denoisers, including a prefix-LM "S-denoiser", and trained a 20B model. GLM (Du et al., ACL 2022) used autoregressive blank infilling: spans are cut out of a bidirectionally encoded context and regenerated left to right. Vision-language models such as PaliGemma (Beyer et al., 2024) use a prefix-LM mask over the image tokens and the prompt.

In practice the prefix idea mostly survives as a loss mask, not an attention mask. Supervised fine-tuning of a chat model keeps causal attention but drops the prompt tokens from the loss (label −100-100 in Hugging Face). This keeps exact KV caching and trains only on the response. Two other variants change the block rather than the mask. Mixture-of-experts decoders (Mixtral, DeepSeek-V3) route each token through a few expert MLPs, which decouples the parameter count from the per-token compute (see Mixture of experts (MoE)). Sliding-window attention (Mistral 7B) bounds the cache size.

The families

modelyearsizestokenscontextattentionvocab
GPT-32020125M–175B300B2,048MHA, dense + banded sparse layers50,257
PaLM20228B, 62B, 540B780B2,048multi-query, parallel blocks256K
Llama20237B–65B1.0T–1.4T2,048MHA32K
Llama 220237B, 13B, 70B2T4,096GQA in 70B32K
Llama 3.120248B, 70B, 405B~15T128KGQA, 8 KV heads128K
Mistral 7B20237.3Bnot disclosed8,192GQA + sliding window 4,09632K
Mixtral 8×7B202447B total, 13B activenot disclosed32KGQA, top-2 of 8 expert MLPs32K
Qwen2.520240.5B–72B18T32K–128KGQA, biases on q, k, v~152K
DeepSeek-V32024671B total, 37B active14.8T128Kmulti-head latent attention + MoE128K
Main decoder-only families, as published by their authors (tokens = pre-training tokens)

Read the table by columns. Token counts grew 60-fold in four years (300B to 18T), much faster than parameters. A 7B or 8B model trained on 2T–15T tokens is far past the compute-optimal ratio of about 20 tokens per parameter (see Pretraining and continued pretraining). This is deliberate: a smaller, longer-trained model is cheaper to serve. Vocabularies grew from 32K to 128K–256K to cover more languages and code with fewer tokens. Attention moved from full multi-head to grouped-query and latent variants, all to shrink the KV cache. The families themselves are covered in GPT architecture, Llama, Mistral, Qwen and Phi.

Configurations from papers and model cards

GPT-3 175B
96 layers, d=12288d=12288, 96 heads of 128, context 2,048, 300B tokens, batch 3.2M tokens, peak lr 0.6×10−40.6\times10^{-4}
PaLM 540B
118 layers, d=18432d=18432, 48 heads, multi-query attention, SwiGLU, attention and MLP in parallel, RoPE, no biases, 256K SentencePiece vocabulary, 780B tokens on 6,144 TPU v4 chips
Llama 3 8B
32 layers, d=4096d=4096, 32 query heads, 8 KV heads, dff=14336d_{ff}=14336, vocabulary 128,256, RoPE base 500,000, 15T+ tokens
Llama 3.1 405B
126 layers, d=16384d=16384, 128 query heads, 8 KV heads, dff=53248d_{ff}=53248, context 128K, trained with about 3.8×10253.8\times10^{25} FLOPs
Mistral 7B
32 layers, d=4096d=4096, 32 query heads, 8 KV heads, dff=14336d_{ff}=14336, sliding window 4,096, rolling-buffer KV cache, vocabulary 32,000
Qwen2.5 7B
28 layers, d=3584d=3584, 28 query heads, 4 KV heads, dff=18944d_{ff}=18944, RoPE base 10610^6, biases on the q, k, v projections

Emergent abilities and how to read them

Wei et al. ("Emergent Abilities of Large Language Models", TMLR 2022) collected tasks on which decoder-only models score near chance up to some scale and then jump: multi-step arithmetic, some BIG-Bench tasks, chain-of-thought gains. Schaeffer, Miranda and Koyejo ("Are Emergent Abilities of Large Language Models a Mirage?", NeurIPS 2023) argued that many jumps come from the metric. If per-token accuracy pp improves smoothly, exact match on a kk-token answer scores pkp^k, which stays near 0 and then rises steeply. With per-token or log-likelihood metrics the same models improve smoothly. Both views agree that capability grows predictably with scale in the underlying loss. They disagree on how sudden the useful behaviour is.

Worked example: one sequence, three objectives, one chat

Take a sequence of T=12T=12 tokens. The prefix LM uses a prefix of P=4P=4, and the masked LM masks round(0.15⋅12)=round(1.8)=2\mathrm{round}(0.15\cdot12)=\mathrm{round}(1.8)=2 positions.

objectivetargetsshare of tokensallowed pairsearlier K, V fixed?
causal LM1191.7%78yes
prefix LM, P = 4866.7%84only after the prefix
masked LM, 15%216.7%144no
Targets and attention pairs for T = 12, P = 4 (computed in Python)

Counting by hand

  1. 1Causal: positions 1–11 each predict the next token, so 11 targets. Allowed pairs =∑i=112i=12⋅13/2=78=\sum_{i=1}^{12}i=12\cdot13/2=78.
  2. 2Prefix LM: positions 4–11 predict x5…x12x_5\dots x_{12}, so 12−4=812-4=8 targets. Pairs =42+∑i=512i=16+68=84=4^2+\sum_{i=5}^{12}i=16+68=84. The 6 extra pairs over causal are the upper triangle of the 4×44\times4 prefix block.
  3. 3Masked LM: 2 targets, and every token sees every token: 122=14412^2=144 pairs. For a 4,096-token sequence the gap is 4,095 targets against round(614.4)=614\mathrm{round}(614.4)=614.
  4. 4Chat with K=8K=8 turns of u=50u=50 user and a=200a=200 reply tokens. Causal with a KV cache processes each token once: 8⋅250=2,0008\cdot250=2{,}000 tokens.
  5. 5Prefix LM: turn kk re-encodes (k−1)⋅250+50(k-1)\cdot250+50 prefix tokens and generates 200, which is 250k250k per turn. In total 250⋅(1+⋯+8)=250⋅36=9,000250\cdot(1+\dots+8)=250\cdot36=9{,}000 tokens, 4.5×4.5\times the causal model, matching (K+1)/2(K+1)/2.
python
1import torch, torch.nn.functional as F
2from transformers import AutoModelForCausalLM, AutoTokenizer
3
4name = "Qwen/Qwen2.5-0.5B" # any decoder-only checkpoint
5tok = AutoTokenizer.from_pretrained(name)
6model = AutoModelForCausalLM.from_pretrained(name, torch_dtype=torch.float32).eval()
7
8# 1) Pre-training loss. labels = inputs; the model shifts by one internally,
9# so a sequence of T tokens contributes T - 1 targets.
10ids = tok("Decoder-only models predict the next token.", return_tensors="pt").input_ids
11with torch.no_grad():
12 out = model(input_ids=ids, labels=ids)
13 logits = out.logits[:, :-1] # position t predicts token t + 1
14 manual = F.cross_entropy(logits.reshape(-1, logits.size(-1)), ids[:, 1:].reshape(-1))
15print(ids.shape[1] - 1, out.loss.item(), manual.item()) # T - 1 targets, the same loss twice
16
17# 2) Answer-only loss (how SFT is usually done): attention stays causal,
18# prompt positions get label -100 and drop out of the cross-entropy.
19p_ids = tok("Q: What is 7 * 6?\nA:", return_tensors="pt").input_ids
20a_ids = tok(" 42", return_tensors="pt").input_ids
21ids2 = torch.cat([p_ids, a_ids], dim=1)
22labels = ids2.clone()
23labels[:, : p_ids.shape[1]] = -100
24with torch.no_grad():
25 answer_loss = model(input_ids=ids2, labels=labels).loss
26
27# 3) The causal mask makes caching exact: encode the prefix once, then step one token.
28with torch.no_grad():
29 full = model(input_ids=ids2).logits[:, -1]
30 pre = model(input_ids=ids2[:, :-1], use_cache=True)
31 step = model(input_ids=ids2[:, -1:], past_key_values=pre.past_key_values).logits[:, -1]
32print(torch.allclose(full, step, atol=1e-4)) # should be True: earlier keys/values never change
33
34# 4) A prefix-LM mask (UniLM / UL2 style): bidirectional inside the first P tokens.
35def prefix_lm_mask(T: int, P: int) -> torch.Tensor:
36 m = torch.tril(torch.ones(T, T, dtype=torch.bool))
37 m[:P, :P] = True # True = may attend
38 return m
39
40print(prefix_lm_mask(6, 3).int())
41# [[1,1,1,0,0,0], [1,1,1,0,0,0], [1,1,1,0,0,0], [1,1,1,1,0,0], [1,1,1,1,1,0], [1,1,1,1,1,1]]
42q = k = v = torch.randn(1, 4, 6, 16) # (batch, heads, T, d_head)
43y = F.scaled_dot_product_attention(q, k, v, attn_mask=prefix_lm_mask(6, 3))
Next-token loss, answer-only loss, exact KV-cache reuse and a prefix-LM mask with Hugging Face transformers and PyTorch (written for transformers 4.4x; not run here)

Use when

  • +Every token of every document is a target: maximum supervision per FLOP
  • +One stack, one loss, one data format; training and inference share the same computation
  • +Any task can be posed as a prompt: zero-shot, few-shot and chat use the same interface
  • +Exact, reusable KV caching across tokens, turns and requests
  • +The best-studied scaling behaviour and the largest ecosystem of kernels, serving systems and checkpoints

Avoid when

  • −The input is read only left to right, which is weaker than bidirectional encoding for some understanding tasks at equal size
  • −Prompt and output share one KV cache, so long inputs make the cache large
  • −Pure next-token pre-training gives a base model that continues text rather than following instructions; SFT and preference tuning are needed
  • −Generation is sequential, one forward step per output token

Where it's used

Chat assistants

Instruction-tuned decoder-only models behind most assistants; the conversation is the prefix.

Code models

Code completion and repair, where left-to-right generation matches how code is written; fill-in-the-middle is added as a data transformation.

Multimodal LLMs

Image or audio encoders project into the token stream of a decoder-only backbone, sometimes with a prefix-LM mask over the non-text tokens.

Check yourself

A chatbot keeps the full conversation as context. Why can a causal decoder reuse its KV cache from turn to turn while a prefix LM with a bidirectional prefix cannot?

Check yourself

Over one 4,096-token sequence, how many loss terms do a causal LM and BERT-style 15% masked LM get?

Remember

  • ✓Decoder-only models train one causal stack on −∑tlog⁡p(xt+1∣x≤t)-\sum_t\log p(x_{t+1}\mid x_{\le t}): T−1T-1 targets per sequence, against about 0.15T0.15T for masked LM and T−PT-P for a prefix LM.
  • ✓They won through simplicity (one stack, one loss, raw text), maximum training signal, prompting and in-context learning as a universal interface, and exact KV caching.
  • ✓Causal masking keeps earlier keys and values fixed. A prefix LM must re-encode its prefix as it grows: about (K+1)/2(K+1)/2 times more work over KK chat turns.
  • ✓Wang et al. (2022): a causal decoder is best zero-shot after pre-training alone; non-causal plus denoising plus multitask finetuning is best after adaptation.
  • ✓The families (GPT, PaLM, Llama, Mistral, Qwen, DeepSeek) share the stack; they differ in tokens (300B to 18T), vocabulary (32K to 256K), attention (MHA, MQA, GQA, MLA) and MoE.
← back to Deep Learning roadmap