# Decoder-only models

> Decoder-only models (GPT, PaLM, Llama, Mistral, Qwen, DeepSeek) train one causal stack on next-token prediction. Every token is a target (T − 1 per sequence vs ~15% for masked LM), prompts double as task specifications, and the causal mask makes the KV cache exact and reusable.

Source: https://www.mlfoundry.in/learn/decoder-only-models  
Section: Deep Learning › Large Language Models (LLMs)  
Level: core · 14 min read · updated 2026-10-13

**Decoder-only model:** is a language model family in which a single stack of causally masked transformer blocks is pre-trained on raw text to predict each next token, and every task is then posed as continuing a text prompt. The line starts with OpenAI's GPT (Radford, Narasimhan, Salimans and Sutskever, "Improving Language Understanding by Generative Pre-Training", 2018). GPT-3 (Brown et al., "Language Models are Few-Shot Learners", NeurIPS 2020) made it dominant by showing that one large model can do new tasks from a few examples in the prompt. PaLM, Llama, Mistral, Qwen, Gemma and DeepSeek all follow it. The block-level maths (masked attention, parameter and FLOP counts) is in **Decoder-only transformer**. This article covers the family: why it won, its variants, and how the main models differ.

In 2018–2020 three designs competed for pre-training. **Encoder-only models** such as **BERT** read text bidirectionally and are trained to fill in masked tokens. They are strong at classification and retrieval but cannot generate text natively. **Encoder–decoder models** such as **T5** and **BART** read an input with one stack and write an output with another. Decoder-only models read and write with the same causal stack. The T5 paper's controlled comparison (Raffel et al., 2020) favoured encoder–decoders with a denoising objective for fine-tuned transfer. Yet almost every frontier model since 2020 is decoder-only. The reasons are mostly about scale and use, not about any single benchmark.

## Three objectives, one sequence

$$
\mathcal L_{\text{CLM}}=-\sum_{t=1}^{T-1}\log p(x_{t+1}\mid x_{\le t}),\qquad \mathcal L_{\text{PLM}}=-\sum_{t=P}^{T-1}\log p(x_{t+1}\mid x_{\le t}),\qquad \mathcal L_{\text{MLM}}=-\sum_{t\in\mathcal M}\log p(x_t\mid x_{\setminus\mathcal M})
$$

- $\text{CLM}$: causal LM (GPT, Llama): position $t$ sees $x_{\le t}$; all $T-1$ positions are targets
- $\text{PLM}$: prefix LM: the first $P$ tokens attend to each other in both directions; only the $T-P$ continuation tokens are targets
- $\text{MLM}$: masked LM (BERT): full attention; only the masked set $\mathcal M$, about 15% of positions, is predicted
- $P$: prefix length: the conditioning part of the sequence (the input, in encoder–decoder terms)

$$
M^{\text{CLM}}_{ij}=\mathbb 1[j\le i],\qquad M^{\text{PLM}}_{ij}=\mathbb 1[j\le i\ \lor\ (i\le P\land j\le P)],\qquad M^{\text{MLM}}_{ij}=1
$$

- $M_{ij}$: 1 if query position $i$ may attend to key position $j$
- $\sum_{ij}M^{\text{CLM}}_{ij}$: $T(T+1)/2$ allowed pairs; the prefix LM adds the upper triangle of the $P\times P$ block

The three models can share the same weights and differ only in the mask and the set of loss positions. That makes the comparison clean. The causal mask is the only one under which every position is both an input and a target, and under which no position's representation depends on anything after it.

> Figure: The decoder-only lineage. GPT, GPT-2 and GPT-3 established the recipe between 2018 and 2020: scale one causal stack and prompt it. From 2022 Google, Meta, Mistral AI, Alibaba and DeepSeek built their own families on it, with sizes shown as published. The bottom row shows what changed inside the block around 2023. The stack, the mask and the loss stayed the same.

## Why decoder-only won

**Every token is a training signal.** Under the causal mask one forward pass over $T$ tokens yields $T-1$ next-token losses. BERT's masked LM predicts only the 15% of positions it hides, and ELECTRA (Clark, Luong, Le and Manning, ICLR 2020) was motivated by this waste. A prefix LM gives up the targets inside its prefix. With a fixed budget of tokens and FLOPs, the causal objective extracts the most supervision. It also needs no corruption hyperparameters: no mask rate, no span lengths, no sentinels.

**Simplicity.** There is one stack, one loss and one data format. Any document is a training example as it stands, with no input/output split. Training and inference run the same computation: inference is training's forward pass with sampling appended. The same code, kernels and parallelism serve every size from 125M to 405B. Scaling-law work (Kaplan et al., 2020; Hoffmann et al., 2022) was done on decoder-only models, so their behaviour at scale is the best mapped.

**In-context learning.** A decoder-only model can be given a task by writing it into the prefix: an instruction, a few input–output examples, and then the query. GPT-3 showed that this few-shot prompting improves sharply with scale, with no gradient updates and no task-specific head. Because the model is trained to continue arbitrary text, any task that can be phrased as text fits the same interface. Instruction tuning and **RLHF** later turned this interface into chat assistants (see **Supervised fine-tuning (SFT)**, **Instruction tuning**).

$$
p\big(y\mid \underbrace{(x_1,y_1),\dots,(x_k,y_k)}_{\text{demonstrations}},\,x\big)=\prod_{t}p\big(y_t\mid \text{demos},\,x,\,y_{<t}\big)
$$

- $(x_i,y_i)$: worked examples written into the prompt; the weights stay fixed
- $k$: $k=0$ is zero-shot, $k=1$ one-shot; GPT-3 used up to as many as fit in its 2,048-token window

**KV caching is exact.** With a causal mask, appending a token never changes the keys and values of earlier tokens. A server can therefore compute a prompt once, keep its **KV cache**, and pay one forward step per new token. A system prompt shared by many users can be cached once, and a chat history can be extended turn by turn without recomputation. Under a bidirectional or prefix-LM mask, a longer prefix changes every earlier representation, so the prefix must be re-encoded. Over a $K$-turn conversation this costs about $(K+1)/2$ times more:

$$
\underbrace{K\,(u+a)}_{\text{causal + KV cache}}\qquad\text{vs}\qquad \underbrace{\sum_{k=1}^{K}\big[(k-1)(u+a)+u+a\big]=(u+a)\,\frac{K(K+1)}{2}}_{\text{prefix LM, history re-encoded each turn}}
$$

- $u,\ a$: user and assistant tokens per turn
- $K$: number of turns; the ratio grows linearly in K

Wang, Roberts, Hesslow, Le Scao, Chung, Beltagy, Launay and Raffel tested the architecture question directly ("What Language Model Architecture and Pretraining Objective Work Best for Zero-Shot Generalization?", ICML 2022). They trained causal decoders, non-causal (prefix) decoders and encoder–decoders with over 5B parameters on more than 170B tokens. A causal decoder trained with the plain LM objective had the best zero-shot performance straight after pre-training. A non-causal model trained with masked-LM-style denoising and then multitask-finetuned was best overall. Pre-trained causal decoders could also be adapted cheaply into non-causal ones. So the decoder-only choice is not a free win on every metric. It is the best default when the model must be used as-is, by prompting, across open-ended tasks.

> Figure: One training sequence under each objective. The grid is the attention mask (row i = query, column j = key), the green dots mark positions that carry a loss term, and the chart counts the work over a chat. At the default T = 16 the causal LM supervises 15 of 16 positions (93.8%) with 136 of 256 query–key pairs. The prefix LM with P = 6 supervises 10 (62.5%) and allows 151 pairs, because the 6 × 6 prefix block is full. The masked LM sees all 256 pairs but predicts only 2 masked tokens (12.5%). At T = 32 it masks 5 (15.6%) against the causal LM's 31 (96.9%). The chat panel assumes 50 user and 200 reply tokens per turn. With a KV cache, a causal model processes 2,000 tokens over 8 turns. A prefix LM that re-encodes its growing bidirectional history processes 9,000 (4.5×). At 20 turns the totals are 5,000 and 52,500 (10.5×).

## Prefix-LM and other variants

The prefix LM is the main hybrid: bidirectional attention over the input, causal over the output, all in one stack. UniLM (Dong et al., "Unified Language Model Pre-training for Natural Language Understanding and Generation", NeurIPS 2019) trained one transformer under bidirectional, left-to-right and sequence-to-sequence masks. The T5 paper included a prefix LM in its architecture comparison. UL2 (Tay et al., "UL2: Unifying Language Learning Paradigms", ICLR 2023) mixed several denoisers, including a prefix-LM "S-denoiser", and trained a 20B model. GLM (Du et al., ACL 2022) used autoregressive blank infilling: spans are cut out of a bidirectionally encoded context and regenerated left to right. Vision-language models such as PaliGemma (Beyer et al., 2024) use a prefix-LM mask over the image tokens and the prompt.

In practice the prefix idea mostly survives as a **loss mask**, not an attention mask. Supervised fine-tuning of a chat model keeps causal attention but drops the prompt tokens from the loss (label $-100$ in Hugging Face). This keeps exact KV caching and trains only on the response. Two other variants change the block rather than the mask. Mixture-of-experts decoders (Mixtral, DeepSeek-V3) route each token through a few expert MLPs, which decouples the parameter count from the per-token compute (see **Mixture of experts (MoE)**). Sliding-window attention (Mistral 7B) bounds the cache size.

## The families

Main decoder-only families, as published by their authors (tokens = pre-training tokens)

| model | year | sizes | tokens | context | attention | vocab |
| --- | --- | --- | --- | --- | --- | --- |
| GPT-3 | 2020 | 125M–175B | 300B | 2,048 | MHA, dense + banded sparse layers | 50,257 |
| PaLM | 2022 | 8B, 62B, 540B | 780B | 2,048 | multi-query, parallel blocks | 256K |
| Llama | 2023 | 7B–65B | 1.0T–1.4T | 2,048 | MHA | 32K |
| Llama 2 | 2023 | 7B, 13B, 70B | 2T | 4,096 | GQA in 70B | 32K |
| Llama 3.1 | 2024 | 8B, 70B, 405B | ~15T | 128K | GQA, 8 KV heads | 128K |
| Mistral 7B | 2023 | 7.3B | not disclosed | 8,192 | GQA + sliding window 4,096 | 32K |
| Mixtral 8×7B | 2024 | 47B total, 13B active | not disclosed | 32K | GQA, top-2 of 8 expert MLPs | 32K |
| Qwen2.5 | 2024 | 0.5B–72B | 18T | 32K–128K | GQA, biases on q, k, v | ~152K |
| DeepSeek-V3 | 2024 | 671B total, 37B active | 14.8T | 128K | multi-head latent attention + MoE | 128K |

Read the table by columns. Token counts grew 60-fold in four years (300B to 18T), much faster than parameters. A 7B or 8B model trained on 2T–15T tokens is far past the compute-optimal ratio of about 20 tokens per parameter (see **Pretraining and continued pretraining**). This is deliberate: a smaller, longer-trained model is cheaper to serve. Vocabularies grew from 32K to 128K–256K to cover more languages and code with fewer tokens. Attention moved from full multi-head to grouped-query and latent variants, all to shrink the KV cache. The families themselves are covered in **GPT architecture**, **Llama**, **Mistral**, **Qwen** and **Phi**.

**Configurations from papers and model cards**

- **GPT-3 175B:** 96 layers, $d=12288$, 96 heads of 128, context 2,048, 300B tokens, batch 3.2M tokens, peak lr $0.6\times10^{-4}$
- **PaLM 540B:** 118 layers, $d=18432$, 48 heads, multi-query attention, SwiGLU, attention and MLP in parallel, RoPE, no biases, 256K SentencePiece vocabulary, 780B tokens on 6,144 TPU v4 chips
- **Llama 3 8B:** 32 layers, $d=4096$, 32 query heads, 8 KV heads, $d_{ff}=14336$, vocabulary 128,256, RoPE base 500,000, 15T+ tokens
- **Llama 3.1 405B:** 126 layers, $d=16384$, 128 query heads, 8 KV heads, $d_{ff}=53248$, context 128K, trained with about $3.8\times10^{25}$ FLOPs
- **Mistral 7B:** 32 layers, $d=4096$, 32 query heads, 8 KV heads, $d_{ff}=14336$, sliding window 4,096, rolling-buffer KV cache, vocabulary 32,000
- **Qwen2.5 7B:** 28 layers, $d=3584$, 28 query heads, 4 KV heads, $d_{ff}=18944$, RoPE base $10^6$, biases on the q, k, v projections

## Emergent abilities and how to read them

Wei et al. ("Emergent Abilities of Large Language Models", TMLR 2022) collected tasks on which decoder-only models score near chance up to some scale and then jump: multi-step arithmetic, some BIG-Bench tasks, chain-of-thought gains. Schaeffer, Miranda and Koyejo ("Are Emergent Abilities of Large Language Models a Mirage?", NeurIPS 2023) argued that many jumps come from the metric. If per-token accuracy $p$ improves smoothly, exact match on a $k$-token answer scores $p^k$, which stays near 0 and then rises steeply. With per-token or log-likelihood metrics the same models improve smoothly. Both views agree that capability grows predictably with scale in the underlying loss. They disagree on how sudden the useful behaviour is.

## Worked example: one sequence, three objectives, one chat

Take a sequence of $T=12$ tokens. The prefix LM uses a prefix of $P=4$, and the masked LM masks $\mathrm{round}(0.15\cdot12)=\mathrm{round}(1.8)=2$ positions.

Targets and attention pairs for T = 12, P = 4 (computed in Python)

| objective | targets | share of tokens | allowed pairs | earlier K, V fixed? |
| --- | --- | --- | --- | --- |
| causal LM | 11 | 91.7% | 78 | yes |
| prefix LM, P = 4 | 8 | 66.7% | 84 | only after the prefix |
| masked LM, 15% | 2 | 16.7% | 144 | no |

**Counting by hand**

1. Causal: positions 1–11 each predict the next token, so 11 targets. Allowed pairs $=\sum_{i=1}^{12}i=12\cdot13/2=78$.
2. Prefix LM: positions 4–11 predict $x_5\dots x_{12}$, so $12-4=8$ targets. Pairs $=4^2+\sum_{i=5}^{12}i=16+68=84$. The 6 extra pairs over causal are the upper triangle of the $4\times4$ prefix block.
3. Masked LM: 2 targets, and every token sees every token: $12^2=144$ pairs. For a 4,096-token sequence the gap is 4,095 targets against $\mathrm{round}(614.4)=614$.
4. Chat with $K=8$ turns of $u=50$ user and $a=200$ reply tokens. Causal with a KV cache processes each token once: $8\cdot250=2{,}000$ tokens.
5. Prefix LM: turn $k$ re-encodes $(k-1)\cdot250+50$ prefix tokens and generates 200, which is $250k$ per turn. In total $250\cdot(1+\dots+8)=250\cdot36=9{,}000$ tokens, $4.5\times$ the causal model, matching $(K+1)/2$.

```python
import torch, torch.nn.functional as F
from transformers import AutoModelForCausalLM, AutoTokenizer

name = "Qwen/Qwen2.5-0.5B"                      # any decoder-only checkpoint
tok = AutoTokenizer.from_pretrained(name)
model = AutoModelForCausalLM.from_pretrained(name, torch_dtype=torch.float32).eval()

# 1) Pre-training loss. labels = inputs; the model shifts by one internally,
#    so a sequence of T tokens contributes T - 1 targets.
ids = tok("Decoder-only models predict the next token.", return_tensors="pt").input_ids
with torch.no_grad():
    out = model(input_ids=ids, labels=ids)
    logits = out.logits[:, :-1]                 # position t predicts token t + 1
    manual = F.cross_entropy(logits.reshape(-1, logits.size(-1)), ids[:, 1:].reshape(-1))
print(ids.shape[1] - 1, out.loss.item(), manual.item())   # T - 1 targets, the same loss twice

# 2) Answer-only loss (how SFT is usually done): attention stays causal,
#    prompt positions get label -100 and drop out of the cross-entropy.
p_ids = tok("Q: What is 7 * 6?\nA:", return_tensors="pt").input_ids
a_ids = tok(" 42", return_tensors="pt").input_ids
ids2 = torch.cat([p_ids, a_ids], dim=1)
labels = ids2.clone()
labels[:, : p_ids.shape[1]] = -100
with torch.no_grad():
    answer_loss = model(input_ids=ids2, labels=labels).loss

# 3) The causal mask makes caching exact: encode the prefix once, then step one token.
with torch.no_grad():
    full = model(input_ids=ids2).logits[:, -1]
    pre = model(input_ids=ids2[:, :-1], use_cache=True)
    step = model(input_ids=ids2[:, -1:], past_key_values=pre.past_key_values).logits[:, -1]
print(torch.allclose(full, step, atol=1e-4))    # should be True: earlier keys/values never change

# 4) A prefix-LM mask (UniLM / UL2 style): bidirectional inside the first P tokens.
def prefix_lm_mask(T: int, P: int) -> torch.Tensor:
    m = torch.tril(torch.ones(T, T, dtype=torch.bool))
    m[:P, :P] = True                            # True = may attend
    return m

print(prefix_lm_mask(6, 3).int())
# [[1,1,1,0,0,0], [1,1,1,0,0,0], [1,1,1,0,0,0], [1,1,1,1,0,0], [1,1,1,1,1,0], [1,1,1,1,1,1]]
q = k = v = torch.randn(1, 4, 6, 16)            # (batch, heads, T, d_head)
y = F.scaled_dot_product_attention(q, k, v, attn_mask=prefix_lm_mask(6, 3))
```

Next-token loss, answer-only loss, exact KV-cache reuse and a prefix-LM mask with Hugging Face transformers and PyTorch (written for transformers 4.4x; not run here)

> **The mask decides what can be cached:** A KV cache is valid only if old keys and values never depend on new tokens. The causal mask guarantees this, which is why prompt caching, shared system prompts, paged KV blocks and **Speculative decoding** are all straightforward for decoder-only models. A bidirectional prefix gives better input representations but turns every extension of the prefix into a recomputation.

> **Over-trained is not mis-trained:** Llama 3 8B saw about 15T tokens, roughly 1,900 per parameter, nearly 100 times the compute-optimal ratio. Calling this wasteful misreads the goal. Chinchilla optimises loss per training FLOP, but a deployed model is paid for per inference token. A smaller model trained longer gives a better model at a fixed serving cost.

> **Decoder-only does not mean no conditioning:** Translation, summarisation and question answering all work in a decoder-only model: the source is simply the prefix. What it gives up compared with **Encoder–decoder models** is bidirectional attention over that source and a separate set of weights for reading versus writing. At scale and with instruction tuning, the gap has mostly closed for text. Encoder–decoders still compete where the input is long, fixed and in a different modality, as in speech recognition.

**Strengths**

- Every token of every document is a target: maximum supervision per FLOP
- One stack, one loss, one data format; training and inference share the same computation
- Any task can be posed as a prompt: zero-shot, few-shot and chat use the same interface
- Exact, reusable KV caching across tokens, turns and requests
- The best-studied scaling behaviour and the largest ecosystem of kernels, serving systems and checkpoints

**Limitations**

- The input is read only left to right, which is weaker than bidirectional encoding for some understanding tasks at equal size
- Prompt and output share one KV cache, so long inputs make the cache large
- Pure next-token pre-training gives a base model that continues text rather than following instructions; SFT and preference tuning are needed
- Generation is sequential, one forward step per output token

- **Chat assistants:** Instruction-tuned decoder-only models behind most assistants; the conversation is the prefix.
- **Code models:** Code completion and repair, where left-to-right generation matches how code is written; fill-in-the-middle is added as a data transformation.
- **Multimodal LLMs:** Image or audio encoders project into the token stream of a decoder-only backbone, sometimes with a prefix-LM mask over the non-text tokens.

**Question:** A chatbot keeps the full conversation as context. Why can a causal decoder reuse its KV cache from turn to turn while a prefix LM with a bidirectional prefix cannot?

- A. Prefix LMs do not use keys and values
- B. Under a causal mask earlier tokens never attend to later ones, so their keys and values are unchanged when text is appended; in a bidirectional prefix every token attends to the new tokens
- C. Causal models have fewer layers
- D. The KV cache stores logits, which the prefix LM recomputes

**Answer:** B. Under a causal mask earlier tokens never attend to later ones, so their keys and values are unchanged when text is appended; in a bidirectional prefix every token attends to the new tokens — Keys and values at layer $\ell>1$ are functions of the previous layer's outputs. If position $j$ can attend to a new token, all of its higher-layer keys and values change. Over $K$ turns the re-encoding costs about $(K+1)/2$ times the causal total: $4.5\times$ for 8 turns.

**Question:** Over one 4,096-token sequence, how many loss terms do a causal LM and BERT-style 15% masked LM get?

- A. 4,096 and 4,096
- B. 4,095 and about 614
- C. 2,048 and about 614
- D. 614 and 4,095

**Answer:** B. 4,095 and about 614 — The causal LM predicts every next token, which gives $T-1=4{,}095$ targets. The masked LM predicts only the masked positions, $0.15\cdot4096\approx614$. The forward and backward cost per sequence is about the same, so the causal objective gets roughly 6.7 times more targets per FLOP.

**Key points**

- Decoder-only models train one causal stack on $-\sum_t\log p(x_{t+1}\mid x_{\le t})$: $T-1$ targets per sequence, against about $0.15T$ for masked LM and $T-P$ for a prefix LM.
- They won through simplicity (one stack, one loss, raw text), maximum training signal, prompting and in-context learning as a universal interface, and exact KV caching.
- Causal masking keeps earlier keys and values fixed. A prefix LM must re-encode its prefix as it grows: about $(K+1)/2$ times more work over $K$ chat turns.
- Wang et al. (2022): a causal decoder is best zero-shot after pre-training alone; non-causal plus denoising plus multitask finetuning is best after adaptation.
- The families (GPT, PaLM, Llama, Mistral, Qwen, DeepSeek) share the stack; they differ in tokens (300B to 18T), vocabulary (32K to 256K), attention (MHA, MQA, GQA, MLA) and MoE.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
