# Decoder-only transformer

> A decoder-only transformer is a stack of causal self-attention and MLP blocks trained to predict the next token. It has about 12·L·d² parameters, costs about 6N FLOPs per training token, and caches every layer's keys and values so each generated token costs one forward step.

Source: https://www.mlfoundry.in/learn/decoder-only-transformer  
Section: Deep Learning › Attention and Transformer Architectures  
Level: core · 15 min read · updated 2026-10-12

**Decoder-only transformer:** is a language model built from one stack of $L$ identical blocks. Each block has **causal** multi-head self-attention, where position $i$ sees only positions $j\le i$, and a position-wise MLP, both wrapped in residual connections. There is no encoder and no **Cross-attention**. A token embedding feeds the stack, and a linear **LM head** on the last layer turns each position into logits over the vocabulary. Training maximises $\sum_t\log p(x_t\mid x_{<t})$ on raw text. Generation samples one token at a time and appends it to the input. GPT, GPT-2, GPT-3, Llama, Mistral and most current large language models have this shape.

The original Transformer (Vaswani et al., 2017) was an **Encoder–decoder transformer** for translation. Liu et al. ("Generating Wikipedia by Summarizing Long Sequences", ICLR 2018) showed that the decoder alone, fed the source and target as one sequence, works well for long inputs. Radford et al. at OpenAI then made it the standard pre-training architecture. **GPT** ("Improving Language Understanding by Generative Pre-Training", 2018) was 12 layers, $d=768$, 12 heads, a 512-token context and about 117M parameters, pre-trained on BooksCorpus and fine-tuned per task. **GPT-2** ("Language Models are Unsupervised Multitask Learners", 2019) moved layer normalization to the input of each sub-block, added a final layer norm, used a 1024-token context and a 50,257-token byte-level BPE vocabulary, and scaled to 1.5B parameters (48 layers, $d=1600$). **GPT-3** (Brown et al., "Language Models are Few-Shot Learners", NeurIPS 2020) kept the design at 175B parameters: 96 layers, $d=12288$, 96 heads of 128, a 2048-token context and 300B training tokens. Its layers alternate dense and locally banded sparse attention.

Why drop the encoder? Next-token prediction needs no labels, so every token of every document is a training target. One causal stack also serves every task: the "input" is just the prefix. An encoder–decoder spends parameters on two stacks and needs an input/output split. At the same compute, the decoder-only model trains on all positions in parallel under one mask and caches cleanly at inference.

## The block

$$
\begin{aligned}h^{(0)}&=E[x_{1:T}]\;(+\,P_{1:T})\\ a^{(\ell)}&=h^{(\ell-1)}+\mathrm{MHA}_{\text{causal}}\big(\mathrm{Norm}(h^{(\ell-1)})\big)\\ h^{(\ell)}&=a^{(\ell)}+\mathrm{MLP}\big(\mathrm{Norm}(a^{(\ell)})\big)\qquad \ell=1\dots L\\ p(x_{t+1}\mid x_{\le t})&=\mathrm{softmax}\big(W_U\,\mathrm{Norm}(h^{(L)}_t)\big)\end{aligned}
$$

- $E\in\mathbb R^{V\times d}$: token embedding; GPT-2 and GPT-3 **tie** $W_U=E$, Llama keeps a separate output matrix
- $P$: learned absolute positions (GPT-1/2/3); modern models drop $P$ and rotate $q,k$ inside attention instead (see **Rotary positional embeddings (RoPE)**)
- $\mathrm{MHA}_{\text{causal}}$: $H$ heads of **Scaled dot-product attention** with the mask $j\le i$ (see **Causal attention**, **Multi-head attention**)
- $\mathrm{Norm}$: applied **before** each sub-layer (pre-norm): LayerNorm in GPT-2/3, RMSNorm in Llama
- $\mathrm{MLP}$: $d\to d_{ff}\to d$ per position: GELU with $d_{ff}=4d$ (GPT), or SwiGLU $W_2(\mathrm{SiLU}(W_1x)\odot W_3x)$ with $d_{ff}\approx\tfrac{8}{3}d$ (Llama)
- $h^{(L)}_t$: the last layer at position $t$ predicts token $t+1$; all $T$ predictions come out of one forward pass

Training uses **teacher forcing** for free. Shift the sequence by one, $\text{input}=x_{1:T-1}$ and $\text{target}=x_{2:T}$, and take the cross-entropy at every position. The causal mask guarantees that the prediction at $t$ never saw $x_{t+1}$. One sequence of 4096 tokens therefore gives 4095 training targets in a single pass, which is why decoder-only pre-training is so sample-efficient per FLOP.

## Counting parameters and FLOPs

$$
N\approx \underbrace{4d^2}_{W_Q,W_K,W_V,W_O}\cdot L+\underbrace{2\,d\,d_{ff}}_{\text{MLP}}\cdot L\;\overset{d_{ff}=4d}{=}\;12\,L\,d^2,\qquad C_{\text{train}}\approx 6N\ \text{FLOPs per token}\;+\;6\,L\,T\,d
$$

- $12Ld^2$: non-embedding parameters (Kaplan et al., "Scaling Laws for Neural Language Models", 2020); biases and norms add only $O(Ld)$
- $Vd$: the embedding (and an untied head) comes on top: 31.6% of GPT-2 small's 124M, 0.4% of GPT-3
- $6N$: a weight is used in one multiply-add (2 FLOPs) per token forward and two in the backward pass (gradient w.r.t. input and w.r.t. weight): $2+4=6$
- $6LTd$: the $QK^\top$ and $AV$ products at context $T$ (forward $2LTd$ per token in Kaplan et al.), small unless $T$ is comparable to $2d$
- $C\approx6ND$: total training compute for $D$ tokens, the budget used by scaling laws

The rule is accurate for GPT-style models: $12Ld^2$ is 99.9% of GPT-2 small's non-embedding weights. Modern recipes move it a little. SwiGLU has three matrices, so with $d_{ff}=\tfrac83d$ it keeps $8d^2$ per layer, but Llama 2 7B rounds $d_{ff}$ up to 11008 ($2.69d$), giving $8.06d^2$. **Grouped-query attention (GQA)** shrinks $W_K,W_V$: Llama 3 8B has 8 KV heads for 32 query heads and $d_{ff}=14336$, so a layer is $2.5d^2+10.5d^2=13d^2$, and $12Ld^2$ covers 92.3% of its non-embedding weights. The 128,256-token vocabulary, untied, adds another 1.05B parameters.

**The modern recipe (Llama family, most open models since 2023)**

- **Normalization:** pre-norm **RMSNorm** (Zhang and Sennrich, 2019): $x/\sqrt{\overline{x^2}+\epsilon}\cdot g$, no mean subtraction, no bias; one more at the end of the stack
- **Positions:** **Rotary positional embeddings (RoPE)** (Su et al., 2021) on $q$ and $k$ in every layer; no position table
- **MLP:** **SwiGLU** (Shazeer, "GLU Variants Improve Transformer", 2020), $d_{ff}\approx\tfrac83d$ so the parameter count matches a $4d$ GELU MLP
- **Attention:** **Grouped-query attention (GQA)** (Ainslie et al., 2023): Llama 2 34B/70B and all Llama 3 sizes use 8 KV heads
- **Linear layers:** no biases
- **Llama 2 7B:** $L=32$, $d=4096$, 32 heads, $d_{ff}=11008$, $V=32000$, context 4096, 6,738,415,616 parameters, 2T training tokens
- **Llama 2 70B:** $L=80$, $d=8192$, 64 query heads, 8 KV heads, $d_{ff}=28672$, 68,976,648,192 parameters

## Inference: prefill, decode and the KV cache

Generation has two phases. **Prefill** runs the whole prompt through the stack in one causal pass, exactly like training. **Decode** then produces one token per step. Because of the mask, earlier positions never change, so each layer stores the keys and values it has already computed (the **KV cache**, see **Causal attention**). A decode step only computes $q,k,v$ for the new token, appends $k,v$ to the cache, and attends over it. Each step costs about $2N$ FLOPs plus $4LTd$ for attention, but it must read every weight and the whole cache from memory. At small batch, decode speed is therefore set by memory bandwidth, not arithmetic.

$$
\text{KV bytes}=2\cdot L\cdot n_{kv}\cdot d_h\cdot T\cdot B\cdot b
$$

- $2$: one key and one value vector
- $n_{kv},\,d_h$: key/value heads and head size; $n_{kv}=H$ for MHA, $1$ for **Multi-query attention (MQA)**, $H/g$ for GQA
- $T,\,B$: tokens in context and sequences in the batch
- $b$: bytes per number, 2 for fp16/bf16

> Figure: The left panel draws one pre-norm block of the selected model, and the right panel counts it from its real configuration. Llama 2 7B gives 6,738,415,616 parameters. 12·L·d² = 6.44B is 99.5% of the 6.48B non-embedding weights. The MLP is 64.2% of all weights, attention 31.9% and the untied embeddings 3.9%. At T = 4096 a training token costs 6N = 3.89e10 FLOPs plus 3.22e9 for attention (7.7%); at T = 32K the attention share reaches 39.9%. Its KV cache is 512 KiB per token, 2.00 GiB at 4096 tokens, against 12.55 GiB of bf16 weights. The bold curve crosses the weights line near 25K tokens, or already at 4096 tokens with batch 8 (16 GiB). Llama 2 70B with its 8 KV heads needs 320 KiB per token (1.25 GiB at 4096). Switch it to MHA and the cache becomes 10 GiB and the model 78.37B parameters. GPT-2 small has 124,439,808 parameters, 31.6% of them in the embedding. GPT-3 needs 4.5 MiB of cache per token, 9 GiB for one 2048-token sequence.

## Worked example: GPT-2 small, counted by hand

GPT-2 small has $L=12$, $d=768$, $H=12$, $d_{ff}=3072$, $V=50257$ and 1024 learned positions. Every linear layer has a bias, LayerNorm has a gain and a bias, and the LM head is tied to $E$.

Exact parameter count (checked in Node against the 124,439,808 of the released checkpoint)

| component | formula | parameters |
| --- | --- | --- |
| token embedding wte | V·d = 50257·768 | 38,597,376 |
| position embedding wpe | 1024·d | 786,432 |
| attention, per layer | d·3d + 3d + d·d + d | 2,362,368 |
| MLP, per layer | d·4d + 4d + 4d·d + d | 4,722,432 |
| 2 LayerNorms, per layer | 2·2d | 3,072 |
| one layer | 12d² + 13d | 7,087,872 |
| 12 layers |  | 85,054,464 |
| final LayerNorm | 2d | 1,536 |
| total (head tied) |  | 124,439,808 |

**From parameters to compute and memory**

1. $12Ld^2=12\cdot12\cdot768^2=84{,}934{,}656$, against $85{,}056{,}000$ actual non-embedding weights (99.9%). The embeddings are $39{,}383{,}808$, 31.6% of the model, which is why small models are often quoted "without embeddings".
2. Training FLOPs per token at $T=1024$: $6N=6\cdot85{,}056{,}000=5.10\times10^8$, plus $6LTd=6\cdot12\cdot1024\cdot768=5.66\times10^7$ for attention, 10.0% of the total.
3. Scale up: Llama 2 7B trained on $D=2\times10^{12}$ tokens costs $6ND=6\cdot6.738\times10^9\cdot2\times10^{12}\approx8.1\times10^{22}$ FLOPs.
4. KV cache of GPT-2 small in fp16: $2\cdot12\cdot12\cdot64\cdot2=36{,}864$ bytes $=36$ KiB per token, 36 MiB for a full 1024-token context.
5. Llama 2 70B with GQA: $2\cdot80\cdot8\cdot128\cdot2=327{,}680$ bytes $=320$ KiB per token. With 64 KV heads it would be $2.5$ MiB, 8 times more.

```python
import torch, torch.nn as nn, torch.nn.functional as F

class RMSNorm(nn.Module):
    def __init__(self, d, eps=1e-5):
        super().__init__()
        self.eps, self.w = eps, nn.Parameter(torch.ones(d))
    def forward(self, x):
        return self.w * x * torch.rsqrt(x.pow(2).mean(-1, keepdim=True) + self.eps)

def rope(x, pos, base=10000.0):
    # x (B, h, T, dh), pos (T,) absolute positions; rotates the pairs (x[i], x[i + dh/2])
    dh = x.size(-1)
    inv = base ** (-torch.arange(0, dh, 2, device=x.device).float() / dh)  # (dh/2,)
    ang = pos.float()[:, None] * inv[None, :]                               # (T, dh/2)
    cos, sin = ang.cos(), ang.sin()
    x1, x2 = x[..., : dh // 2], x[..., dh // 2 :]
    return torch.cat([x1 * cos - x2 * sin, x1 * sin + x2 * cos], -1)

class Block(nn.Module):
    def __init__(self, d, H, kv, dff):
        super().__init__()
        self.H, self.kv, self.dh = H, kv, d // H
        self.n1, self.n2 = RMSNorm(d), RMSNorm(d)
        self.wq = nn.Linear(d, d, bias=False)
        self.wkv = nn.Linear(d, 2 * kv * self.dh, bias=False)  # GQA: only kv heads of K and V
        self.wo = nn.Linear(d, d, bias=False)
        self.w13 = nn.Linear(d, 2 * dff, bias=False)           # SwiGLU gate and up projections
        self.w2 = nn.Linear(dff, d, bias=False)

    def forward(self, x, pos, cache=None):
        B, T, d = x.shape
        h = self.n1(x)                                                            # pre-norm
        q = self.wq(h).view(B, T, self.H, self.dh).transpose(1, 2)               # (B, H, T, dh)
        k, v = self.wkv(h).view(B, T, 2, self.kv, self.dh).permute(2, 0, 3, 1, 4)  # each (B, kv, T, dh)
        q, k = rope(q, pos), rope(k, pos)
        if cache is not None:                                                     # KV cache
            if cache["k"] is not None:
                k, v = torch.cat([cache["k"], k], 2), torch.cat([cache["v"], v], 2)
            cache["k"], cache["v"] = k, v
        k = k.repeat_interleave(self.H // self.kv, 1)                             # share each K/V head
        v = v.repeat_interleave(self.H // self.kv, 1)
        mask = torch.arange(k.size(2), device=x.device)[None, :] <= pos[:, None]  # (T, S): key j <= query i
        a = F.scaled_dot_product_attention(q, k, v, attn_mask=mask)
        x = x + self.wo(a.transpose(1, 2).reshape(B, T, d))
        g, u = self.w13(self.n2(x)).chunk(2, -1)
        return x + self.w2(F.silu(g) * u)

class DecoderLM(nn.Module):
    def __init__(self, V, d=512, L=8, H=8, kv=2, dff=1408):
        super().__init__()
        self.emb = nn.Embedding(V, d)
        self.blocks = nn.ModuleList(Block(d, H, kv, dff) for _ in range(L))
        self.norm = RMSNorm(d)
        self.head = nn.Linear(d, V, bias=False)                                  # untied, as in Llama

    def forward(self, idx, start=0, caches=None):
        pos = torch.arange(start, start + idx.size(1), device=idx.device)
        x = self.emb(idx)
        for i, blk in enumerate(self.blocks):
            x = blk(x, pos, None if caches is None else caches[i])
        return self.head(self.norm(x))                                            # logits for the NEXT token

V = 1000
model = DecoderLM(V)
idx = torch.randint(0, V, (2, 16))
logits = model(idx[:, :-1])                                                       # inputs x_1..x_15
loss = F.cross_entropy(logits.reshape(-1, V), idx[:, 1:].reshape(-1))            # targets x_2..x_16
print(logits.shape)                                     # torch.Size([2, 15, 1000])
print(sum(p.numel() for p in model.parameters()))       # 23577088 (12*L*d^2 would say 25165824)

@torch.no_grad()
def generate(model, prompt, steps):
    caches = [{"k": None, "v": None} for _ in model.blocks]
    logits = model(prompt, 0, caches)                   # prefill: the whole prompt in one pass
    out, t = prompt, prompt.size(1)
    for _ in range(steps):
        nxt = logits[:, -1].argmax(-1, keepdim=True)     # greedy next token
        out = torch.cat([out, nxt], 1)
        logits = model(nxt, t, caches)                   # decode: one token, cached K/V reused
        t += 1
    return out, caches

out, caches = generate(model.eval(), idx[:, :4], 6)
print(out.shape, caches[0]["k"].shape)                  # torch.Size([2, 10]) torch.Size([2, 2, 10, 64])
```

A Llama-style decoder-only LM (pre-RMSNorm, RoPE, SwiGLU, GQA) with next-token loss and KV-cached generation

> **Off-by-one between logits and targets:** The output at position $t$ predicts $x_{t+1}$. Feeding $x_{1:T}$ and scoring against $x_{1:T}$ trains the model to copy its input, and the loss drops to near zero without learning anything. Shift once: inputs $x_{1:T-1}$, targets $x_{2:T}$. With padding, also mask the loss at padded targets and keep left-padded prompts consistent with the positions you pass to RoPE.

> **Why 6N is the number to remember:** Per token, the weights dominate compute until $T$ approaches $2d$: the attention term $6LTd$ divided by $6\cdot12Ld^2$ is $T/12d$. For Llama 2 7B at its 4096 context that is 8.3% by this estimate (7.7% with the exact $N$). Multiply $6N$ by the token count and you have the training budget. Scaling-law work (Kaplan et al., 2020; Hoffmann et al.'s Chinchilla, 2022) is phrased in exactly these units, $C\approx6ND$.

> **Pre-norm and the final norm:** GPT-2 moved LayerNorm from after each residual addition (post-norm, as in Vaswani et al.) to the input of each sub-block. The residual stream then carries an un-normalised sum of all layer outputs, so a **final** norm before the LM head is required. Pre-norm trains stably at depth without long learning-rate warm-up (Xiong et al., 2020), which is why every large decoder-only model uses it (see **Transformer decoder**).

**Strengths**

- One stack, one objective: every token of unlabelled text is a training target
- Any task becomes a prompt; few-shot and instruction following come from the same model
- Training is fully parallel over positions under one causal mask
- Clean incremental inference: the KV cache makes each new token one forward step

**Limitations**

- Generation is sequential, one token per forward pass, and memory-bandwidth bound at small batch
- The KV cache grows with T·B and can exceed the weights for long contexts or large batches
- Attention cost is still quadratic in context for prefill and training
- Causal attention gives each prefix token a one-sided view, which encoders avoid for classification or retrieval embeddings

**Question:** A GPT-style model has L = 24 layers, d = 2048, d_ff = 4d and standard multi-head attention. About how many non-embedding parameters does it have?

- A. 0.30B
- B. 1.21B
- C. 4.83B
- D. 0.10B

**Answer:** B. 1.21B — $12Ld^2=12\cdot24\cdot2048^2=1{,}207{,}959{,}552\approx1.21$B: $4d^2$ for $W_Q,W_K,W_V,W_O$ and $8d^2$ for the two MLP matrices in each layer. Biases and norms add only $O(Ld)$.

**Question:** Llama 2 70B uses 64 query heads but only 8 key/value heads (d_h = 128, L = 80). How large is its bf16 KV cache per token?

- A. 2.5 MiB
- B. 40 KiB
- C. 320 KiB
- D. 20 MiB

**Answer:** C. 320 KiB — $2\cdot L\cdot n_{kv}\cdot d_h\cdot2=2\cdot80\cdot8\cdot128\cdot2=327{,}680$ bytes $=320$ KiB. With 64 KV heads (plain MHA) it would be 2.5 MiB. Query heads do not enter the formula, because queries are never cached.

**Key points**

- Non-embedding parameters $N\approx12Ld^2$ ($4d^2$ attention + $8d^2$ MLP per layer); training costs $\approx6N$ FLOPs per token, so $C\approx6ND$.
- Decoder-only = causal self-attention + MLP blocks, no cross-attention; the output at $t$ predicts $x_{t+1}$, trained with cross-entropy on all positions at once.
- KV cache $=2\cdot L\cdot n_{kv}\cdot d_h\cdot T\cdot B\cdot b$ bytes: 512 KiB per token for Llama 2 7B, 320 KiB for Llama 2 70B with 8 KV heads.
- Modern recipe: pre-norm RMSNorm, RoPE, SwiGLU (d_ff ≈ 8d/3), grouped-query attention, no biases, untied LM head.
- Prefill is compute-bound and parallel; decode is one token per step and limited by reading weights plus cache from memory.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
