Decoder-only transformer

A decoder-only transformer is a stack of causal self-attention and MLP blocks trained to predict the next token. It has about 12·L·d² parameters, costs about 6N FLOPs per training token, and caches every layer's keys and values so each generated token costs one forward step.

15 min read999 wordscore0 viewsupdated 2026-10-12

Definition

Decoder-only transformer is a language model built from one stack of LL identical blocks. Each block has causal multi-head self-attention, where position ii sees only positions j≤ij\le i, and a position-wise MLP, both wrapped in residual connections. There is no encoder and no Cross-attention. A token embedding feeds the stack, and a linear LM head on the last layer turns each position into logits over the vocabulary. Training maximises ∑tlog⁡p(xt∣x<t)\sum_t\log p(x_t\mid x_{<t}) on raw text. Generation samples one token at a time and appends it to the input. GPT, GPT-2, GPT-3, Llama, Mistral and most current large language models have this shape.

The original Transformer (Vaswani et al., 2017) was an Encoder–decoder transformer for translation. Liu et al. ("Generating Wikipedia by Summarizing Long Sequences", ICLR 2018) showed that the decoder alone, fed the source and target as one sequence, works well for long inputs. Radford et al. at OpenAI then made it the standard pre-training architecture. GPT ("Improving Language Understanding by Generative Pre-Training", 2018) was 12 layers, d=768d=768, 12 heads, a 512-token context and about 117M parameters, pre-trained on BooksCorpus and fine-tuned per task. GPT-2 ("Language Models are Unsupervised Multitask Learners", 2019) moved layer normalization to the input of each sub-block, added a final layer norm, used a 1024-token context and a 50,257-token byte-level BPE vocabulary, and scaled to 1.5B parameters (48 layers, d=1600d=1600). GPT-3 (Brown et al., "Language Models are Few-Shot Learners", NeurIPS 2020) kept the design at 175B parameters: 96 layers, d=12288d=12288, 96 heads of 128, a 2048-token context and 300B training tokens. Its layers alternate dense and locally banded sparse attention.

Why drop the encoder? Next-token prediction needs no labels, so every token of every document is a training target. One causal stack also serves every task: the "input" is just the prefix. An encoder–decoder spends parameters on two stacks and needs an input/output split. At the same compute, the decoder-only model trains on all positions in parallel under one mask and caches cleanly at inference.

The block

h(0)=E[x1:T]  (+ P1:T)a(ℓ)=h(ℓ−1)+MHAcausal(Norm(h(ℓ−1)))h(ℓ)=a(ℓ)+MLP(Norm(a(ℓ)))ℓ=1…Lp(xt+1∣x≤t)=softmax(WU Norm(ht(L)))\begin{aligned}h^{(0)}&=E[x_{1:T}]\;(+\,P_{1:T})\\ a^{(\ell)}&=h^{(\ell-1)}+\mathrm{MHA}_{\text{causal}}\big(\mathrm{Norm}(h^{(\ell-1)})\big)\\ h^{(\ell)}&=a^{(\ell)}+\mathrm{MLP}\big(\mathrm{Norm}(a^{(\ell)})\big)\qquad \ell=1\dots L\\ p(x_{t+1}\mid x_{\le t})&=\mathrm{softmax}\big(W_U\,\mathrm{Norm}(h^{(L)}_t)\big)\end{aligned}
E∈RV×dE\in\mathbb R^{V\times d}
token embedding; GPT-2 and GPT-3 tie WU=EW_U=E, Llama keeps a separate output matrix
PP
learned absolute positions (GPT-1/2/3); modern models drop PP and rotate q,kq,k inside attention instead (see Rotary positional embeddings (RoPE))
MHAcausal\mathrm{MHA}_{\text{causal}}
HH heads of Scaled dot-product attention with the mask j≤ij\le i (see Causal attention, Multi-head attention)
Norm\mathrm{Norm}
applied before each sub-layer (pre-norm): LayerNorm in GPT-2/3, RMSNorm in Llama
MLP\mathrm{MLP}
d→dff→dd\to d_{ff}\to d per position: GELU with dff=4dd_{ff}=4d (GPT), or SwiGLU W2(SiLU(W1x)⊙W3x)W_2(\mathrm{SiLU}(W_1x)\odot W_3x) with dff≈83dd_{ff}\approx\tfrac{8}{3}d (Llama)
ht(L)h^{(L)}_t
the last layer at position tt predicts token t+1t+1; all TT predictions come out of one forward pass

Training uses teacher forcing for free. Shift the sequence by one, input=x1:T−1\text{input}=x_{1:T-1} and target=x2:T\text{target}=x_{2:T}, and take the cross-entropy at every position. The causal mask guarantees that the prediction at tt never saw xt+1x_{t+1}. One sequence of 4096 tokens therefore gives 4095 training targets in a single pass, which is why decoder-only pre-training is so sample-efficient per FLOP.

Counting parameters and FLOPs

N≈4d2⏟WQ,WK,WV,WO⋅L+2 d dff⏟MLP⋅L  =dff=4d  12 L d2,Ctrain≈6N FLOPs per token  +  6 L T dN\approx \underbrace{4d^2}_{W_Q,W_K,W_V,W_O}\cdot L+\underbrace{2\,d\,d_{ff}}_{\text{MLP}}\cdot L\;\overset{d_{ff}=4d}{=}\;12\,L\,d^2,\qquad C_{\text{train}}\approx 6N\ \text{FLOPs per token}\;+\;6\,L\,T\,d
12Ld212Ld^2
non-embedding parameters (Kaplan et al., "Scaling Laws for Neural Language Models", 2020); biases and norms add only O(Ld)O(Ld)
VdVd
the embedding (and an untied head) comes on top: 31.6% of GPT-2 small's 124M, 0.4% of GPT-3
6N6N
a weight is used in one multiply-add (2 FLOPs) per token forward and two in the backward pass (gradient w.r.t. input and w.r.t. weight): 2+4=62+4=6
6LTd6LTd
the QK⊤QK^\top and AVAV products at context TT (forward 2LTd2LTd per token in Kaplan et al.), small unless TT is comparable to 2d2d
C≈6NDC\approx6ND
total training compute for DD tokens, the budget used by scaling laws

The rule is accurate for GPT-style models: 12Ld212Ld^2 is 99.9% of GPT-2 small's non-embedding weights. Modern recipes move it a little. SwiGLU has three matrices, so with dff=83dd_{ff}=\tfrac83d it keeps 8d28d^2 per layer, but Llama 2 7B rounds dffd_{ff} up to 11008 (2.69d2.69d), giving 8.06d28.06d^2. Grouped-query attention (GQA) shrinks WK,WVW_K,W_V: Llama 3 8B has 8 KV heads for 32 query heads and dff=14336d_{ff}=14336, so a layer is 2.5d2+10.5d2=13d22.5d^2+10.5d^2=13d^2, and 12Ld212Ld^2 covers 92.3% of its non-embedding weights. The 128,256-token vocabulary, untied, adds another 1.05B parameters.

The modern recipe (Llama family, most open models since 2023)

Normalization
pre-norm RMSNorm (Zhang and Sennrich, 2019): x/x2‾+ϵ⋅gx/\sqrt{\overline{x^2}+\epsilon}\cdot g, no mean subtraction, no bias; one more at the end of the stack
Positions
Rotary positional embeddings (RoPE) (Su et al., 2021) on qq and kk in every layer; no position table
MLP
SwiGLU (Shazeer, "GLU Variants Improve Transformer", 2020), dff≈83dd_{ff}\approx\tfrac83d so the parameter count matches a 4d4d GELU MLP
Attention
Grouped-query attention (GQA) (Ainslie et al., 2023): Llama 2 34B/70B and all Llama 3 sizes use 8 KV heads
Linear layers
no biases
Llama 2 7B
L=32L=32, d=4096d=4096, 32 heads, dff=11008d_{ff}=11008, V=32000V=32000, context 4096, 6,738,415,616 parameters, 2T training tokens
Llama 2 70B
L=80L=80, d=8192d=8192, 64 query heads, 8 KV heads, dff=28672d_{ff}=28672, 68,976,648,192 parameters

Inference: prefill, decode and the KV cache

Generation has two phases. Prefill runs the whole prompt through the stack in one causal pass, exactly like training. Decode then produces one token per step. Because of the mask, earlier positions never change, so each layer stores the keys and values it has already computed (the KV cache, see Causal attention). A decode step only computes q,k,vq,k,v for the new token, appends k,vk,v to the cache, and attends over it. Each step costs about 2N2N FLOPs plus 4LTd4LTd for attention, but it must read every weight and the whole cache from memory. At small batch, decode speed is therefore set by memory bandwidth, not arithmetic.

KV bytes=2⋅L⋅nkv⋅dh⋅T⋅B⋅b\text{KV bytes}=2\cdot L\cdot n_{kv}\cdot d_h\cdot T\cdot B\cdot b
22
one key and one value vector
nkv, dhn_{kv},\,d_h
key/value heads and head size; nkv=Hn_{kv}=H for MHA, 11 for Multi-query attention (MQA), H/gH/g for GQA
T, BT,\,B
tokens in context and sequences in the batch
bb
bytes per number, 2 for fp16/bf16
N ≈ 12·L·d² · train ≈ 6N + 6·L·T·d FLOPs/token · KV = 2·L·n_kv·d_h·T·B·2 bytessoftmax over V = 32,000LM headfinal RMSNorm → d × V× L = 32MLPSwiGLU 4096→11008(×2)→4096RMSNorm (pre-norm)causal self-attentionH=32, n_kv=32, d_h=128, RoPERMSNorm (pre-norm)++token embeddingd = 4096, no position tabletokens x₁ … x_T, causal mask j ≤ iper layer: 4d² attention + 8d² MLPhere: attn 67.1M + MLP 135.3M12d² = 201.3M per layertrained context 4,096Llama 2 7B: N = 6,738,415,616 (6.74B)12·L·d² = 6.44B = 99.5% of non-embedding 6.48B■ embed/head 3.9%■ attention 31.9%■ MLP 64.2%■ norms 0.0%train FLOPs/token at T = 4,096: 6N = 3.89e10+ attention 6·L·T·d = 3.22e9 (7.7% of total)weights 12.55 GiB · KV cache 2.00 GiB (512 KiB/token)KV cache GiB (log), B = 1, 2 bytes per number1e-31e-21e-11101001,00010,000100,0005122K8K32K128Kweights— n_kv = 32— n_kv = 1context Tone decode step reads all weights + the whole cache:14.55 GiB for 1 new token · KV 2.00 GiB/sequenceL = 32 · d = 4096 · d_ff = 11008 · V = 32,000 · T = 4,096 · B = 1
model
K/V heads
fig.The left panel draws one pre-norm block of the selected model, and the right panel counts it from its real configuration. Llama 2 7B gives 6,738,415,616 parameters. 12·L·d² = 6.44B is 99.5% of the 6.48B non-embedding weights. The MLP is 64.2% of all weights, attention 31.9% and the untied embeddings 3.9%. At T = 4096 a training token costs 6N = 3.89e10 FLOPs plus 3.22e9 for attention (7.7%); at T = 32K the attention share reaches 39.9%. Its KV cache is 512 KiB per token, 2.00 GiB at 4096 tokens, against 12.55 GiB of bf16 weights. The bold curve crosses the weights line near 25K tokens, or already at 4096 tokens with batch 8 (16 GiB). Llama 2 70B with its 8 KV heads needs 320 KiB per token (1.25 GiB at 4096). Switch it to MHA and the cache becomes 10 GiB and the model 78.37B parameters. GPT-2 small has 124,439,808 parameters, 31.6% of them in the embedding. GPT-3 needs 4.5 MiB of cache per token, 9 GiB for one 2048-token sequence.

Worked example: GPT-2 small, counted by hand

GPT-2 small has L=12L=12, d=768d=768, H=12H=12, dff=3072d_{ff}=3072, V=50257V=50257 and 1024 learned positions. Every linear layer has a bias, LayerNorm has a gain and a bias, and the LM head is tied to EE.

componentformulaparameters
token embedding wteV·d = 50257·76838,597,376
position embedding wpe1024·d786,432
attention, per layerd·3d + 3d + d·d + d2,362,368
MLP, per layerd·4d + 4d + 4d·d + d4,722,432
2 LayerNorms, per layer2·2d3,072
one layer12d² + 13d7,087,872
12 layers85,054,464
final LayerNorm2d1,536
total (head tied)124,439,808
Exact parameter count (checked in Node against the 124,439,808 of the released checkpoint)

From parameters to compute and memory

  1. 112Ld2=12⋅12⋅7682=84,934,65612Ld^2=12\cdot12\cdot768^2=84{,}934{,}656, against 85,056,00085{,}056{,}000 actual non-embedding weights (99.9%). The embeddings are 39,383,80839{,}383{,}808, 31.6% of the model, which is why small models are often quoted "without embeddings".
  2. 2Training FLOPs per token at T=1024T=1024: 6N=6⋅85,056,000=5.10×1086N=6\cdot85{,}056{,}000=5.10\times10^8, plus 6LTd=6⋅12⋅1024⋅768=5.66×1076LTd=6\cdot12\cdot1024\cdot768=5.66\times10^7 for attention, 10.0% of the total.
  3. 3Scale up: Llama 2 7B trained on D=2×1012D=2\times10^{12} tokens costs 6ND=6⋅6.738×109⋅2×1012≈8.1×10226ND=6\cdot6.738\times10^9\cdot2\times10^{12}\approx8.1\times10^{22} FLOPs.
  4. 4KV cache of GPT-2 small in fp16: 2⋅12⋅12⋅64⋅2=36,8642\cdot12\cdot12\cdot64\cdot2=36{,}864 bytes =36=36 KiB per token, 36 MiB for a full 1024-token context.
  5. 5Llama 2 70B with GQA: 2⋅80⋅8⋅128⋅2=327,6802\cdot80\cdot8\cdot128\cdot2=327{,}680 bytes =320=320 KiB per token. With 64 KV heads it would be 2.52.5 MiB, 8 times more.
python
1import torch, torch.nn as nn, torch.nn.functional as F
2
3class RMSNorm(nn.Module):
4 def __init__(self, d, eps=1e-5):
5 super().__init__()
6 self.eps, self.w = eps, nn.Parameter(torch.ones(d))
7 def forward(self, x):
8 return self.w * x * torch.rsqrt(x.pow(2).mean(-1, keepdim=True) + self.eps)
9
10def rope(x, pos, base=10000.0):
11 # x (B, h, T, dh), pos (T,) absolute positions; rotates the pairs (x[i], x[i + dh/2])
12 dh = x.size(-1)
13 inv = base ** (-torch.arange(0, dh, 2, device=x.device).float() / dh) # (dh/2,)
14 ang = pos.float()[:, None] * inv[None, :] # (T, dh/2)
15 cos, sin = ang.cos(), ang.sin()
16 x1, x2 = x[..., : dh // 2], x[..., dh // 2 :]
17 return torch.cat([x1 * cos - x2 * sin, x1 * sin + x2 * cos], -1)
18
19class Block(nn.Module):
20 def __init__(self, d, H, kv, dff):
21 super().__init__()
22 self.H, self.kv, self.dh = H, kv, d // H
23 self.n1, self.n2 = RMSNorm(d), RMSNorm(d)
24 self.wq = nn.Linear(d, d, bias=False)
25 self.wkv = nn.Linear(d, 2 * kv * self.dh, bias=False) # GQA: only kv heads of K and V
26 self.wo = nn.Linear(d, d, bias=False)
27 self.w13 = nn.Linear(d, 2 * dff, bias=False) # SwiGLU gate and up projections
28 self.w2 = nn.Linear(dff, d, bias=False)
29
30 def forward(self, x, pos, cache=None):
31 B, T, d = x.shape
32 h = self.n1(x) # pre-norm
33 q = self.wq(h).view(B, T, self.H, self.dh).transpose(1, 2) # (B, H, T, dh)
34 k, v = self.wkv(h).view(B, T, 2, self.kv, self.dh).permute(2, 0, 3, 1, 4) # each (B, kv, T, dh)
35 q, k = rope(q, pos), rope(k, pos)
36 if cache is not None: # KV cache
37 if cache["k"] is not None:
38 k, v = torch.cat([cache["k"], k], 2), torch.cat([cache["v"], v], 2)
39 cache["k"], cache["v"] = k, v
40 k = k.repeat_interleave(self.H // self.kv, 1) # share each K/V head
41 v = v.repeat_interleave(self.H // self.kv, 1)
42 mask = torch.arange(k.size(2), device=x.device)[None, :] <= pos[:, None] # (T, S): key j <= query i
43 a = F.scaled_dot_product_attention(q, k, v, attn_mask=mask)
44 x = x + self.wo(a.transpose(1, 2).reshape(B, T, d))
45 g, u = self.w13(self.n2(x)).chunk(2, -1)
46 return x + self.w2(F.silu(g) * u)
47
48class DecoderLM(nn.Module):
49 def __init__(self, V, d=512, L=8, H=8, kv=2, dff=1408):
50 super().__init__()
51 self.emb = nn.Embedding(V, d)
52 self.blocks = nn.ModuleList(Block(d, H, kv, dff) for _ in range(L))
53 self.norm = RMSNorm(d)
54 self.head = nn.Linear(d, V, bias=False) # untied, as in Llama
55
56 def forward(self, idx, start=0, caches=None):
57 pos = torch.arange(start, start + idx.size(1), device=idx.device)
58 x = self.emb(idx)
59 for i, blk in enumerate(self.blocks):
60 x = blk(x, pos, None if caches is None else caches[i])
61 return self.head(self.norm(x)) # logits for the NEXT token
62
63V = 1000
64model = DecoderLM(V)
65idx = torch.randint(0, V, (2, 16))
66logits = model(idx[:, :-1]) # inputs x_1..x_15
67loss = F.cross_entropy(logits.reshape(-1, V), idx[:, 1:].reshape(-1)) # targets x_2..x_16
68print(logits.shape) # torch.Size([2, 15, 1000])
69print(sum(p.numel() for p in model.parameters())) # 23577088 (12*L*d^2 would say 25165824)
70
71@torch.no_grad()
72def generate(model, prompt, steps):
73 caches = [{"k": None, "v": None} for _ in model.blocks]
74 logits = model(prompt, 0, caches) # prefill: the whole prompt in one pass
75 out, t = prompt, prompt.size(1)
76 for _ in range(steps):
77 nxt = logits[:, -1].argmax(-1, keepdim=True) # greedy next token
78 out = torch.cat([out, nxt], 1)
79 logits = model(nxt, t, caches) # decode: one token, cached K/V reused
80 t += 1
81 return out, caches
82
83out, caches = generate(model.eval(), idx[:, :4], 6)
84print(out.shape, caches[0]["k"].shape) # torch.Size([2, 10]) torch.Size([2, 2, 10, 64])
A Llama-style decoder-only LM (pre-RMSNorm, RoPE, SwiGLU, GQA) with next-token loss and KV-cached generation

Use when

  • +One stack, one objective: every token of unlabelled text is a training target
  • +Any task becomes a prompt; few-shot and instruction following come from the same model
  • +Training is fully parallel over positions under one causal mask
  • +Clean incremental inference: the KV cache makes each new token one forward step

Avoid when

  • −Generation is sequential, one token per forward pass, and memory-bandwidth bound at small batch
  • −The KV cache grows with T·B and can exceed the weights for long contexts or large batches
  • −Attention cost is still quadratic in context for prefill and training
  • −Causal attention gives each prefix token a one-sided view, which encoders avoid for classification or retrieval embeddings

Check yourself

A GPT-style model has L = 24 layers, d = 2048, d_ff = 4d and standard multi-head attention. About how many non-embedding parameters does it have?

Check yourself

Llama 2 70B uses 64 query heads but only 8 key/value heads (d_h = 128, L = 80). How large is its bf16 KV cache per token?

Remember

  • ✓Non-embedding parameters N≈12Ld2N\approx12Ld^2 (4d24d^2 attention + 8d28d^2 MLP per layer); training costs ≈6N\approx6N FLOPs per token, so C≈6NDC\approx6ND.
  • ✓Decoder-only = causal self-attention + MLP blocks, no cross-attention; the output at tt predicts xt+1x_{t+1}, trained with cross-entropy on all positions at once.
  • ✓KV cache =2⋅L⋅nkv⋅dh⋅T⋅B⋅b=2\cdot L\cdot n_{kv}\cdot d_h\cdot T\cdot B\cdot b bytes: 512 KiB per token for Llama 2 7B, 320 KiB for Llama 2 70B with 8 KV heads.
  • ✓Modern recipe: pre-norm RMSNorm, RoPE, SwiGLU (d_ff ≈ 8d/3), grouped-query attention, no biases, untied LM head.
  • ✓Prefill is compute-bound and parallel; decode is one token per step and limited by reading weights plus cache from memory.
← back to Deep Learning roadmap