Definition
Decoder-only transformer is a language model built from one stack of identical blocks. Each block has causal multi-head self-attention, where position sees only positions , and a position-wise MLP, both wrapped in residual connections. There is no encoder and no Cross-attention. A token embedding feeds the stack, and a linear LM head on the last layer turns each position into logits over the vocabulary. Training maximises on raw text. Generation samples one token at a time and appends it to the input. GPT, GPT-2, GPT-3, Llama, Mistral and most current large language models have this shape.
The original Transformer (Vaswani et al., 2017) was an Encoder–decoder transformer for translation. Liu et al. ("Generating Wikipedia by Summarizing Long Sequences", ICLR 2018) showed that the decoder alone, fed the source and target as one sequence, works well for long inputs. Radford et al. at OpenAI then made it the standard pre-training architecture. GPT ("Improving Language Understanding by Generative Pre-Training", 2018) was 12 layers, , 12 heads, a 512-token context and about 117M parameters, pre-trained on BooksCorpus and fine-tuned per task. GPT-2 ("Language Models are Unsupervised Multitask Learners", 2019) moved layer normalization to the input of each sub-block, added a final layer norm, used a 1024-token context and a 50,257-token byte-level BPE vocabulary, and scaled to 1.5B parameters (48 layers, ). GPT-3 (Brown et al., "Language Models are Few-Shot Learners", NeurIPS 2020) kept the design at 175B parameters: 96 layers, , 96 heads of 128, a 2048-token context and 300B training tokens. Its layers alternate dense and locally banded sparse attention.
Why drop the encoder? Next-token prediction needs no labels, so every token of every document is a training target. One causal stack also serves every task: the "input" is just the prefix. An encoder–decoder spends parameters on two stacks and needs an input/output split. At the same compute, the decoder-only model trains on all positions in parallel under one mask and caches cleanly at inference.
The block
- token embedding; GPT-2 and GPT-3 tie , Llama keeps a separate output matrix
- learned absolute positions (GPT-1/2/3); modern models drop and rotate inside attention instead (see Rotary positional embeddings (RoPE))
- heads of Scaled dot-product attention with the mask (see Causal attention, Multi-head attention)
- applied before each sub-layer (pre-norm): LayerNorm in GPT-2/3, RMSNorm in Llama
- per position: GELU with (GPT), or SwiGLU with (Llama)
- the last layer at position predicts token ; all predictions come out of one forward pass
Training uses teacher forcing for free. Shift the sequence by one, and , and take the cross-entropy at every position. The causal mask guarantees that the prediction at never saw . One sequence of 4096 tokens therefore gives 4095 training targets in a single pass, which is why decoder-only pre-training is so sample-efficient per FLOP.
Counting parameters and FLOPs
- non-embedding parameters (Kaplan et al., "Scaling Laws for Neural Language Models", 2020); biases and norms add only
- the embedding (and an untied head) comes on top: 31.6% of GPT-2 small's 124M, 0.4% of GPT-3
- a weight is used in one multiply-add (2 FLOPs) per token forward and two in the backward pass (gradient w.r.t. input and w.r.t. weight):
- the and products at context (forward per token in Kaplan et al.), small unless is comparable to
- total training compute for tokens, the budget used by scaling laws
The rule is accurate for GPT-style models: is 99.9% of GPT-2 small's non-embedding weights. Modern recipes move it a little. SwiGLU has three matrices, so with it keeps per layer, but Llama 2 7B rounds up to 11008 (), giving . Grouped-query attention (GQA) shrinks : Llama 3 8B has 8 KV heads for 32 query heads and , so a layer is , and covers 92.3% of its non-embedding weights. The 128,256-token vocabulary, untied, adds another 1.05B parameters.
The modern recipe (Llama family, most open models since 2023)
- Normalization
- pre-norm RMSNorm (Zhang and Sennrich, 2019): , no mean subtraction, no bias; one more at the end of the stack
- Positions
- Rotary positional embeddings (RoPE) (Su et al., 2021) on and in every layer; no position table
- MLP
- SwiGLU (Shazeer, "GLU Variants Improve Transformer", 2020), so the parameter count matches a GELU MLP
- Attention
- Grouped-query attention (GQA) (Ainslie et al., 2023): Llama 2 34B/70B and all Llama 3 sizes use 8 KV heads
- Linear layers
- no biases
- Llama 2 7B
- , , 32 heads, , , context 4096, 6,738,415,616 parameters, 2T training tokens
- Llama 2 70B
- , , 64 query heads, 8 KV heads, , 68,976,648,192 parameters
Inference: prefill, decode and the KV cache
Generation has two phases. Prefill runs the whole prompt through the stack in one causal pass, exactly like training. Decode then produces one token per step. Because of the mask, earlier positions never change, so each layer stores the keys and values it has already computed (the KV cache, see Causal attention). A decode step only computes for the new token, appends to the cache, and attends over it. Each step costs about FLOPs plus for attention, but it must read every weight and the whole cache from memory. At small batch, decode speed is therefore set by memory bandwidth, not arithmetic.
- one key and one value vector
- key/value heads and head size; for MHA, for Multi-query attention (MQA), for GQA
- tokens in context and sequences in the batch
- bytes per number, 2 for fp16/bf16
Worked example: GPT-2 small, counted by hand
GPT-2 small has , , , , and 1024 learned positions. Every linear layer has a bias, LayerNorm has a gain and a bias, and the LM head is tied to .
| component | formula | parameters |
|---|---|---|
| token embedding wte | V·d = 50257·768 | 38,597,376 |
| position embedding wpe | 1024·d | 786,432 |
| attention, per layer | d·3d + 3d + d·d + d | 2,362,368 |
| MLP, per layer | d·4d + 4d + 4d·d + d | 4,722,432 |
| 2 LayerNorms, per layer | 2·2d | 3,072 |
| one layer | 12d² + 13d | 7,087,872 |
| 12 layers | 85,054,464 | |
| final LayerNorm | 2d | 1,536 |
| total (head tied) | 124,439,808 |
From parameters to compute and memory
- 1, against actual non-embedding weights (99.9%). The embeddings are , 31.6% of the model, which is why small models are often quoted "without embeddings".
- 2Training FLOPs per token at : , plus for attention, 10.0% of the total.
- 3Scale up: Llama 2 7B trained on tokens costs FLOPs.
- 4KV cache of GPT-2 small in fp16: bytes KiB per token, 36 MiB for a full 1024-token context.
- 5Llama 2 70B with GQA: bytes KiB per token. With 64 KV heads it would be MiB, 8 times more.
1import torch, torch.nn as nn, torch.nn.functional as F23class RMSNorm(nn.Module):4 def __init__(self, d, eps=1e-5):5 super().__init__()6 self.eps, self.w = eps, nn.Parameter(torch.ones(d))7 def forward(self, x):8 return self.w * x * torch.rsqrt(x.pow(2).mean(-1, keepdim=True) + self.eps)910def rope(x, pos, base=10000.0):11 # x (B, h, T, dh), pos (T,) absolute positions; rotates the pairs (x[i], x[i + dh/2])12 dh = x.size(-1)13 inv = base ** (-torch.arange(0, dh, 2, device=x.device).float() / dh) # (dh/2,)14 ang = pos.float()[:, None] * inv[None, :] # (T, dh/2)15 cos, sin = ang.cos(), ang.sin()16 x1, x2 = x[..., : dh // 2], x[..., dh // 2 :]17 return torch.cat([x1 * cos - x2 * sin, x1 * sin + x2 * cos], -1)1819class Block(nn.Module):20 def __init__(self, d, H, kv, dff):21 super().__init__()22 self.H, self.kv, self.dh = H, kv, d // H23 self.n1, self.n2 = RMSNorm(d), RMSNorm(d)24 self.wq = nn.Linear(d, d, bias=False)25 self.wkv = nn.Linear(d, 2 * kv * self.dh, bias=False) # GQA: only kv heads of K and V26 self.wo = nn.Linear(d, d, bias=False)27 self.w13 = nn.Linear(d, 2 * dff, bias=False) # SwiGLU gate and up projections28 self.w2 = nn.Linear(dff, d, bias=False)2930 def forward(self, x, pos, cache=None):31 B, T, d = x.shape32 h = self.n1(x) # pre-norm33 q = self.wq(h).view(B, T, self.H, self.dh).transpose(1, 2) # (B, H, T, dh)34 k, v = self.wkv(h).view(B, T, 2, self.kv, self.dh).permute(2, 0, 3, 1, 4) # each (B, kv, T, dh)35 q, k = rope(q, pos), rope(k, pos)36 if cache is not None: # KV cache37 if cache["k"] is not None:38 k, v = torch.cat([cache["k"], k], 2), torch.cat([cache["v"], v], 2)39 cache["k"], cache["v"] = k, v40 k = k.repeat_interleave(self.H // self.kv, 1) # share each K/V head41 v = v.repeat_interleave(self.H // self.kv, 1)42 mask = torch.arange(k.size(2), device=x.device)[None, :] <= pos[:, None] # (T, S): key j <= query i43 a = F.scaled_dot_product_attention(q, k, v, attn_mask=mask)44 x = x + self.wo(a.transpose(1, 2).reshape(B, T, d))45 g, u = self.w13(self.n2(x)).chunk(2, -1)46 return x + self.w2(F.silu(g) * u)4748class DecoderLM(nn.Module):49 def __init__(self, V, d=512, L=8, H=8, kv=2, dff=1408):50 super().__init__()51 self.emb = nn.Embedding(V, d)52 self.blocks = nn.ModuleList(Block(d, H, kv, dff) for _ in range(L))53 self.norm = RMSNorm(d)54 self.head = nn.Linear(d, V, bias=False) # untied, as in Llama5556 def forward(self, idx, start=0, caches=None):57 pos = torch.arange(start, start + idx.size(1), device=idx.device)58 x = self.emb(idx)59 for i, blk in enumerate(self.blocks):60 x = blk(x, pos, None if caches is None else caches[i])61 return self.head(self.norm(x)) # logits for the NEXT token6263V = 100064model = DecoderLM(V)65idx = torch.randint(0, V, (2, 16))66logits = model(idx[:, :-1]) # inputs x_1..x_1567loss = F.cross_entropy(logits.reshape(-1, V), idx[:, 1:].reshape(-1)) # targets x_2..x_1668print(logits.shape) # torch.Size([2, 15, 1000])69print(sum(p.numel() for p in model.parameters())) # 23577088 (12*L*d^2 would say 25165824)7071@torch.no_grad()72def generate(model, prompt, steps):73 caches = [{"k": None, "v": None} for _ in model.blocks]74 logits = model(prompt, 0, caches) # prefill: the whole prompt in one pass75 out, t = prompt, prompt.size(1)76 for _ in range(steps):77 nxt = logits[:, -1].argmax(-1, keepdim=True) # greedy next token78 out = torch.cat([out, nxt], 1)79 logits = model(nxt, t, caches) # decode: one token, cached K/V reused80 t += 181 return out, caches8283out, caches = generate(model.eval(), idx[:, :4], 6)84print(out.shape, caches[0]["k"].shape) # torch.Size([2, 10]) torch.Size([2, 2, 10, 64])Use when
- +One stack, one objective: every token of unlabelled text is a training target
- +Any task becomes a prompt; few-shot and instruction following come from the same model
- +Training is fully parallel over positions under one causal mask
- +Clean incremental inference: the KV cache makes each new token one forward step
Avoid when
- −Generation is sequential, one token per forward pass, and memory-bandwidth bound at small batch
- −The KV cache grows with T·B and can exceed the weights for long contexts or large batches
- −Attention cost is still quadratic in context for prefill and training
- −Causal attention gives each prefix token a one-sided view, which encoders avoid for classification or retrieval embeddings
Check yourself
A GPT-style model has L = 24 layers, d = 2048, d_ff = 4d and standard multi-head attention. About how many non-embedding parameters does it have?
Check yourself
Llama 2 70B uses 64 query heads but only 8 key/value heads (d_h = 128, L = 80). How large is its bf16 KV cache per token?
Remember
- ✓Non-embedding parameters ( attention + MLP per layer); training costs FLOPs per token, so .
- ✓Decoder-only = causal self-attention + MLP blocks, no cross-attention; the output at predicts , trained with cross-entropy on all positions at once.
- ✓KV cache bytes: 512 KiB per token for Llama 2 7B, 320 KiB for Llama 2 70B with 8 KV heads.
- ✓Modern recipe: pre-norm RMSNorm, RoPE, SwiGLU (d_ff ≈ 8d/3), grouped-query attention, no biases, untied LM head.
- ✓Prefill is compute-bound and parallel; decode is one token per step and limited by reading weights plus cache from memory.