Definition
GPT architecture is the decoder-only transformer design of OpenAI's three Generative Pre-trained Transformer papers: Radford, Narasimhan, Salimans and Sutskever, "Improving Language Understanding by Generative Pre-Training" (2018, GPT-1); Radford, Wu, Child, Luan, Amodei and Sutskever, "Language Models are Unsupervised Multitask Learners" (2019, GPT-2); and Brown et al., "Language Models are Few-Shot Learners" (NeurIPS 2020, GPT-3). Each is a stack of causal self-attention and GELU MLP blocks over learned token and position embeddings, trained only to predict the next token, with the output layer tied to the input embedding. The three papers change very little in the block. What changes is the scale, from 117M to 175B parameters, and how the model is used: fine-tuned per task, then zero-shot, then few-shot from examples in the prompt.
In 2017 the best NLP systems were task-specific architectures trained on labelled data, sometimes initialised with Word embeddings or with Contextual embeddings such as ELMo. GPT-1 asked whether one generic network, pre-trained as a language model on unlabelled books, could be fine-tuned to every task with only a new linear output layer. The answer was yes: it improved the state of the art on 9 of the 12 tasks it was tested on. GPT-2 dropped the fine-tuning and showed that a larger model trained on more varied web text already performs tasks zero-shot when the prompt is phrased as text. GPT-3 scaled that model about 100× more and found that a handful of worked examples in the prompt, with no gradient updates, is often enough.
The generic shape of this model is covered in Decoder-only transformer, and the mask in Causal attention. This article is about the specific choices OpenAI made, why each was made, and what the published configurations look like.
The block, generation by generation
- residual stream, (token plus learned position embeddings)
- masked multi-head self-attention, heads of size (see Multi-head attention)
- with
- LayerNorm with gain and bias; pre-LN models apply one more, , before the head
- : the same matrix embeds and un-embeds
Post-LN to pre-LN. In GPT-1 every addition is followed by a LayerNorm, so the gradient reaching layer 1 has passed through normalisations. Deep post-LN stacks need a careful learning-rate warm-up and become unstable as depth grows. GPT-2 put the norm on the branch instead. Now the residual trunk is a plain sum , and the gradient has an identity path from the loss to every layer. Because nothing on the trunk is normalised any more, its scale grows with depth, so GPT-2 adds a final LayerNorm before the head. Xiong et al. ("On Layer Normalization in the Transformer Architecture", ICML 2020) later showed that pre-LN gradients are well behaved at initialisation and can be trained without warm-up.
Learned positions. All three models add a learned table instead of the sinusoids of Vaswani et al. (see Positional encoding). The table costs only parameters, 0.6% of GPT-2 small. The price is a hard context limit, 512, 1024 and 2048 tokens. Position has no embedding, so the model cannot even be run past it. Later families replaced the table with Rotary positional embeddings (RoPE).
GELU. The MLP uses the Gaussian Error Linear Unit of Hendrycks and Gimpel (2016), , in its tanh approximation . It is a smooth ReLU that lets small negative values through. GPT-1 was among the first large models to use it, and GELU became the default MLP activation until gated variants such as SwiGLU took over (see Llama).
Weight tying. The head that turns into logits is the token embedding itself (Press and Wolf, "Using the Output Embedding to Improve Language Models", 2017; Inan et al., 2017). A token's input vector and its output vector are then the same row. Tying saves parameters, 38.6M in GPT-2 small, which is 31% of the model. It also shares the gradients. The input row of a rare token is otherwise updated only when that token appears in the input. Tied, it also gets the output layer's gradient from every softmax.
GPT-2's initialisation: scaling the residual path by 1/√N
GPT-2 initialises every weight from , as GPT-1 did, with one change. In the paper's words, it "scale[s] the weights of residual layers at initialization by a factor of where is the number of residual layers". There are two residual branches per block, so . In the released code and in Hugging Face this means the two output projections that write into the trunk, attn.c_proj and mlp.c_proj, get standard deviation .
- the -th residual branch (attention or MLP); with independent weights its output is uncorrelated with everything already on the trunk
- variance of the embeddings, (GPT-2's
wteandwpeinit) - variance one branch adds. Its input is LayerNorm-ed to unit scale, so is the same at every depth and is proportional to
- number of residual layers; 24 in GPT-2 small, 96 in GPT-2 XL, 192 in GPT-3 175B
In a pre-LN stack every branch sees a normalised input, so every branch adds about the same variance. Without the rescaling, the trunk's variance grows linearly with depth. A 48-layer model then starts with a residual stream many times larger than a 12-layer one, and each late layer's update is small relative to it. Dividing by makes the total added variance independent of depth. Note what it does not fix: still grows with width, because is a fixed standard deviation, not a fan-in rule.
Published configurations
- GPT-1
- , , 12 heads, MLP 3072, context 512, BPE with 40,000 merges, about 117M parameters. BooksCorpus (about 7,000 unpublished books). Adam, peak lr , 2000 warm-up steps then cosine decay, 100 epochs of minibatches of 64 × 512 tokens, dropout 0.1. Fine-tuning adds the LM loss with weight 0.5
- GPT-2
- four sizes: , , , , with 12, 16, 20 and 25 heads. The paper calls them 117M, 345M, 762M and 1542M; the released checkpoints count 124M, 355M, 774M and 1558M with the embeddings
- GPT-2 data and tokenizer
- WebText: outbound links from Reddit with at least 3 karma, 8 million documents, 40 GB. Byte-level BPE (see Byte-pair encoding (BPE)) with 50,257 tokens that never merges across character categories. Context 1024, batch 512 sequences
- GPT-3
- eight sizes, all trained on 300B tokens with context 2048: Small 125M (12, 768), Medium 350M (24, 1024), Large 760M (24, 1536), XL 1.3B (24, 2048), 2.7B (32, 2560), 6.7B (32, 4096), 13B (40, 5140), 175B (96, 12288 with 96 heads of 128)
- GPT-3 optimiser
- Adam , , ; gradient clipping at norm 1.0; weight decay 0.1; warm-up over 375M tokens, then cosine decay to 10% over 260B tokens; the batch grows from 32K tokens to its full size over the first 4–12B tokens. The peak lr falls from (Small) to (175B), and the batch grows from 0.5M to 3.2M tokens
- GPT-3 data
- filtered Common Crawl (410B tokens, 60% of the mix), WebText2 (22%), Books1 (8%), Books2 (8%), Wikipedia (3%); documents are packed into full 2048-token sequences, separated by an end-of-text token
- GPT-3 compute
- FLOPs for 175B, about 3,640 petaflop/s-days
The GPT-3 table has two quirks worth knowing when you reproduce it. XL lists 24 heads of size 128 with , and 13B lists with 40 heads of 128. Neither satisfies , so the printed numbers cannot all be exact. The parameter counts follow from and alone, and those match the stated sizes.
Worked example: what 1/√N buys GPT-2 small
GPT-2 small has , so residual branches, and . Follow one MLP branch at initialisation. The LayerNorm output has unit mean square. The first layer has fan-in and std . GELU is the tanh form, and is taken by numerical integration.
| quantity | plain init | GPT-2 init |
|---|---|---|
| pre-activation variance d·0.02² | 0.3072 | 0.3072 |
| E[GELU(h)²], h ~ N(0, 0.3072) | 0.10646 | 0.10646 |
| c_proj std σ_p | 0.02 | 0.02/√24 = 0.00408 |
| variance added per branch c = 4d·σ_p²·E[GELU²] | 0.1308 | 0.00545 |
| Var after 12 MLP branches, v₀ + 12c | 1.5703 | 0.0659 |
| std of the residual stream | 1.253 | 0.257 |
| same at L = 48, d = 1600 (GPT-2 1542M) | 5.581 | 0.570 |
Step by step
- 1Pre-activation: with , so , exactly.
- 2After GELU: . The projection has fan-in , so each output coordinate gets variance .
- 3Plain: per branch. Twelve branches plus give , std . After the first branch alone the std is already , sixteen times the embeddings' .
- 4Scaled: , so each branch adds . The total is , std . For any depth, makes the MLP branches add in total, so a 6-layer and a 96-layer model of width 768 both end at .
- 5The attention branches add a little more on top. At init , so each head averages the values of all visible positions, and that average has a much smaller variance than one value.
GPT-3 and in-context learning
GPT-3 "uses the same model and architecture as GPT-2, including the modified initialization, pre-normalization, and reversible tokenization", with one exception: layers alternate between dense attention and the locally banded sparse attention of the Sparse Transformer (Child et al., 2019; see Sparse attention). Eight sizes from 125M to 175B were trained on the same 300B tokens so that the paper could plot each ability against scale.
The paper's central object is in-context learning. The task is specified entirely in the prompt and the weights never change. Zero-shot gives a natural-language instruction. One-shot adds one demonstration. Few-shot adds as many as fit in the 2048-token window, typically 10 to 100. On LAMBADA, 175B scored 76.2% zero-shot and 86.4% few-shot. On TriviaQA (closed book) it scored 64.3%, 68.0% and 71.2% at zero, one and few shots. The gap between zero-shot and few-shot grew with model size, which the authors read as larger models being better meta-learners.
- number of demonstrations: 0, 1, or as many as fit (few-shot)
- the pre-trained next-token model; nothing is fine-tuned
- for multiple-choice tasks the paper compares the likelihood of each completion; free-form answers use beam search
1import math, torch, torch.nn as nn, torch.nn.functional as F23class Block(nn.Module):4 def __init__(self, d, H):5 super().__init__()6 self.H = H7 self.ln_1, self.ln_2 = nn.LayerNorm(d), nn.LayerNorm(d)8 self.c_attn = nn.Linear(d, 3 * d) # q, k, v in one matmul9 self.attn_proj = nn.Linear(d, d) # writes into the residual stream10 self.c_fc = nn.Linear(d, 4 * d)11 self.mlp_proj = nn.Linear(4 * d, d) # writes into the residual stream1213 def forward(self, x):14 B, T, d = x.shape15 q, k, v = self.c_attn(self.ln_1(x)).split(d, dim=-1) # pre-LN16 q, k, v = (t.view(B, T, self.H, d // self.H).transpose(1, 2) for t in (q, k, v))17 a = F.scaled_dot_product_attention(q, k, v, is_causal=True)18 x = x + self.attn_proj(a.transpose(1, 2).reshape(B, T, d))19 return x + self.mlp_proj(F.gelu(self.c_fc(self.ln_2(x)), approximate="tanh"))2021class GPT2(nn.Module):22 def __init__(self, V=50257, T=1024, L=12, d=768, H=12):23 super().__init__()24 self.wte, self.wpe = nn.Embedding(V, d), nn.Embedding(T, d) # learned positions25 self.blocks = nn.ModuleList(Block(d, H) for _ in range(L))26 self.ln_f = nn.LayerNorm(d) # added in GPT-227 self.head = nn.Linear(d, V, bias=False)28 self.head.weight = self.wte.weight # weight tying29 for name, p in self.named_parameters():30 if p.dim() == 1: # biases 0, LayerNorm gains 131 nn.init.zeros_(p) if name.endswith("bias") else nn.init.ones_(p)32 continue33 std = 0.02 / math.sqrt(2 * L) if name.endswith("proj.weight") else 0.0234 nn.init.normal_(p, 0.0, std) # residual projections get 0.02/sqrt(N), N = 2L35 nn.init.normal_(self.wpe.weight, 0.0, 0.01)3637 def forward(self, idx, targets=None):38 pos = torch.arange(idx.size(1), device=idx.device)39 x = self.wte(idx) + self.wpe(pos)40 for blk in self.blocks:41 x = blk(x)42 logits = self.head(self.ln_f(x))43 loss = None if targets is None else F.cross_entropy(logits.flatten(0, 1), targets.flatten())44 return logits, loss4546model = GPT2()47print(sum(p.numel() for p in model.parameters())) # tied weights are counted once: 124,439,8084849# The released checkpoint, few-shot style: the "training data" is in the prompt.50from transformers import GPT2LMHeadModel, GPT2TokenizerFast51tok, lm = GPT2TokenizerFast.from_pretrained("gpt2"), GPT2LMHeadModel.from_pretrained("gpt2")52prompt = "sea otter => loutre de mer\ncheese => fromage\npeppermint =>"53out = lm.generate(**tok(prompt, return_tensors="pt"), max_new_tokens=5, do_sample=False)54print(tok.decode(out[0])) # 124M is far too small to do this reliably; 175B mostly canUse when
- +One architecture and one objective for every task; only the prompt or a linear head changes
- +Pre-LN with a final LayerNorm and depth-scaled init trains stably to 96 layers
- +Tied embeddings save V·d parameters, 31% of GPT-2 small
- +The parameter count is predictable: 12·L·d² is 99.9% of the non-embedding weights
- +Few-shot in-context learning at 175B, with no gradient updates
Avoid when
- −Learned positions fix the context at 512/1024/2048 tokens and do not extrapolate
- −Full multi-head attention: the KV cache of 175B is 4.5 MiB per token (see KV cache)
- −Biases everywhere and a 4d GELU MLP; later recipes (RMSNorm, SwiGLU, no biases) do better per FLOP
- −GPT-3's 300B tokens are far below the compute-optimal amount for 175B parameters
- −Few-shot accuracy is sensitive to the choice, format and order of the examples
Check yourself
Why did GPT-2 need a final LayerNorm before the head, when GPT-1 did not?
Check yourself
GPT-2 1542M has L = 48. What standard deviation does it use for attn.c_proj and mlp.c_proj at init?
Remember
- ✓GPT-1 → GPT-2 → GPT-3 kept one block (masked MHA + GELU MLP of width , learned positions, tied head) and grew from 117M to 175B parameters.
- ✓GPT-2 moved LayerNorm to the branch inputs (pre-LN), added a final LayerNorm, and scaled residual projections by with . That makes the variance the branches add at init independent of depth.
- ✓Parameters per layer are ; is 99.9% of the non-embedding weights, and reproduces GPT-3's FLOPs.
- ✓GPT-3: 8 sizes all on 300B tokens, context 2048, alternating dense and locally banded sparse attention, Adam β₂ = 0.95, cosine decay to 10%.
- ✓In-context learning: the task is defined by instructions and K examples in the prompt, with frozen weights; the few-shot gain grows with scale.