GPT architecture

GPT-1 (2018), GPT-2 (2019) and GPT-3 (2020) are one decoder-only design scaled 1,500×: learned positions, GELU MLPs, a tied head, pre-LN with a final LayerNorm and residual init ÷√N from GPT-2 on, and at 175B (96 layers, d = 12288, 300B tokens) few-shot learning from the prompt.

15 min read1465 wordscore0 viewsupdated 2026-10-13

Definition

GPT architecture is the decoder-only transformer design of OpenAI's three Generative Pre-trained Transformer papers: Radford, Narasimhan, Salimans and Sutskever, "Improving Language Understanding by Generative Pre-Training" (2018, GPT-1); Radford, Wu, Child, Luan, Amodei and Sutskever, "Language Models are Unsupervised Multitask Learners" (2019, GPT-2); and Brown et al., "Language Models are Few-Shot Learners" (NeurIPS 2020, GPT-3). Each is a stack of causal self-attention and GELU MLP blocks over learned token and position embeddings, trained only to predict the next token, with the output layer tied to the input embedding. The three papers change very little in the block. What changes is the scale, from 117M to 175B parameters, and how the model is used: fine-tuned per task, then zero-shot, then few-shot from examples in the prompt.

In 2017 the best NLP systems were task-specific architectures trained on labelled data, sometimes initialised with Word embeddings or with Contextual embeddings such as ELMo. GPT-1 asked whether one generic network, pre-trained as a language model on unlabelled books, could be fine-tuned to every task with only a new linear output layer. The answer was yes: it improved the state of the art on 9 of the 12 tasks it was tested on. GPT-2 dropped the fine-tuning and showed that a larger model trained on more varied web text already performs tasks zero-shot when the prompt is phrased as text. GPT-3 scaled that model about 100× more and found that a handful of worked examples in the prompt, with no gradient updates, is often enough.

The generic shape of this model is covered in Decoder-only transformer, and the mask in Causal attention. This article is about the specific choices OpenAI made, why each was made, and what the published configurations look like.

The block, generation by generation

GPT-1 (2018) · post-LNtied head · V = 40,478(no final LayerNorm)×12 · d = 768 · 12 headsmasked MHA+LayerNorm (on trunk)MLP 768→3072→768+LayerNorm (on trunk)wte + wpe (512)tokens x₁ … x_T (causal)117M · BooksCorpusfine-tuned per taskGPT-2 (2019) · pre-LNtied head · V = 50,257final LayerNorm×48 · d = 1600 · 25 headsLayerNormmasked MHA+LayerNormMLP 1600→6400→1600+wte + wpe (1024)tokens x₁ … x_T (causal)1.5B · WebText 40 GBc_proj init × 1/√N, N = 2LGPT-3 (2020) · pre-LNtied head · V = 50,257final LayerNorm×96 · d = 12288 · 96 headsLayerNormdense ⇄ banded sparse+LayerNormMLP 12288→49152→12288+wte + wpe (2048)tokens x₁ … x_T (causal)175B · 300B tokensalternate layershover: where LayerNorm sits
fig.One block of each generation, at its largest published size. GPT-1 is the original post-LN Transformer decoder: LayerNorm sits on the residual trunk after each addition, so every activation passes through it. GPT-2 moves LayerNorm onto the branch, in front of attention and the MLP, so the trunk is a pure sum. It adds a final LayerNorm before the head and scales the residual output projections by 1/√N at init. GPT-3 keeps the GPT-2 block and alternates dense causal attention with a locally banded sparse pattern from layer to layer (the band drawn here is illustrative; the paper does not give its width). All three use learned absolute positions, a GELU MLP four times wider than d, and an output head tied to the token embedding. Hover the bottom label to highlight every LayerNorm.
GPT-1 (post-LN):  a=LN(h+MHA(h)),h′=LN(a+MLP(a))GPT-2/3 (pre-LN):  a=h+MHA(LN(h)),h′=a+MLP(LN(a))\textbf{GPT-1 (post-LN): }\ a=\mathrm{LN}\big(h+\mathrm{MHA}(h)\big),\quad h'=\mathrm{LN}\big(a+\mathrm{MLP}(a)\big)\qquad \textbf{GPT-2/3 (pre-LN): }\ a=h+\mathrm{MHA}(\mathrm{LN}(h)),\quad h'=a+\mathrm{MLP}(\mathrm{LN}(a))
hh
residual stream, h(0)=We[x]+Wp[1:T]h^{(0)}=W_e[x]+W_p[1{:}T] (token plus learned position embeddings)
MHA\mathrm{MHA}
masked multi-head self-attention, HH heads of size d/Hd/H (see Multi-head attention)
MLP(u)\mathrm{MLP}(u)
W2 GELU(W1u+b1)+b2W_2\,\mathrm{GELU}(W_1u+b_1)+b_2 with W1∈R4d×dW_1\in\mathbb R^{4d\times d}
LN\mathrm{LN}
LayerNorm with gain and bias; pre-LN models apply one more, LNf\mathrm{LN}_f, before the head
p(xt+1∣x≤t)p(x_{t+1}\mid x_{\le t})
softmax(We LNf(ht(L)))\mathrm{softmax}\big(W_e\,\mathrm{LN}_f(h^{(L)}_t)\big): the same matrix WeW_e embeds and un-embeds

Post-LN to pre-LN. In GPT-1 every addition is followed by a LayerNorm, so the gradient reaching layer 1 has passed through 2L2L normalisations. Deep post-LN stacks need a careful learning-rate warm-up and become unstable as depth grows. GPT-2 put the norm on the branch instead. Now the residual trunk is a plain sum h(L)=h(0)+∑ℓfℓh^{(L)}=h^{(0)}+\sum_\ell f_\ell, and the gradient has an identity path from the loss to every layer. Because nothing on the trunk is normalised any more, its scale grows with depth, so GPT-2 adds a final LayerNorm LNf\mathrm{LN}_f before the head. Xiong et al. ("On Layer Normalization in the Transformer Architecture", ICML 2020) later showed that pre-LN gradients are well behaved at initialisation and can be trained without warm-up.

Learned positions. All three models add a learned table Wp∈RT×dW_p\in\mathbb R^{T\times d} instead of the sinusoids of Vaswani et al. (see Positional encoding). The table costs only TdTd parameters, 0.6% of GPT-2 small. The price is a hard context limit, 512, 1024 and 2048 tokens. Position T+1T+1 has no embedding, so the model cannot even be run past it. Later families replaced the table with Rotary positional embeddings (RoPE).

GELU. The MLP uses the Gaussian Error Linear Unit of Hendrycks and Gimpel (2016), GELU(x)=x Φ(x)\mathrm{GELU}(x)=x\,\Phi(x), in its tanh approximation 0.5x(1+tanh⁡(2/π(x+0.044715x3)))0.5x\big(1+\tanh(\sqrt{2/\pi}(x+0.044715x^3))\big). It is a smooth ReLU that lets small negative values through. GPT-1 was among the first large models to use it, and GELU became the default MLP activation until gated variants such as SwiGLU took over (see Llama).

Weight tying. The head WU∈RV×dW_U\in\mathbb R^{V\times d} that turns ht(L)h^{(L)}_t into logits is the token embedding WeW_e itself (Press and Wolf, "Using the Output Embedding to Improve Language Models", 2017; Inan et al., 2017). A token's input vector and its output vector are then the same row. Tying saves VdVd parameters, 38.6M in GPT-2 small, which is 31% of the model. It also shares the gradients. The input row of a rare token is otherwise updated only when that token appears in the input. Tied, it also gets the output layer's gradient from every softmax.

GPT-2's initialisation: scaling the residual path by 1/√N

GPT-2 initialises every weight from N(0,0.022)\mathcal N(0,0.02^2), as GPT-1 did, with one change. In the paper's words, it "scale[s] the weights of residual layers at initialization by a factor of 1/N1/\sqrt N where NN is the number of residual layers". There are two residual branches per block, so N=2LN=2L. In the released code and in Hugging Face this means the two output projections that write into the trunk, attn.c_proj and mlp.c_proj, get standard deviation 0.02/2L0.02/\sqrt{2L}.

h(L)=h(0)+∑n=1Nfn(LN(h(n−1))),Var h(L)=v0+∑n=1Ncn,cn∝σp2 ⇒ σp=0.02N gives ∑ncn=consth^{(L)}=h^{(0)}+\sum_{n=1}^{N}f_n\big(\mathrm{LN}(h^{(n-1)})\big),\qquad \mathrm{Var}\,h^{(L)}=v_0+\sum_{n=1}^{N}c_n,\qquad c_n\propto\sigma_p^2\ \Rightarrow\ \sigma_p=\frac{0.02}{\sqrt N}\ \text{gives}\ \sum_n c_n=\text{const}
fnf_n
the nn-th residual branch (attention or MLP); with independent weights its output is uncorrelated with everything already on the trunk
v0v_0
variance of the embeddings, 0.022+0.0120.02^2+0.01^2 (GPT-2's wte and wpe init)
cnc_n
variance one branch adds. Its input is LayerNorm-ed to unit scale, so cnc_n is the same at every depth and is proportional to σp2\sigma_p^2
N=2LN=2L
number of residual layers; 24 in GPT-2 small, 96 in GPT-2 XL, 192 in GPT-3 175B

In a pre-LN stack every branch sees a normalised input, so every branch adds about the same variance. Without the rescaling, the trunk's variance grows linearly with depth. A 48-layer model then starts with a residual stream many times larger than a 12-layer one, and each late layer's update is small relative to it. Dividing σp\sigma_p by N\sqrt N makes the total added variance independent of depth. Note what it does not fix: cnc_n still grows with width, because 0.020.02 is a fixed standard deviation, not a fan-in rule.

Published configurations

GPT-1
L=12L=12, d=768d=768, 12 heads, MLP 3072, context 512, BPE with 40,000 merges, about 117M parameters. BooksCorpus (about 7,000 unpublished books). Adam, peak lr 2.5⋅10−42.5\cdot10^{-4}, 2000 warm-up steps then cosine decay, 100 epochs of minibatches of 64 × 512 tokens, dropout 0.1. Fine-tuning adds the LM loss with weight 0.5
GPT-2
four sizes: (L,d)=(12,768)(L,d)=(12,768), (24,1024)(24,1024), (36,1280)(36,1280), (48,1600)(48,1600), with 12, 16, 20 and 25 heads. The paper calls them 117M, 345M, 762M and 1542M; the released checkpoints count 124M, 355M, 774M and 1558M with the embeddings
GPT-2 data and tokenizer
WebText: outbound links from Reddit with at least 3 karma, 8 million documents, 40 GB. Byte-level BPE (see Byte-pair encoding (BPE)) with 50,257 tokens that never merges across character categories. Context 1024, batch 512 sequences
GPT-3
eight sizes, all trained on 300B tokens with context 2048: Small 125M (12, 768), Medium 350M (24, 1024), Large 760M (24, 1536), XL 1.3B (24, 2048), 2.7B (32, 2560), 6.7B (32, 4096), 13B (40, 5140), 175B (96, 12288 with 96 heads of 128)
GPT-3 optimiser
Adam β1=0.9\beta_1=0.9, β2=0.95\beta_2=0.95, ϵ=10−8\epsilon=10^{-8}; gradient clipping at norm 1.0; weight decay 0.1; warm-up over 375M tokens, then cosine decay to 10% over 260B tokens; the batch grows from 32K tokens to its full size over the first 4–12B tokens. The peak lr falls from 6⋅10−46\cdot10^{-4} (Small) to 0.6⋅10−40.6\cdot10^{-4} (175B), and the batch grows from 0.5M to 3.2M tokens
GPT-3 data
filtered Common Crawl (410B tokens, 60% of the mix), WebText2 (22%), Books1 (8%), Books2 (8%), Wikipedia (3%); documents are packed into full 2048-token sequences, separated by an end-of-text token
GPT-3 compute
3.14⋅10233.14\cdot10^{23} FLOPs for 175B, about 3,640 petaflop/s-days

The GPT-3 table has two quirks worth knowing when you reproduce it. XL lists 24 heads of size 128 with d=2048d=2048, and 13B lists d=5140d=5140 with 40 heads of 128. Neither satisfies H⋅dhead=dH\cdot d_{head}=d, so the printed numbers cannot all be exact. The parameter counts follow from LL and dd alone, and those match the stated sizes.

per layer 12d² + 13d · tied head · C = 6·N·D · residual proj init N(0, (0.02/√(2L))²)GPT-2 117M: L = 12 · d = 768 · H = 12 · d_head = 64 · T = 1,024 · V = 50,257N = 124,439,808 (124.4M) · 12·L·d² = 84.9M = 99.9% of non-embedding 85.1M■ tokens wte 31.0%■ positions wpe 0.6%■ attention 22.8%■ MLP 45.5%■ LayerNorm 0.0%training FLOPs per token: 6N = 7.47e8 + attention 6·L·T·d = 5.66e7 (7.0% of the sum)D = 300B tokens: C = 6ND = 2.24e20 FLOPs = 2.6 PF-daystokens per parameter 2,411 · Chinchilla's rule of thumb 20N = 2.5Bbf16 weights 0.23 GiB · pre-LN + final LayerNormresidual-stream std at init vs layer (log), MLP branches0.010.11101000612layer ℓc_fc pre-activation vard·0.02² = 0.3072c_proj std σ_p0.02/√24 = 0.00408variance added per MLP5.45e-3std after 12 layers0.257other init, same layer1.253 (unscaled)— GPT-2 init 0.02/√N · — plain 0.02 · dashed: LayerNorm output scale 1 · Var(x_ℓ) = v₀ + ℓ·c
model
init
fig.Pick any of the 13 published configurations. The top half counts parameters exactly (biases, LayerNorms, learned positions, tied head) and prices training as C = 6ND. GPT-2 117M gives 124,439,808 parameters, the checkpoint's count. 12·L·d² = 84.9M is 99.9% of the 85.1M non-embedding weights, and the tied token table is 31.0% of the whole model. Trained on the GPT-3 recipe of 300B tokens it would cost 2.24e20 FLOPs (2.6 PF-days), about 2,411 tokens per parameter. GPT-3 175B counts 174.6B, of which attention is 33.2% and MLP 66.4%; 6ND = 3.14e23 FLOPs = 3,638 PF-days, the paper's 3.14e23. That is only 1.7 tokens per parameter, against 3.49T tokens by Chinchilla's later 20-per-parameter rule. The bottom half follows the residual stream at init through the MLP branches (exact in expectation; attention branches add far less). For GPT-2 117M each MLP adds variance 0.1308 at plain N(0, 0.02²) and the stream ends at std 1.253. With c_proj scaled to 0.02/√24 = 0.00408 it adds 0.00545 and ends at 0.257. For GPT-2 1542M the two inits give 5.581 vs 0.570, and for GPT-3 175B 67.182 vs 4.849. The scaling removes the growth with depth, but not with width: d·0.02² is already 4.9 at d = 12288. XL and 13B are flagged because the printed head size does not multiply out to d.

Worked example: what 1/√N buys GPT-2 small

GPT-2 small has L=12L=12, so N=24N=24 residual branches, and d=768d=768. Follow one MLP branch at initialisation. The LayerNorm output uu has unit mean square. The first layer has fan-in dd and std 0.020.02. GELU is the tanh form, and E[GELU(h)2]E[\mathrm{GELU}(h)^2] is taken by numerical integration.

quantityplain initGPT-2 init
pre-activation variance d·0.02²0.30720.3072
E[GELU(h)²], h ~ N(0, 0.3072)0.106460.10646
c_proj std σ_p0.020.02/√24 = 0.00408
variance added per branch c = 4d·σ_p²·E[GELU²]0.13080.00545
Var after 12 MLP branches, v₀ + 12c1.57030.0659
std of the residual stream1.2530.257
same at L = 48, d = 1600 (GPT-2 1542M)5.5810.570
Residual-stream variance from the MLP branches (computed in Python and checked against the figure)

Step by step

  1. 1Pre-activation: hj=∑iWjiuih_j=\sum_i W_{ji}u_i with ∑iui2=d\sum_iu_i^2=d, so Var hj=d⋅0.022=768⋅0.0004=0.3072\mathrm{Var}\,h_j=d\cdot0.02^2=768\cdot0.0004=0.3072, exactly.
  2. 2After GELU: E[GELU(h)2]=0.10646E[\mathrm{GELU}(h)^2]=0.10646. The projection has fan-in 4d=30724d=3072, so each output coordinate gets variance 3072⋅σp2⋅0.106463072\cdot\sigma_p^2\cdot0.10646.
  3. 3Plain: 3072⋅0.0004⋅0.10646=0.13083072\cdot0.0004\cdot0.10646=0.1308 per branch. Twelve branches plus v0=0.0005v_0=0.0005 give 1.57031.5703, std 1.2531.253. After the first branch alone the std is already 0.3620.362, sixteen times the embeddings' 0.0005=0.022\sqrt{0.0005}=0.022.
  4. 4Scaled: σp2=0.0004/24\sigma_p^2=0.0004/24, so each branch adds 0.1308/24=0.005450.1308/24=0.00545. The total is 0.0005+12⋅0.00545=0.06590.0005+12\cdot0.00545=0.0659, std 0.2570.257. For any depth, σp=0.02/2L\sigma_p=0.02/\sqrt{2L} makes the LL MLP branches add 0.1308/2=0.06540.1308/2=0.0654 in total, so a 6-layer and a 96-layer model of width 768 both end at 0.06590.0659.
  5. 5The attention branches add a little more on top. At init q⋅k≈0q\cdot k\approx0, so each head averages the values of all visible positions, and that average has a much smaller variance than one value.

GPT-3 and in-context learning

GPT-3 "uses the same model and architecture as GPT-2, including the modified initialization, pre-normalization, and reversible tokenization", with one exception: layers alternate between dense attention and the locally banded sparse attention of the Sparse Transformer (Child et al., 2019; see Sparse attention). Eight sizes from 125M to 175B were trained on the same 300B tokens so that the paper could plot each ability against scale.

The paper's central object is in-context learning. The task is specified entirely in the prompt and the weights never change. Zero-shot gives a natural-language instruction. One-shot adds one demonstration. Few-shot adds as many as fit in the 2048-token window, typically 10 to 100. On LAMBADA, 175B scored 76.2% zero-shot and 86.4% few-shot. On TriviaQA (closed book) it scored 64.3%, 68.0% and 71.2% at zero, one and few shots. The gap between zero-shot and few-shot grew with model size, which the authors read as larger models being better meta-learners.

y^=arg⁡max⁡y pθ(y∣instruction, (x1,y1),…,(xK,yK)⏟prompt, ≤2048 tokens, x),θ fixed\hat y=\arg\max_y\ p_\theta\big(y\mid \underbrace{\text{instruction},\ (x_1,y_1),\dots,(x_K,y_K)}_{\text{prompt, }\le 2048\text{ tokens}},\ x\big),\qquad \theta\ \text{fixed}
KK
number of demonstrations: 0, 1, or as many as fit (few-shot)
pθp_\theta
the pre-trained next-token model; nothing is fine-tuned
arg⁡max⁡y\arg\max_y
for multiple-choice tasks the paper compares the likelihood of each completion; free-form answers use beam search
python
1import math, torch, torch.nn as nn, torch.nn.functional as F
2
3class Block(nn.Module):
4 def __init__(self, d, H):
5 super().__init__()
6 self.H = H
7 self.ln_1, self.ln_2 = nn.LayerNorm(d), nn.LayerNorm(d)
8 self.c_attn = nn.Linear(d, 3 * d) # q, k, v in one matmul
9 self.attn_proj = nn.Linear(d, d) # writes into the residual stream
10 self.c_fc = nn.Linear(d, 4 * d)
11 self.mlp_proj = nn.Linear(4 * d, d) # writes into the residual stream
12
13 def forward(self, x):
14 B, T, d = x.shape
15 q, k, v = self.c_attn(self.ln_1(x)).split(d, dim=-1) # pre-LN
16 q, k, v = (t.view(B, T, self.H, d // self.H).transpose(1, 2) for t in (q, k, v))
17 a = F.scaled_dot_product_attention(q, k, v, is_causal=True)
18 x = x + self.attn_proj(a.transpose(1, 2).reshape(B, T, d))
19 return x + self.mlp_proj(F.gelu(self.c_fc(self.ln_2(x)), approximate="tanh"))
20
21class GPT2(nn.Module):
22 def __init__(self, V=50257, T=1024, L=12, d=768, H=12):
23 super().__init__()
24 self.wte, self.wpe = nn.Embedding(V, d), nn.Embedding(T, d) # learned positions
25 self.blocks = nn.ModuleList(Block(d, H) for _ in range(L))
26 self.ln_f = nn.LayerNorm(d) # added in GPT-2
27 self.head = nn.Linear(d, V, bias=False)
28 self.head.weight = self.wte.weight # weight tying
29 for name, p in self.named_parameters():
30 if p.dim() == 1: # biases 0, LayerNorm gains 1
31 nn.init.zeros_(p) if name.endswith("bias") else nn.init.ones_(p)
32 continue
33 std = 0.02 / math.sqrt(2 * L) if name.endswith("proj.weight") else 0.02
34 nn.init.normal_(p, 0.0, std) # residual projections get 0.02/sqrt(N), N = 2L
35 nn.init.normal_(self.wpe.weight, 0.0, 0.01)
36
37 def forward(self, idx, targets=None):
38 pos = torch.arange(idx.size(1), device=idx.device)
39 x = self.wte(idx) + self.wpe(pos)
40 for blk in self.blocks:
41 x = blk(x)
42 logits = self.head(self.ln_f(x))
43 loss = None if targets is None else F.cross_entropy(logits.flatten(0, 1), targets.flatten())
44 return logits, loss
45
46model = GPT2()
47print(sum(p.numel() for p in model.parameters())) # tied weights are counted once: 124,439,808
48
49# The released checkpoint, few-shot style: the "training data" is in the prompt.
50from transformers import GPT2LMHeadModel, GPT2TokenizerFast
51tok, lm = GPT2TokenizerFast.from_pretrained("gpt2"), GPT2LMHeadModel.from_pretrained("gpt2")
52prompt = "sea otter => loutre de mer\ncheese => fromage\npeppermint =>"
53out = lm.generate(**tok(prompt, return_tensors="pt"), max_new_tokens=5, do_sample=False)
54print(tok.decode(out[0])) # 124M is far too small to do this reliably; 175B mostly can
A GPT-2 block in PyTorch with pre-LN, learned positions, tied head and the 1/√(2L) init, plus the same count from Hugging Face (not run here; the 124,439,808 comes from the formula above)

Use when

  • +One architecture and one objective for every task; only the prompt or a linear head changes
  • +Pre-LN with a final LayerNorm and depth-scaled init trains stably to 96 layers
  • +Tied embeddings save V·d parameters, 31% of GPT-2 small
  • +The parameter count is predictable: 12·L·d² is 99.9% of the non-embedding weights
  • +Few-shot in-context learning at 175B, with no gradient updates

Avoid when

  • −Learned positions fix the context at 512/1024/2048 tokens and do not extrapolate
  • −Full multi-head attention: the KV cache of 175B is 4.5 MiB per token (see KV cache)
  • −Biases everywhere and a 4d GELU MLP; later recipes (RMSNorm, SwiGLU, no biases) do better per FLOP
  • −GPT-3's 300B tokens are far below the compute-optimal amount for 175B parameters
  • −Few-shot accuracy is sensitive to the choice, format and order of the examples

Check yourself

Why did GPT-2 need a final LayerNorm before the head, when GPT-1 did not?

Check yourself

GPT-2 1542M has L = 48. What standard deviation does it use for attn.c_proj and mlp.c_proj at init?

Remember

  • ✓GPT-1 → GPT-2 → GPT-3 kept one block (masked MHA + GELU MLP of width 4d4d, learned positions, tied head) and grew from 117M to 175B parameters.
  • ✓GPT-2 moved LayerNorm to the branch inputs (pre-LN), added a final LayerNorm, and scaled residual projections by 1/N1/\sqrt N with N=2LN=2L. That makes the variance the branches add at init independent of depth.
  • ✓Parameters per layer are 12d2+13d12d^2+13d; 12Ld212Ld^2 is 99.9% of the non-embedding weights, and C≈6NDC\approx6ND reproduces GPT-3's 3.14⋅10233.14\cdot10^{23} FLOPs.
  • ✓GPT-3: 8 sizes all on 300B tokens, context 2048, alternating dense and locally banded sparse attention, Adam β₂ = 0.95, cosine decay to 10%.
  • ✓In-context learning: the task is defined by instructions and K examples in the prompt, with frozen weights; the few-shot gain grows with scale.
← back to Deep Learning roadmap