# GPT architecture

> GPT-1 (2018), GPT-2 (2019) and GPT-3 (2020) are one decoder-only design scaled 1,500×: learned positions, GELU MLPs, a tied head, pre-LN with a final LayerNorm and residual init ÷√N from GPT-2 on, and at 175B (96 layers, d = 12288, 300B tokens) few-shot learning from the prompt.

Source: https://www.mlfoundry.in/learn/gpt-architecture  
Section: Deep Learning › Large Language Models (LLMs)  
Level: core · 15 min read · updated 2026-10-13

**GPT architecture:** is the decoder-only transformer design of OpenAI's three Generative Pre-trained Transformer papers: Radford, Narasimhan, Salimans and Sutskever, "Improving Language Understanding by Generative Pre-Training" (2018, **GPT-1**); Radford, Wu, Child, Luan, Amodei and Sutskever, "Language Models are Unsupervised Multitask Learners" (2019, **GPT-2**); and Brown et al., "Language Models are Few-Shot Learners" (NeurIPS 2020, **GPT-3**). Each is a stack of causal self-attention and GELU MLP blocks over learned token and position embeddings, trained only to predict the next token, with the output layer tied to the input embedding. The three papers change very little in the block. What changes is the scale, from 117M to 175B parameters, and how the model is used: fine-tuned per task, then zero-shot, then few-shot from examples in the prompt.

In 2017 the best NLP systems were task-specific architectures trained on labelled data, sometimes initialised with **Word embeddings** or with **Contextual embeddings** such as ELMo. GPT-1 asked whether one generic network, pre-trained as a language model on unlabelled books, could be fine-tuned to every task with only a new linear output layer. The answer was yes: it improved the state of the art on 9 of the 12 tasks it was tested on. GPT-2 dropped the fine-tuning and showed that a larger model trained on more varied web text already performs tasks zero-shot when the prompt is phrased as text. GPT-3 scaled that model about 100× more and found that a handful of worked examples in the prompt, with no gradient updates, is often enough.

The generic shape of this model is covered in **Decoder-only transformer**, and the mask in **Causal attention**. This article is about the specific choices OpenAI made, why each was made, and what the published configurations look like.

## The block, generation by generation

> Figure: One block of each generation, at its largest published size. GPT-1 is the original post-LN Transformer decoder: LayerNorm sits on the residual trunk after each addition, so every activation passes through it. GPT-2 moves LayerNorm onto the branch, in front of attention and the MLP, so the trunk is a pure sum. It adds a final LayerNorm before the head and scales the residual output projections by 1/√N at init. GPT-3 keeps the GPT-2 block and alternates dense causal attention with a locally banded sparse pattern from layer to layer (the band drawn here is illustrative; the paper does not give its width). All three use learned absolute positions, a GELU MLP four times wider than d, and an output head tied to the token embedding. Hover the bottom label to highlight every LayerNorm.

$$
\textbf{GPT-1 (post-LN): }\ a=\mathrm{LN}\big(h+\mathrm{MHA}(h)\big),\quad h'=\mathrm{LN}\big(a+\mathrm{MLP}(a)\big)\qquad \textbf{GPT-2/3 (pre-LN): }\ a=h+\mathrm{MHA}(\mathrm{LN}(h)),\quad h'=a+\mathrm{MLP}(\mathrm{LN}(a))
$$

- $h$: residual stream, $h^{(0)}=W_e[x]+W_p[1{:}T]$ (token plus learned position embeddings)
- $\mathrm{MHA}$: masked multi-head self-attention, $H$ heads of size $d/H$ (see **Multi-head attention**)
- $\mathrm{MLP}(u)$: $W_2\,\mathrm{GELU}(W_1u+b_1)+b_2$ with $W_1\in\mathbb R^{4d\times d}$
- $\mathrm{LN}$: LayerNorm with gain and bias; pre-LN models apply one more, $\mathrm{LN}_f$, before the head
- $p(x_{t+1}\mid x_{\le t})$: $\mathrm{softmax}\big(W_e\,\mathrm{LN}_f(h^{(L)}_t)\big)$: the same matrix $W_e$ embeds and un-embeds

**Post-LN to pre-LN.** In GPT-1 every addition is followed by a LayerNorm, so the gradient reaching layer 1 has passed through $2L$ normalisations. Deep post-LN stacks need a careful learning-rate warm-up and become unstable as depth grows. GPT-2 put the norm on the branch instead. Now the residual trunk is a plain sum $h^{(L)}=h^{(0)}+\sum_\ell f_\ell$, and the gradient has an identity path from the loss to every layer. Because nothing on the trunk is normalised any more, its scale grows with depth, so GPT-2 adds a final LayerNorm $\mathrm{LN}_f$ before the head. Xiong et al. ("On Layer Normalization in the Transformer Architecture", ICML 2020) later showed that pre-LN gradients are well behaved at initialisation and can be trained without warm-up.

**Learned positions.** All three models add a learned table $W_p\in\mathbb R^{T\times d}$ instead of the sinusoids of Vaswani et al. (see **Positional encoding**). The table costs only $Td$ parameters, 0.6% of GPT-2 small. The price is a hard context limit, 512, 1024 and 2048 tokens. Position $T+1$ has no embedding, so the model cannot even be run past it. Later families replaced the table with **Rotary positional embeddings (RoPE)**.

**GELU.** The MLP uses the Gaussian Error Linear Unit of Hendrycks and Gimpel (2016), $\mathrm{GELU}(x)=x\,\Phi(x)$, in its tanh approximation $0.5x\big(1+\tanh(\sqrt{2/\pi}(x+0.044715x^3))\big)$. It is a smooth ReLU that lets small negative values through. GPT-1 was among the first large models to use it, and GELU became the default MLP activation until gated variants such as SwiGLU took over (see **Llama**).

**Weight tying.** The head $W_U\in\mathbb R^{V\times d}$ that turns $h^{(L)}_t$ into logits is the token embedding $W_e$ itself (Press and Wolf, "Using the Output Embedding to Improve Language Models", 2017; Inan et al., 2017). A token's input vector and its output vector are then the same row. Tying saves $Vd$ parameters, 38.6M in GPT-2 small, which is 31% of the model. It also shares the gradients. The input row of a rare token is otherwise updated only when that token appears in the input. Tied, it also gets the output layer's gradient from every softmax.

## GPT-2's initialisation: scaling the residual path by 1/√N

GPT-2 initialises every weight from $\mathcal N(0,0.02^2)$, as GPT-1 did, with one change. In the paper's words, it "scale[s] the weights of residual layers at initialization by a factor of $1/\sqrt N$ where $N$ is the number of residual layers". There are two residual branches per block, so $N=2L$. In the released code and in Hugging Face this means the two output projections that write into the trunk, `attn.c_proj` and `mlp.c_proj`, get standard deviation $0.02/\sqrt{2L}$.

$$
h^{(L)}=h^{(0)}+\sum_{n=1}^{N}f_n\big(\mathrm{LN}(h^{(n-1)})\big),\qquad \mathrm{Var}\,h^{(L)}=v_0+\sum_{n=1}^{N}c_n,\qquad c_n\propto\sigma_p^2\ \Rightarrow\ \sigma_p=\frac{0.02}{\sqrt N}\ \text{gives}\ \sum_n c_n=\text{const}
$$

- $f_n$: the $n$-th residual branch (attention or MLP); with independent weights its output is uncorrelated with everything already on the trunk
- $v_0$: variance of the embeddings, $0.02^2+0.01^2$ (GPT-2's `wte` and `wpe` init)
- $c_n$: variance one branch adds. Its input is LayerNorm-ed to unit scale, so $c_n$ is the same at every depth and is proportional to $\sigma_p^2$
- $N=2L$: number of residual layers; 24 in GPT-2 small, 96 in GPT-2 XL, 192 in GPT-3 175B

In a pre-LN stack every branch sees a normalised input, so every branch adds about the same variance. Without the rescaling, the trunk's variance grows linearly with depth. A 48-layer model then starts with a residual stream many times larger than a 12-layer one, and each late layer's update is small relative to it. Dividing $\sigma_p$ by $\sqrt N$ makes the total added variance independent of depth. Note what it does not fix: $c_n$ still grows with width, because $0.02$ is a fixed standard deviation, not a fan-in rule.

**Published configurations**

- **GPT-1:** $L=12$, $d=768$, 12 heads, MLP 3072, context 512, BPE with 40,000 merges, about 117M parameters. BooksCorpus (about 7,000 unpublished books). Adam, peak lr $2.5\cdot10^{-4}$, 2000 warm-up steps then cosine decay, 100 epochs of minibatches of 64 × 512 tokens, dropout 0.1. Fine-tuning adds the LM loss with weight 0.5
- **GPT-2:** four sizes: $(L,d)=(12,768)$, $(24,1024)$, $(36,1280)$, $(48,1600)$, with 12, 16, 20 and 25 heads. The paper calls them 117M, 345M, 762M and 1542M; the released checkpoints count 124M, 355M, 774M and 1558M with the embeddings
- **GPT-2 data and tokenizer:** WebText: outbound links from Reddit with at least 3 karma, 8 million documents, 40 GB. Byte-level BPE (see **Byte-pair encoding (BPE)**) with 50,257 tokens that never merges across character categories. Context 1024, batch 512 sequences
- **GPT-3:** eight sizes, all trained on 300B tokens with context 2048: Small 125M (12, 768), Medium 350M (24, 1024), Large 760M (24, 1536), XL 1.3B (24, 2048), 2.7B (32, 2560), 6.7B (32, 4096), 13B (40, 5140), 175B (96, 12288 with 96 heads of 128)
- **GPT-3 optimiser:** Adam $\beta_1=0.9$, $\beta_2=0.95$, $\epsilon=10^{-8}$; gradient clipping at norm 1.0; weight decay 0.1; warm-up over 375M tokens, then cosine decay to 10% over 260B tokens; the batch grows from 32K tokens to its full size over the first 4–12B tokens. The peak lr falls from $6\cdot10^{-4}$ (Small) to $0.6\cdot10^{-4}$ (175B), and the batch grows from 0.5M to 3.2M tokens
- **GPT-3 data:** filtered Common Crawl (410B tokens, 60% of the mix), WebText2 (22%), Books1 (8%), Books2 (8%), Wikipedia (3%); documents are packed into full 2048-token sequences, separated by an end-of-text token
- **GPT-3 compute:** $3.14\cdot10^{23}$ FLOPs for 175B, about 3,640 petaflop/s-days

The GPT-3 table has two quirks worth knowing when you reproduce it. XL lists 24 heads of size 128 with $d=2048$, and 13B lists $d=5140$ with 40 heads of 128. Neither satisfies $H\cdot d_{head}=d$, so the printed numbers cannot all be exact. The parameter counts follow from $L$ and $d$ alone, and those match the stated sizes.

> Figure: Pick any of the 13 published configurations. The top half counts parameters exactly (biases, LayerNorms, learned positions, tied head) and prices training as C = 6ND. GPT-2 117M gives 124,439,808 parameters, the checkpoint's count. 12·L·d² = 84.9M is 99.9% of the 85.1M non-embedding weights, and the tied token table is 31.0% of the whole model. Trained on the GPT-3 recipe of 300B tokens it would cost 2.24e20 FLOPs (2.6 PF-days), about 2,411 tokens per parameter. GPT-3 175B counts 174.6B, of which attention is 33.2% and MLP 66.4%; 6ND = 3.14e23 FLOPs = 3,638 PF-days, the paper's 3.14e23. That is only 1.7 tokens per parameter, against 3.49T tokens by Chinchilla's later 20-per-parameter rule. The bottom half follows the residual stream at init through the MLP branches (exact in expectation; attention branches add far less). For GPT-2 117M each MLP adds variance 0.1308 at plain N(0, 0.02²) and the stream ends at std 1.253. With c_proj scaled to 0.02/√24 = 0.00408 it adds 0.00545 and ends at 0.257. For GPT-2 1542M the two inits give 5.581 vs 0.570, and for GPT-3 175B 67.182 vs 4.849. The scaling removes the growth with depth, but not with width: d·0.02² is already 4.9 at d = 12288. XL and 13B are flagged because the printed head size does not multiply out to d.

## Worked example: what 1/√N buys GPT-2 small

GPT-2 small has $L=12$, so $N=24$ residual branches, and $d=768$. Follow one MLP branch at initialisation. The LayerNorm output $u$ has unit mean square. The first layer has fan-in $d$ and std $0.02$. GELU is the tanh form, and $E[\mathrm{GELU}(h)^2]$ is taken by numerical integration.

Residual-stream variance from the MLP branches (computed in Python and checked against the figure)

| quantity | plain init | GPT-2 init |
| --- | --- | --- |
| pre-activation variance d·0.02² | 0.3072 | 0.3072 |
| E[GELU(h)²], h ~ N(0, 0.3072) | 0.10646 | 0.10646 |
| c_proj std σ_p | 0.02 | 0.02/√24 = 0.00408 |
| variance added per branch c = 4d·σ_p²·E[GELU²] | 0.1308 | 0.00545 |
| Var after 12 MLP branches, v₀ + 12c | 1.5703 | 0.0659 |
| std of the residual stream | 1.253 | 0.257 |
| same at L = 48, d = 1600 (GPT-2 1542M) | 5.581 | 0.570 |

**Step by step**

1. Pre-activation: $h_j=\sum_i W_{ji}u_i$ with $\sum_iu_i^2=d$, so $\mathrm{Var}\,h_j=d\cdot0.02^2=768\cdot0.0004=0.3072$, exactly.
2. After GELU: $E[\mathrm{GELU}(h)^2]=0.10646$. The projection has fan-in $4d=3072$, so each output coordinate gets variance $3072\cdot\sigma_p^2\cdot0.10646$.
3. Plain: $3072\cdot0.0004\cdot0.10646=0.1308$ per branch. Twelve branches plus $v_0=0.0005$ give $1.5703$, std $1.253$. After the first branch alone the std is already $0.362$, sixteen times the embeddings' $\sqrt{0.0005}=0.022$.
4. Scaled: $\sigma_p^2=0.0004/24$, so each branch adds $0.1308/24=0.00545$. The total is $0.0005+12\cdot0.00545=0.0659$, std $0.257$. For any depth, $\sigma_p=0.02/\sqrt{2L}$ makes the $L$ MLP branches add $0.1308/2=0.0654$ in total, so a 6-layer and a 96-layer model of width 768 both end at $0.0659$.
5. The attention branches add a little more on top. At init $q\cdot k\approx0$, so each head averages the values of all visible positions, and that average has a much smaller variance than one value.

## GPT-3 and in-context learning

GPT-3 "uses the same model and architecture as GPT-2, including the modified initialization, pre-normalization, and reversible tokenization", with one exception: layers alternate between dense attention and the locally banded sparse attention of the Sparse Transformer (Child et al., 2019; see **Sparse attention**). Eight sizes from 125M to 175B were trained on the same 300B tokens so that the paper could plot each ability against scale.

The paper's central object is **in-context learning**. The task is specified entirely in the prompt and the weights never change. *Zero-shot* gives a natural-language instruction. *One-shot* adds one demonstration. *Few-shot* adds as many as fit in the 2048-token window, typically 10 to 100. On LAMBADA, 175B scored 76.2% zero-shot and 86.4% few-shot. On TriviaQA (closed book) it scored 64.3%, 68.0% and 71.2% at zero, one and few shots. The gap between zero-shot and few-shot grew with model size, which the authors read as larger models being better meta-learners.

$$
\hat y=\arg\max_y\ p_\theta\big(y\mid \underbrace{\text{instruction},\ (x_1,y_1),\dots,(x_K,y_K)}_{\text{prompt, }\le 2048\text{ tokens}},\ x\big),\qquad \theta\ \text{fixed}
$$

- $K$: number of demonstrations: 0, 1, or as many as fit (few-shot)
- $p_\theta$: the pre-trained next-token model; nothing is fine-tuned
- $\arg\max_y$: for multiple-choice tasks the paper compares the likelihood of each completion; free-form answers use beam search

```python
import math, torch, torch.nn as nn, torch.nn.functional as F

class Block(nn.Module):
    def __init__(self, d, H):
        super().__init__()
        self.H = H
        self.ln_1, self.ln_2 = nn.LayerNorm(d), nn.LayerNorm(d)
        self.c_attn = nn.Linear(d, 3 * d)          # q, k, v in one matmul
        self.attn_proj = nn.Linear(d, d)           # writes into the residual stream
        self.c_fc = nn.Linear(d, 4 * d)
        self.mlp_proj = nn.Linear(4 * d, d)        # writes into the residual stream

    def forward(self, x):
        B, T, d = x.shape
        q, k, v = self.c_attn(self.ln_1(x)).split(d, dim=-1)              # pre-LN
        q, k, v = (t.view(B, T, self.H, d // self.H).transpose(1, 2) for t in (q, k, v))
        a = F.scaled_dot_product_attention(q, k, v, is_causal=True)
        x = x + self.attn_proj(a.transpose(1, 2).reshape(B, T, d))
        return x + self.mlp_proj(F.gelu(self.c_fc(self.ln_2(x)), approximate="tanh"))

class GPT2(nn.Module):
    def __init__(self, V=50257, T=1024, L=12, d=768, H=12):
        super().__init__()
        self.wte, self.wpe = nn.Embedding(V, d), nn.Embedding(T, d)   # learned positions
        self.blocks = nn.ModuleList(Block(d, H) for _ in range(L))
        self.ln_f = nn.LayerNorm(d)                                   # added in GPT-2
        self.head = nn.Linear(d, V, bias=False)
        self.head.weight = self.wte.weight                            # weight tying
        for name, p in self.named_parameters():
            if p.dim() == 1:                       # biases 0, LayerNorm gains 1
                nn.init.zeros_(p) if name.endswith("bias") else nn.init.ones_(p)
                continue
            std = 0.02 / math.sqrt(2 * L) if name.endswith("proj.weight") else 0.02
            nn.init.normal_(p, 0.0, std)           # residual projections get 0.02/sqrt(N), N = 2L
        nn.init.normal_(self.wpe.weight, 0.0, 0.01)

    def forward(self, idx, targets=None):
        pos = torch.arange(idx.size(1), device=idx.device)
        x = self.wte(idx) + self.wpe(pos)
        for blk in self.blocks:
            x = blk(x)
        logits = self.head(self.ln_f(x))
        loss = None if targets is None else F.cross_entropy(logits.flatten(0, 1), targets.flatten())
        return logits, loss

model = GPT2()
print(sum(p.numel() for p in model.parameters()))   # tied weights are counted once: 124,439,808

# The released checkpoint, few-shot style: the "training data" is in the prompt.
from transformers import GPT2LMHeadModel, GPT2TokenizerFast
tok, lm = GPT2TokenizerFast.from_pretrained("gpt2"), GPT2LMHeadModel.from_pretrained("gpt2")
prompt = "sea otter => loutre de mer\ncheese => fromage\npeppermint =>"
out = lm.generate(**tok(prompt, return_tensors="pt"), max_new_tokens=5, do_sample=False)
print(tok.decode(out[0]))   # 124M is far too small to do this reliably; 175B mostly can
```

A GPT-2 block in PyTorch with pre-LN, learned positions, tied head and the 1/√(2L) init, plus the same count from Hugging Face (not run here; the 124,439,808 comes from the formula above)

> **The context limit is in the weights:** With a learned position table, position 1025 of GPT-2 has no embedding at all. Running longer inputs means truncating, sliding a window, or training new rows. Unlike **Rotary positional embeddings (RoPE)** or **ALiBi**, there is no frequency to rescale. This is one reason later open families dropped learned positions (see **Context window optimization**).

> **Pre-LN moves the norm off the highway:** Post-LN normalises the trunk itself. Pre-LN leaves the trunk as a running sum and normalises only what each branch reads. That gives an identity path for gradients, but it also means the trunk's scale is set by initialisation and depth. GPT-2's two additions, the final LayerNorm and the $1/\sqrt N$ init, are exactly the fixes that pre-LN needs.

> **GPT-3 was under-trained by later standards:** Every GPT-3 size saw 300B tokens, 1.7 per parameter at 175B. Hoffmann et al. ("Training Compute-Optimal Large Language Models", 2022) found that compute-optimal training uses about 20 tokens per parameter. Their 70B Chinchilla, trained on 1.4T tokens, outperformed the much larger Gopher (280B) that had been trained on 300B tokens. **Llama** pushed further, training a 7B model on 1T tokens.

**Strengths**

- One architecture and one objective for every task; only the prompt or a linear head changes
- Pre-LN with a final LayerNorm and depth-scaled init trains stably to 96 layers
- Tied embeddings save V·d parameters, 31% of GPT-2 small
- The parameter count is predictable: 12·L·d² is 99.9% of the non-embedding weights
- Few-shot in-context learning at 175B, with no gradient updates

**Limitations**

- Learned positions fix the context at 512/1024/2048 tokens and do not extrapolate
- Full multi-head attention: the KV cache of 175B is 4.5 MiB per token (see **KV cache**)
- Biases everywhere and a 4d GELU MLP; later recipes (RMSNorm, SwiGLU, no biases) do better per FLOP
- GPT-3's 300B tokens are far below the compute-optimal amount for 175B parameters
- Few-shot accuracy is sensitive to the choice, format and order of the examples

**Question:** Why did GPT-2 need a final LayerNorm before the head, when GPT-1 did not?

- A. GPT-2 has a larger vocabulary
- B. In pre-LN the trunk is never normalised, so its scale grows with depth; GPT-1's post-LN already ends in a LayerNorm
- C. The tied head requires unit-norm inputs
- D. Byte-level BPE produces larger activations

**Answer:** B. In pre-LN the trunk is never normalised, so its scale grows with depth; GPT-1's post-LN already ends in a LayerNorm — Post-LN applies LayerNorm after every residual addition, so the last block's output is already normalised. Pre-LN only normalises the branch inputs, which leaves the trunk as a raw sum $h^{(0)}+\sum f_n$. Its scale depends on depth, so $\mathrm{LN}_f$ is added before the un-embedding.

**Question:** GPT-2 1542M has L = 48. What standard deviation does it use for attn.c_proj and mlp.c_proj at init?

- A. 0.02
- B. 0.02/√48 ≈ 0.00289
- C. 0.02/√96 ≈ 0.00204
- D. 0.02/48

**Answer:** C. 0.02/√96 ≈ 0.00204 — $N$ counts residual layers, two per block, so $N=96$ and $\sigma_p=0.02/\sqrt{96}=0.00204$. The variance each branch adds is then divided by 96, so the 96 contributions sum to about what a single unscaled branch would add.

**Key points**

- GPT-1 → GPT-2 → GPT-3 kept one block (masked MHA + GELU MLP of width $4d$, learned positions, tied head) and grew from 117M to 175B parameters.
- GPT-2 moved LayerNorm to the branch inputs (pre-LN), added a final LayerNorm, and scaled residual projections by $1/\sqrt N$ with $N=2L$. That makes the variance the branches add at init independent of depth.
- Parameters per layer are $12d^2+13d$; $12Ld^2$ is 99.9% of the non-embedding weights, and $C\approx6ND$ reproduces GPT-3's $3.14\cdot10^{23}$ FLOPs.
- GPT-3: 8 sizes all on 300B tokens, context 2048, alternating dense and locally banded sparse attention, Adam β₂ = 0.95, cosine decay to 10%.
- In-context learning: the task is defined by instructions and K examples in the prompt, with frozen weights; the few-shot gain grows with scale.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
