# Transformer encoder

> A Transformer encoder is a stack of identical layers, each a multi-head self-attention sublayer and a position-wise feed-forward sublayer wrapped in residual connections and LayerNorm. Every token attends to every other token, and each layer has about 12d² parameters.

Source: https://www.mlfoundry.in/learn/transformer-encoder  
Section: Deep Learning › Attention and Transformer Architectures  
Level: core · 14 min read · updated 2026-10-12

**Transformer encoder:** is a stack of $N$ identical layers that maps a sequence of $n$ input vectors $x_1,\dots,x_n\in\mathbb R^{d}$ (token embeddings plus position information) to $n$ **contextual** vectors of the same width. Each layer has two sublayers: **multi-head self-attention**, in which every position attends to every position, and a **position-wise feed-forward network** applied to each token separately. Each sublayer is wrapped in a residual connection and a **LayerNorm**. There is no causal mask. The only mask hides padding.

The encoder is the left half of the architecture in Vaswani et al., "Attention Is All You Need" (NeurIPS 2017), the model described in **Encoder–decoder transformer**. Its job there was to read a source sentence and hand the decoder a memory of $n$ vectors. Recurrence passes information one step at a time, so position $i$ needs $O(|i-j|)$ steps to reach position $j$. In an attention layer every pair is one step apart, and all positions are computed in parallel. The paper's encoder had $N=6$ layers of width $d_{\text{model}}=512$, 8 heads and an inner FFN width of 2048.

The encoder alone became the standard text representation model with **BERT** (Devlin, Chang, Lee and Toutanova, "BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding", NAACL 2019). BERT pre-trained an encoder by predicting masked tokens from both sides at once, which a left-to-right decoder cannot do. The same block, with patches as tokens, is the backbone of **Vision Transformers (ViT)**.

## One layer

$$
\begin{aligned}h&=\mathrm{LN}\big(x+\mathrm{Dropout}(\mathrm{MHA}(x,x,x;\ \text{pad mask}))\big)\\ y&=\mathrm{LN}\big(h+\mathrm{Dropout}(\mathrm{FFN}(h))\big),\qquad \mathrm{FFN}(h)=\max(0,\,hW_1+b_1)\,W_2+b_2\end{aligned}
$$

- $x\in\mathbb R^{n\times d}$: the layer input; for layer 1 the embeddings $\times\sqrt d$ plus positional encodings (see **Positional encoding**)
- $\mathrm{MHA}$: multi-head self-attention: queries, keys and values all come from $x$; $h$ heads of width $d/h$ (see **Multi-head attention**)
- $\text{pad mask}$: keys at padding positions get score $-\infty$; nothing else is masked
- $W_1\in\mathbb R^{d\times d_{ff}},\ W_2\in\mathbb R^{d_{ff}\times d}$: the FFN, applied to each row of $h$ independently; $d_{ff}=4d$ in the original and in BERT
- $\mathrm{LN}$: LayerNorm over the $d$ features of each token, with learned gain and bias
- $\mathrm{Dropout}$: applied to each sublayer output before the addition; $P_{\text{drop}}=0.1$ in the base model

The two sublayers do different jobs. Self-attention is the only place where tokens exchange information: row $i$ of the output is a weighted average of value vectors from all positions (see **Self-attention**). The FFN mixes features *within* a token and holds about two thirds of the layer's parameters. The residual connections keep a direct path from the input to the output of every layer, and each sublayer adds an update to it.

Without position information the layer is **permutation-equivariant**: permute the input rows and the output rows are permuted the same way. Attention compares contents only. Word order enters only through the positional encodings added at the bottom, or through relative position terms inside attention (see **Rotary positional embeddings (RoPE)** and **ALiBi**).

## Post-LN and pre-LN

$$
\text{post-LN:}\ \ x_{\ell+1}=\mathrm{LN}\big(x_\ell+F(x_\ell)\big)\qquad\qquad \text{pre-LN:}\ \ x_{\ell+1}=x_\ell+F\big(\mathrm{LN}(x_\ell)\big),\quad \text{output}=\mathrm{LN}(x_N)
$$

- $F$: a sublayer (attention or FFN, with its dropout)
- $\text{post-LN}$: the 2017 Transformer and BERT: normalise after the residual addition
- $\text{pre-LN}$: GPT-2, ViT, T5 and most large models since: normalise the sublayer input and leave the residual sum alone; one extra LN after the last layer

In post-LN the residual path itself passes through a LayerNorm in every sublayer, so the input of layer 1 reaches the output only through $2N$ normalisations. In pre-LN the path from $x_0$ to $x_N$ is a pure sum, $x_N=x_0+\sum_\ell F_\ell(\cdot)$, and the gradient always contains an identity term. Xiong et al., "On Layer Normalization in the Transformer Architecture" (ICML 2020), showed that at initialisation the post-LN gradients of the parameters near the output are large. This is why post-LN needs a learning-rate warm-up. Pre-LN gradients are well behaved, and pre-LN models trained without warm-up in their experiments. Wang et al. (ACL 2019) had earlier used pre-norm to train 30-layer translation encoders.

Pre-LN has its own costs. The residual stream is never normalised inside the stack, so its scale grows with depth. Each later layer then changes it relatively less, and the final LayerNorm is mandatory. When post-LN trains successfully it often reaches slightly better quality. Liu et al. (EMNLP 2020) traced post-LN's instability to its strong dependence on the residual branch and proposed a re-scaled initialisation (Admin) to keep post-LN's quality with pre-LN's stability.

> Figure: A real 8-token encoder at initialisation (d = 16, 2 heads, d_ff = 64, ReLU, weights with std 1/√fan-in). The post-LN and pre-LN stacks share the same weights and differ only in where LayerNorm sits. Post-LN: the residual RMS is exactly 1.00 after every layer, each layer's update is as large as its input (‖F‖/‖x‖ = 0.87 in layer 1, 1.36 in layer 12), and the cosine between a token's state and its input embedding falls from 0.56 after layer 1 to 0.07 after layer 12. Pre-LN with the same weights: the stream's RMS grows from 1.40 at the input to 1.83 after layer 1 and 4.81 after layer 12 (7.74 at 24 layers). The relative update shrinks from 0.81 to 0.24 (0.15 at 24 layers), and the cosine to the input stays at 0.40. The heat map shows layer ℓ's head-1 weights. There is no causal triangle, and the two [PAD] key columns get exactly 0 weight from every real query. The bottom lines count parameters: 3,152,384 per Transformer-base layer, and 85.05M for BERT-base's 12 layers, which with its embeddings and pooler gives 109.48M.

## Parameter count

$$
P_{\text{layer}}=\underbrace{4d^2+4d}_{W_Q,W_K,W_V,W_O}+\underbrace{2\,d\,d_{ff}+d_{ff}+d}_{\text{FFN}}+\underbrace{4d}_{2\ \mathrm{LN}}\ \overset{d_{ff}=4d}{=}\ 12d^2+13d
$$

- $4d^2$: the four $d\times d$ projections; the number of heads does not change it, because $h$ heads of width $d/h$ together span $d$
- $8d^2$: the FFN with $d_{ff}=4d$, twice the attention parameters
- $13d$: biases and LayerNorm gains and biases; negligible for $d$ in the hundreds

Compute follows the same split. Per token and layer, the matrix multiplies cost about $2\cdot12d^2=24d^2$ FLOPs, independent of $n$. The attention scores and the weighted sum add $4nd$ FLOPs per token, so the quadratic part matters only once $n$ is comparable to $6d$: 3072 tokens for $d=512$. Long-input encoders change exactly this term (see **Sparse attention**, **Longformer**, **BigBird**).

Exact parameter counts (computed; biases and LayerNorm included)

| model | N | d | d_ff | heads | per layer | N layers | total |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Transformer base encoder | 6 | 512 | 2048 | 8 | 3,152,384 | 18.91M | + shared 37K × 512 embedding |
| Transformer big encoder | 6 | 1024 | 4096 | 16 | 12,596,224 | 75.58M | + shared embedding |
| BERT-base | 12 | 768 | 3072 | 12 | 7,087,872 | 85.05M | 109.48M (paper: 110M) |
| BERT-large | 24 | 1024 | 4096 | 16 | 12,596,224 | 302.31M | 335.14M (paper: 340M) |

The BERT totals add the 30,522-token WordPiece table, 512 learned position embeddings, 2 segment embeddings, the embedding LayerNorm and the $d\times d$ pooler. For BERT-base the embeddings are 23.84M of the 109.48M, more than 20%.

**BERT (Devlin et al. 2019)**

- **Sizes:** BERT-base $L=12$, $H=768$, $A=12$, 110M; BERT-large $L=24$, $H=1024$, $A=16$, 340M; FFN width $4H$, GELU
- **Input:** WordPiece, 30,000-token vocabulary; [CLS] first, [SEP] between segments; token + segment + learned position embeddings; at most 512 tokens
- **Pre-training tasks:** masked LM: 15% of tokens chosen, of which 80% become [MASK], 10% a random token, 10% unchanged; next-sentence prediction on [CLS]
- **Data:** BooksCorpus (800M words) and English Wikipedia (2,500M words)
- **Schedule:** 1,000,000 steps of 256 sequences × 512 tokens (128,000 tokens per batch), about 40 epochs; Adam, learning rate $10^{-4}$, 10,000 warm-up steps
- **Results (BERT-large):** GLUE 80.5 (+7.7 points), MultiNLI 86.7%, SQuAD v1.1 test F1 93.2, SQuAD v2.0 test F1 83.1
- **LayerNorm:** post-LN, as in the 2017 Transformer

## Worked example: one sublayer, post-LN vs pre-LN

Take one token with $d=4$, $x=(2,\,0,\,1,\,-1)$, and a toy sublayer $F(v)=0.5\,(v_2,\,v_3,\,v_4,\,v_1)$ (a cyclic shift scaled by one half). LayerNorm has gain 1 and bias 0, and $\varepsilon$ is ignored. Both orders use the same $F$.

Computed in Python

| quantity | post-LN | pre-LN |
| --- | --- | --- |
| sublayer input | x = (2, 0, 1, −1) | LN(x) = (1.3416, −0.4472, 0.4472, −1.3416) |
| F(input) | (0, 0.5, −0.5, 1) | (−0.2236, 0.2236, −0.6708, 0.6708) |
| residual sum | (2, 0.5, 0.5, 0) | (1.7764, 0.2236, 0.3292, −0.3292) |
| layer output | LN(sum) = (1.6667, −0.3333, −0.3333, −1) | the sum itself |
| output mean / RMS | 0 / 1 | 0.5 / 0.925 |

**The arithmetic**

1. Post-LN: $F(x)=0.5\,(0,1,-1,2)=(0,\,0.5,\,-0.5,\,1)$, so $x+F(x)=(2,\,0.5,\,0.5,\,0)$. Its mean is $0.75$ and its variance $\big(1.25^2+0.25^2+0.25^2+0.75^2\big)/4=0.5625$, so its standard deviation is $0.75$. Then $\mathrm{LN}=(1.25,-0.25,-0.25,-0.75)/0.75=(1.6667,\,-0.3333,\,-0.3333,\,-1)$.
2. Pre-LN: $x$ has mean $0.5$, variance $1.25$, standard deviation $1.1180$, so $\mathrm{LN}(x)=(1.5,-0.5,0.5,-1.5)/1.1180=(1.3416,\,-0.4472,\,0.4472,\,-1.3416)$.
3. $F(\mathrm{LN}(x))=(-0.2236,\,0.2236,\,-0.6708,\,0.6708)$, and $x+F(\mathrm{LN}(x))=(1.7764,\,0.2236,\,0.3292,\,-0.3292)$.
4. Post-LN always returns a zero-mean, unit-variance vector, so the input's mean and scale are lost at every sublayer. Pre-LN passes $x$ through unchanged and only adds a bounded update: the mean $0.5$ of $x$ survives. Over many layers these updates accumulate, which is the growth in the figure.

```python
import math, torch, torch.nn as nn

def sinusoid(L, d):
    pos = torch.arange(L)[:, None]
    ang = pos / 10000 ** (torch.arange(0, d, 2) / d)
    pe = torch.zeros(L, d)
    pe[:, 0::2], pe[:, 1::2] = torch.sin(ang), torch.cos(ang)
    return pe

class EncoderLayer(nn.Module):
    def __init__(self, d=512, h=8, d_ff=2048, p=0.1, pre_ln=False):
        super().__init__()
        self.attn = nn.MultiheadAttention(d, h, dropout=p, batch_first=True)
        self.ff = nn.Sequential(nn.Linear(d, d_ff), nn.ReLU(), nn.Dropout(p), nn.Linear(d_ff, d))
        self.n1, self.n2 = nn.LayerNorm(d), nn.LayerNorm(d)
        self.drop, self.pre_ln = nn.Dropout(p), pre_ln

    def _sa(self, x, pad):
        return self.drop(self.attn(x, x, x, key_padding_mask=pad, need_weights=False)[0])

    def forward(self, x, pad):                     # x (N, n, d); pad (N, n), True = padding
        if self.pre_ln:
            x = x + self._sa(self.n1(x), pad)
            return x + self.drop(self.ff(self.n2(x)))
        x = self.n1(x + self._sa(x, pad))
        return self.n2(x + self.drop(self.ff(x)))

class Encoder(nn.Module):
    def __init__(self, vocab, N=6, d=512, h=8, d_ff=2048, p=0.1, pre_ln=False, max_len=512):
        super().__init__()
        self.emb, self.d = nn.Embedding(vocab, d, padding_idx=0), d
        self.register_buffer("pe", sinusoid(max_len, d), persistent=False)
        self.layers = nn.ModuleList(EncoderLayer(d, h, d_ff, p, pre_ln) for _ in range(N))
        self.final = nn.LayerNorm(d) if pre_ln else nn.Identity()   # pre-LN needs a last LN
        self.drop = nn.Dropout(p)

    def forward(self, ids):
        pad = ids == 0
        x = self.drop(self.emb(ids) * math.sqrt(self.d) + self.pe[: ids.size(1)])
        for layer in self.layers:
            x = layer(x, pad)
        return self.final(x), pad

enc = Encoder(vocab=37000).eval()
ids = torch.randint(1, 37000, (2, 10)); ids[1, 7:] = 0           # second sentence has 3 pads
out, pad = enc(ids)
print(out.shape)                                                  # torch.Size([2, 10, 512])
print(sum(p.numel() for p in enc.layers[0].parameters()))         # 3152384 = 12*512**2 + 13*512
print(sum(p.numel() for p in nn.TransformerEncoderLayer(512, 8, 2048).parameters()))  # 3152384
print(sum(p.numel() for p in enc.parameters()))                   # 37858304 (18.94M of it embeddings)

# sentence vector: mean over real tokens only
sent = (out * ~pad[..., None]).sum(1) / (~pad).sum(1, keepdim=True)
print(sent.shape)                                                 # torch.Size([2, 512])
```

A Transformer encoder with a post-LN / pre-LN switch, padding mask and parameter check

> **The padding mask is the encoder's only mask, and it is easy to drop:** Batched sentences are padded to a common length. Without `key_padding_mask` every real token also attends to [PAD] vectors, so the output for a sentence depends on how long the other sentences in its batch were. The mask must also be applied when pooling: average over real tokens only, as in the last lines of the code. A causal mask in an encoder is a different mistake. It silently turns the model into a left-to-right one and throws away the bidirectional context that BERT-style training relies on.

> **Pre-LN without the final LayerNorm:** In a pre-LN stack nothing normalises the output of the last layer. Its scale grows with depth (4.81 RMS after 12 layers in the figure, against 1.40 at the input), and a classifier or decoder fed with it sees badly scaled inputs. GPT-2, ViT and T5 all add one LayerNorm after the stack. Post-LN models end in a LayerNorm already.

> **Where the parameters are:** Per layer, the FFN holds $8d^2$ and attention $4d^2$, whatever the number of heads. The attention matrix itself has no parameters, and its cost per token grows with $n$ instead. So "bigger model" means wider $d$ and $d_{ff}$ or more layers, and "longer context" is a separate cost axis. For BERT-base the embeddings, mostly the 30K-token table, are another 23.8M parameters that cost almost no compute: a lookup per token.

> **Reading out an encoder:** An encoder returns one vector per token. Token-level tasks (tagging, extractive QA spans) use these directly. Sequence-level tasks need a pooled vector. BERT uses the final state of the prepended [CLS] token, passed through a $\tanh$ pooler. Mean pooling over real tokens is a common alternative. In an **Encoder–decoder transformer** the whole $n\times d$ output is kept as memory and read by **Cross-attention**.

**Strengths**

- Every token sees the whole input in one layer: bidirectional context, constant path length between any two positions
- All positions are computed in parallel in training and at inference; no recurrence
- Simple, uniform block: $12d^2$ parameters per layer, easy to scale by $N$ and $d$
- The same block works for text, image patches (ViT), audio frames and other token sequences

**Limitations**

- Attention cost and memory grow as $n^2$ per layer; long inputs need sparse or linear attention
- Not generative on its own: there is no causal factorisation to sample from
- Post-LN needs warm-up and careful learning rates; pre-LN grows the residual stream and needs a final LN
- No notion of order without positional encodings

**Question:** A Transformer encoder layer has d = 768 and d_ff = 3072. How many parameters does it have, including biases and LayerNorms?

- A. 2,362,368
- B. 7,087,872
- C. 9,437,184
- D. 4,722,432

**Answer:** B. 7,087,872 — Attention $4d^2+4d=2{,}362{,}368$, FFN $2\,d\,d_{ff}+d_{ff}+d=4{,}722{,}432$, two LayerNorms $4d=3{,}072$. The sum is $7{,}087{,}872=12d^2+13d$, one BERT-base layer. 2,362,368 is attention alone and 4,722,432 the FFN alone. 9,437,184 is $16d^2$, the size of a *decoder* layer with cross-attention.

**Question:** Why does a pre-LN Transformer typically train without learning-rate warm-up while the original post-LN one does not?

- A. Pre-LN has fewer parameters
- B. In pre-LN the residual path is an unnormalised sum, so the gradient keeps an identity term and stays well scaled at initialisation; post-LN puts a LayerNorm on the residual path of every sublayer, and its gradients near the output are large at initialisation
- C. Pre-LN removes dropout
- D. Pre-LN uses a causal mask

**Answer:** B. In pre-LN the residual path is an unnormalised sum, so the gradient keeps an identity term and stays well scaled at initialisation; post-LN puts a LayerNorm on the residual path of every sublayer, and its gradients near the output are large at initialisation — This is the analysis of Xiong et al. (ICML 2020). Both variants have the same parameters, and dropout and masking are unrelated. Pre-LN makes $x_N=x_0+\sum_\ell F_\ell$, so $\partial x_N/\partial x_0$ contains $I$. The price is a residual stream whose scale grows with depth, and the final LayerNorm it requires.

**Key points**

- Encoder layer (post-LN): $h=\mathrm{LN}(x+\mathrm{MHA}(x))$, $y=\mathrm{LN}(h+\mathrm{FFN}(h))$; self-attention is bidirectional, and only padding is masked.
- Parameters per layer: $4d^2$ attention $+\ 8d^2$ FFN ($d_{ff}=4d$) $=12d^2+13d$; 3,152,384 for $d=512$, 7,087,872 for BERT-base.
- Pre-LN $x+F(\mathrm{LN}(x))$ keeps an identity residual path and trains without warm-up, but needs a final LN; post-LN $\mathrm{LN}(x+F(x))$ is the 2017/BERT choice.
- Attention is the only place tokens exchange information; the FFN works per token and holds two thirds of the weights.
- Without positional information the encoder is permutation-equivariant.
- BERT = the encoder alone, pre-trained with masked-token prediction: base 12×768 (110M), large 24×1024 (340M).

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
