Transformer encoder

A Transformer encoder is a stack of identical layers, each a multi-head self-attention sublayer and a position-wise feed-forward sublayer wrapped in residual connections and LayerNorm. Every token attends to every other token, and each layer has about 12d² parameters.

14 min read1122 wordscore0 viewsupdated 2026-10-12

Definition

Transformer encoder is a stack of NN identical layers that maps a sequence of nn input vectors x1,…,xn∈Rdx_1,\dots,x_n\in\mathbb R^{d} (token embeddings plus position information) to nn contextual vectors of the same width. Each layer has two sublayers: multi-head self-attention, in which every position attends to every position, and a position-wise feed-forward network applied to each token separately. Each sublayer is wrapped in a residual connection and a LayerNorm. There is no causal mask. The only mask hides padding.

The encoder is the left half of the architecture in Vaswani et al., "Attention Is All You Need" (NeurIPS 2017), the model described in Encoder–decoder transformer. Its job there was to read a source sentence and hand the decoder a memory of nn vectors. Recurrence passes information one step at a time, so position ii needs O(∣i−j∣)O(|i-j|) steps to reach position jj. In an attention layer every pair is one step apart, and all positions are computed in parallel. The paper's encoder had N=6N=6 layers of width dmodel=512d_{\text{model}}=512, 8 heads and an inner FFN width of 2048.

The encoder alone became the standard text representation model with BERT (Devlin, Chang, Lee and Toutanova, "BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding", NAACL 2019). BERT pre-trained an encoder by predicting masked tokens from both sides at once, which a left-to-right decoder cannot do. The same block, with patches as tokens, is the backbone of Vision Transformers (ViT).

One layer

h=LN(x+Dropout(MHA(x,x,x; pad mask)))y=LN(h+Dropout(FFN(h))),FFN(h)=max⁡(0, hW1+b1) W2+b2\begin{aligned}h&=\mathrm{LN}\big(x+\mathrm{Dropout}(\mathrm{MHA}(x,x,x;\ \text{pad mask}))\big)\\ y&=\mathrm{LN}\big(h+\mathrm{Dropout}(\mathrm{FFN}(h))\big),\qquad \mathrm{FFN}(h)=\max(0,\,hW_1+b_1)\,W_2+b_2\end{aligned}
x∈Rn×dx\in\mathbb R^{n\times d}
the layer input; for layer 1 the embeddings ×d\times\sqrt d plus positional encodings (see Positional encoding)
MHA\mathrm{MHA}
multi-head self-attention: queries, keys and values all come from xx; hh heads of width d/hd/h (see Multi-head attention)
pad mask\text{pad mask}
keys at padding positions get score −∞-\infty; nothing else is masked
W1∈Rd×dff, W2∈Rdff×dW_1\in\mathbb R^{d\times d_{ff}},\ W_2\in\mathbb R^{d_{ff}\times d}
the FFN, applied to each row of hh independently; dff=4dd_{ff}=4d in the original and in BERT
LN\mathrm{LN}
LayerNorm over the dd features of each token, with learned gain and bias
Dropout\mathrm{Dropout}
applied to each sublayer output before the addition; Pdrop=0.1P_{\text{drop}}=0.1 in the base model

The two sublayers do different jobs. Self-attention is the only place where tokens exchange information: row ii of the output is a weighted average of value vectors from all positions (see Self-attention). The FFN mixes features within a token and holds about two thirds of the layer's parameters. The residual connections keep a direct path from the input to the output of every layer, and each sublayer adds an update to it.

Without position information the layer is permutation-equivariant: permute the input rows and the output rows are permuted the same way. Attention compares contents only. Word order enters only through the positional encodings added at the bottom, or through relative position terms inside attention (see Rotary positional embeddings (RoPE) and ALiBi).

Post-LN and pre-LN

post-LN:  xℓ+1=LN(xℓ+F(xℓ))pre-LN:  xℓ+1=xℓ+F(LN(xℓ)),output=LN(xN)\text{post-LN:}\ \ x_{\ell+1}=\mathrm{LN}\big(x_\ell+F(x_\ell)\big)\qquad\qquad \text{pre-LN:}\ \ x_{\ell+1}=x_\ell+F\big(\mathrm{LN}(x_\ell)\big),\quad \text{output}=\mathrm{LN}(x_N)
FF
a sublayer (attention or FFN, with its dropout)
post-LN\text{post-LN}
the 2017 Transformer and BERT: normalise after the residual addition
pre-LN\text{pre-LN}
GPT-2, ViT, T5 and most large models since: normalise the sublayer input and leave the residual sum alone; one extra LN after the last layer

In post-LN the residual path itself passes through a LayerNorm in every sublayer, so the input of layer 1 reaches the output only through 2N2N normalisations. In pre-LN the path from x0x_0 to xNx_N is a pure sum, xN=x0+∑ℓFℓ(⋅)x_N=x_0+\sum_\ell F_\ell(\cdot), and the gradient always contains an identity term. Xiong et al., "On Layer Normalization in the Transformer Architecture" (ICML 2020), showed that at initialisation the post-LN gradients of the parameters near the output are large. This is why post-LN needs a learning-rate warm-up. Pre-LN gradients are well behaved, and pre-LN models trained without warm-up in their experiments. Wang et al. (ACL 2019) had earlier used pre-norm to train 30-layer translation encoders.

Pre-LN has its own costs. The residual stream is never normalised inside the stack, so its scale grows with depth. Each later layer then changes it relatively less, and the final LayerNorm is mandatory. When post-LN trains successfully it often reaches slightly better quality. Liu et al. (EMNLP 2020) traced post-LN's instability to its strong dependence on the residual branch and proposed a re-scaled initialisation (Admin) to keep post-LN's quality with pre-LN's stability.

post-LN: x ← LN(x + MHA(x)), x ← LN(x + FFN(x)) · d = 16, 2 heads, d_ff = 64one encoder layer (× 12)x (from layer below)multi-head self-attnadd x + MHA(x)LayerNormfeed-forwardadd x + FFN(x)LayerNorm— residual (identity) pathsevery layer ends in LN: RMS reset to 1layer 1, head 1: no causal maskthecatsatonthewarm[PAD][PAD]thecatsatonthewarm[PAD][PAD]rows: queries · cols: keysvariant post-LNlayer 1 of 12residual RMS 1.00‖F‖/‖x‖ 0.87cos(xℓ, x₀) 0.56output RMS 1.00real / pad 6 / 2max w on pad 0.000layer 12: RMS post 1.00 · pre 4.81‖F‖/‖x‖ post 1.36 · pre 0.240.02.75.5residual RMS after each layer · post-LN · pre-LN (same weights)layer 1 … 12BERT-base (N=12, d=768, d_ff=3072): per layer attn 2,362,368 + FFN 4,722,432 + LN 3,072= 7,087,872 (12d² = 7,077,888) · stack 85.05M · + embeddings + pooler 24.43M = 109.48M (paper 110M)
LayerNormplot
count params
fig.A real 8-token encoder at initialisation (d = 16, 2 heads, d_ff = 64, ReLU, weights with std 1/√fan-in). The post-LN and pre-LN stacks share the same weights and differ only in where LayerNorm sits. Post-LN: the residual RMS is exactly 1.00 after every layer, each layer's update is as large as its input (‖F‖/‖x‖ = 0.87 in layer 1, 1.36 in layer 12), and the cosine between a token's state and its input embedding falls from 0.56 after layer 1 to 0.07 after layer 12. Pre-LN with the same weights: the stream's RMS grows from 1.40 at the input to 1.83 after layer 1 and 4.81 after layer 12 (7.74 at 24 layers). The relative update shrinks from 0.81 to 0.24 (0.15 at 24 layers), and the cosine to the input stays at 0.40. The heat map shows layer ℓ's head-1 weights. There is no causal triangle, and the two [PAD] key columns get exactly 0 weight from every real query. The bottom lines count parameters: 3,152,384 per Transformer-base layer, and 85.05M for BERT-base's 12 layers, which with its embeddings and pooler gives 109.48M.

Parameter count

Player=4d2+4d⏟WQ,WK,WV,WO+2 d dff+dff+d⏟FFN+4d⏟2 LN =dff=4d 12d2+13dP_{\text{layer}}=\underbrace{4d^2+4d}_{W_Q,W_K,W_V,W_O}+\underbrace{2\,d\,d_{ff}+d_{ff}+d}_{\text{FFN}}+\underbrace{4d}_{2\ \mathrm{LN}}\ \overset{d_{ff}=4d}{=}\ 12d^2+13d
4d24d^2
the four d×dd\times d projections; the number of heads does not change it, because hh heads of width d/hd/h together span dd
8d28d^2
the FFN with dff=4dd_{ff}=4d, twice the attention parameters
13d13d
biases and LayerNorm gains and biases; negligible for dd in the hundreds

Compute follows the same split. Per token and layer, the matrix multiplies cost about 2⋅12d2=24d22\cdot12d^2=24d^2 FLOPs, independent of nn. The attention scores and the weighted sum add 4nd4nd FLOPs per token, so the quadratic part matters only once nn is comparable to 6d6d: 3072 tokens for d=512d=512. Long-input encoders change exactly this term (see Sparse attention, Longformer, BigBird).

modelNdd_ffheadsper layerN layerstotal
Transformer base encoder6512204883,152,38418.91M+ shared 37K × 512 embedding
Transformer big encoder6102440961612,596,22475.58M+ shared embedding
BERT-base127683072127,087,87285.05M109.48M (paper: 110M)
BERT-large24102440961612,596,224302.31M335.14M (paper: 340M)
Exact parameter counts (computed; biases and LayerNorm included)

The BERT totals add the 30,522-token WordPiece table, 512 learned position embeddings, 2 segment embeddings, the embedding LayerNorm and the d×dd\times d pooler. For BERT-base the embeddings are 23.84M of the 109.48M, more than 20%.

BERT (Devlin et al. 2019)

Sizes
BERT-base L=12L=12, H=768H=768, A=12A=12, 110M; BERT-large L=24L=24, H=1024H=1024, A=16A=16, 340M; FFN width 4H4H, GELU
Input
WordPiece, 30,000-token vocabulary; [CLS] first, [SEP] between segments; token + segment + learned position embeddings; at most 512 tokens
Pre-training tasks
masked LM: 15% of tokens chosen, of which 80% become [MASK], 10% a random token, 10% unchanged; next-sentence prediction on [CLS]
Data
BooksCorpus (800M words) and English Wikipedia (2,500M words)
Schedule
1,000,000 steps of 256 sequences × 512 tokens (128,000 tokens per batch), about 40 epochs; Adam, learning rate 10−410^{-4}, 10,000 warm-up steps
Results (BERT-large)
GLUE 80.5 (+7.7 points), MultiNLI 86.7%, SQuAD v1.1 test F1 93.2, SQuAD v2.0 test F1 83.1
LayerNorm
post-LN, as in the 2017 Transformer

Worked example: one sublayer, post-LN vs pre-LN

Take one token with d=4d=4, x=(2, 0, 1, −1)x=(2,\,0,\,1,\,-1), and a toy sublayer F(v)=0.5 (v2, v3, v4, v1)F(v)=0.5\,(v_2,\,v_3,\,v_4,\,v_1) (a cyclic shift scaled by one half). LayerNorm has gain 1 and bias 0, and ε\varepsilon is ignored. Both orders use the same FF.

quantitypost-LNpre-LN
sublayer inputx = (2, 0, 1, −1)LN(x) = (1.3416, −0.4472, 0.4472, −1.3416)
F(input)(0, 0.5, −0.5, 1)(−0.2236, 0.2236, −0.6708, 0.6708)
residual sum(2, 0.5, 0.5, 0)(1.7764, 0.2236, 0.3292, −0.3292)
layer outputLN(sum) = (1.6667, −0.3333, −0.3333, −1)the sum itself
output mean / RMS0 / 10.5 / 0.925
Computed in Python

The arithmetic

  1. 1Post-LN: F(x)=0.5 (0,1,−1,2)=(0, 0.5, −0.5, 1)F(x)=0.5\,(0,1,-1,2)=(0,\,0.5,\,-0.5,\,1), so x+F(x)=(2, 0.5, 0.5, 0)x+F(x)=(2,\,0.5,\,0.5,\,0). Its mean is 0.750.75 and its variance (1.252+0.252+0.252+0.752)/4=0.5625\big(1.25^2+0.25^2+0.25^2+0.75^2\big)/4=0.5625, so its standard deviation is 0.750.75. Then LN=(1.25,−0.25,−0.25,−0.75)/0.75=(1.6667, −0.3333, −0.3333, −1)\mathrm{LN}=(1.25,-0.25,-0.25,-0.75)/0.75=(1.6667,\,-0.3333,\,-0.3333,\,-1).
  2. 2Pre-LN: xx has mean 0.50.5, variance 1.251.25, standard deviation 1.11801.1180, so LN(x)=(1.5,−0.5,0.5,−1.5)/1.1180=(1.3416, −0.4472, 0.4472, −1.3416)\mathrm{LN}(x)=(1.5,-0.5,0.5,-1.5)/1.1180=(1.3416,\,-0.4472,\,0.4472,\,-1.3416).
  3. 3F(LN(x))=(−0.2236, 0.2236, −0.6708, 0.6708)F(\mathrm{LN}(x))=(-0.2236,\,0.2236,\,-0.6708,\,0.6708), and x+F(LN(x))=(1.7764, 0.2236, 0.3292, −0.3292)x+F(\mathrm{LN}(x))=(1.7764,\,0.2236,\,0.3292,\,-0.3292).
  4. 4Post-LN always returns a zero-mean, unit-variance vector, so the input's mean and scale are lost at every sublayer. Pre-LN passes xx through unchanged and only adds a bounded update: the mean 0.50.5 of xx survives. Over many layers these updates accumulate, which is the growth in the figure.
python
1import math, torch, torch.nn as nn
2
3def sinusoid(L, d):
4 pos = torch.arange(L)[:, None]
5 ang = pos / 10000 ** (torch.arange(0, d, 2) / d)
6 pe = torch.zeros(L, d)
7 pe[:, 0::2], pe[:, 1::2] = torch.sin(ang), torch.cos(ang)
8 return pe
9
10class EncoderLayer(nn.Module):
11 def __init__(self, d=512, h=8, d_ff=2048, p=0.1, pre_ln=False):
12 super().__init__()
13 self.attn = nn.MultiheadAttention(d, h, dropout=p, batch_first=True)
14 self.ff = nn.Sequential(nn.Linear(d, d_ff), nn.ReLU(), nn.Dropout(p), nn.Linear(d_ff, d))
15 self.n1, self.n2 = nn.LayerNorm(d), nn.LayerNorm(d)
16 self.drop, self.pre_ln = nn.Dropout(p), pre_ln
17
18 def _sa(self, x, pad):
19 return self.drop(self.attn(x, x, x, key_padding_mask=pad, need_weights=False)[0])
20
21 def forward(self, x, pad): # x (N, n, d); pad (N, n), True = padding
22 if self.pre_ln:
23 x = x + self._sa(self.n1(x), pad)
24 return x + self.drop(self.ff(self.n2(x)))
25 x = self.n1(x + self._sa(x, pad))
26 return self.n2(x + self.drop(self.ff(x)))
27
28class Encoder(nn.Module):
29 def __init__(self, vocab, N=6, d=512, h=8, d_ff=2048, p=0.1, pre_ln=False, max_len=512):
30 super().__init__()
31 self.emb, self.d = nn.Embedding(vocab, d, padding_idx=0), d
32 self.register_buffer("pe", sinusoid(max_len, d), persistent=False)
33 self.layers = nn.ModuleList(EncoderLayer(d, h, d_ff, p, pre_ln) for _ in range(N))
34 self.final = nn.LayerNorm(d) if pre_ln else nn.Identity() # pre-LN needs a last LN
35 self.drop = nn.Dropout(p)
36
37 def forward(self, ids):
38 pad = ids == 0
39 x = self.drop(self.emb(ids) * math.sqrt(self.d) + self.pe[: ids.size(1)])
40 for layer in self.layers:
41 x = layer(x, pad)
42 return self.final(x), pad
43
44enc = Encoder(vocab=37000).eval()
45ids = torch.randint(1, 37000, (2, 10)); ids[1, 7:] = 0 # second sentence has 3 pads
46out, pad = enc(ids)
47print(out.shape) # torch.Size([2, 10, 512])
48print(sum(p.numel() for p in enc.layers[0].parameters())) # 3152384 = 12*512**2 + 13*512
49print(sum(p.numel() for p in nn.TransformerEncoderLayer(512, 8, 2048).parameters())) # 3152384
50print(sum(p.numel() for p in enc.parameters())) # 37858304 (18.94M of it embeddings)
51
52# sentence vector: mean over real tokens only
53sent = (out * ~pad[..., None]).sum(1) / (~pad).sum(1, keepdim=True)
54print(sent.shape) # torch.Size([2, 512])
A Transformer encoder with a post-LN / pre-LN switch, padding mask and parameter check

Use when

  • +Every token sees the whole input in one layer: bidirectional context, constant path length between any two positions
  • +All positions are computed in parallel in training and at inference; no recurrence
  • +Simple, uniform block: 12d212d^2 parameters per layer, easy to scale by NN and dd
  • +The same block works for text, image patches (ViT), audio frames and other token sequences

Avoid when

  • −Attention cost and memory grow as n2n^2 per layer; long inputs need sparse or linear attention
  • −Not generative on its own: there is no causal factorisation to sample from
  • −Post-LN needs warm-up and careful learning rates; pre-LN grows the residual stream and needs a final LN
  • −No notion of order without positional encodings

Check yourself

A Transformer encoder layer has d = 768 and d_ff = 3072. How many parameters does it have, including biases and LayerNorms?

Check yourself

Why does a pre-LN Transformer typically train without learning-rate warm-up while the original post-LN one does not?

Remember

  • ✓Encoder layer (post-LN): h=LN(x+MHA(x))h=\mathrm{LN}(x+\mathrm{MHA}(x)), y=LN(h+FFN(h))y=\mathrm{LN}(h+\mathrm{FFN}(h)); self-attention is bidirectional, and only padding is masked.
  • ✓Parameters per layer: 4d24d^2 attention + 8d2+\ 8d^2 FFN (dff=4dd_{ff}=4d) =12d2+13d=12d^2+13d; 3,152,384 for d=512d=512, 7,087,872 for BERT-base.
  • ✓Pre-LN x+F(LN(x))x+F(\mathrm{LN}(x)) keeps an identity residual path and trains without warm-up, but needs a final LN; post-LN LN(x+F(x))\mathrm{LN}(x+F(x)) is the 2017/BERT choice.
  • ✓Attention is the only place tokens exchange information; the FFN works per token and holds two thirds of the weights.
  • ✓Without positional information the encoder is permutation-equivariant.
  • ✓BERT = the encoder alone, pre-trained with masked-token prediction: base 12×768 (110M), large 24×1024 (340M).
← back to Deep Learning roadmap