Definition
Transformer encoder is a stack of identical layers that maps a sequence of input vectors (token embeddings plus position information) to contextual vectors of the same width. Each layer has two sublayers: multi-head self-attention, in which every position attends to every position, and a position-wise feed-forward network applied to each token separately. Each sublayer is wrapped in a residual connection and a LayerNorm. There is no causal mask. The only mask hides padding.
The encoder is the left half of the architecture in Vaswani et al., "Attention Is All You Need" (NeurIPS 2017), the model described in Encoder–decoder transformer. Its job there was to read a source sentence and hand the decoder a memory of vectors. Recurrence passes information one step at a time, so position needs steps to reach position . In an attention layer every pair is one step apart, and all positions are computed in parallel. The paper's encoder had layers of width , 8 heads and an inner FFN width of 2048.
The encoder alone became the standard text representation model with BERT (Devlin, Chang, Lee and Toutanova, "BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding", NAACL 2019). BERT pre-trained an encoder by predicting masked tokens from both sides at once, which a left-to-right decoder cannot do. The same block, with patches as tokens, is the backbone of Vision Transformers (ViT).
One layer
- the layer input; for layer 1 the embeddings plus positional encodings (see Positional encoding)
- multi-head self-attention: queries, keys and values all come from ; heads of width (see Multi-head attention)
- keys at padding positions get score ; nothing else is masked
- the FFN, applied to each row of independently; in the original and in BERT
- LayerNorm over the features of each token, with learned gain and bias
- applied to each sublayer output before the addition; in the base model
The two sublayers do different jobs. Self-attention is the only place where tokens exchange information: row of the output is a weighted average of value vectors from all positions (see Self-attention). The FFN mixes features within a token and holds about two thirds of the layer's parameters. The residual connections keep a direct path from the input to the output of every layer, and each sublayer adds an update to it.
Without position information the layer is permutation-equivariant: permute the input rows and the output rows are permuted the same way. Attention compares contents only. Word order enters only through the positional encodings added at the bottom, or through relative position terms inside attention (see Rotary positional embeddings (RoPE) and ALiBi).
Post-LN and pre-LN
- a sublayer (attention or FFN, with its dropout)
- the 2017 Transformer and BERT: normalise after the residual addition
- GPT-2, ViT, T5 and most large models since: normalise the sublayer input and leave the residual sum alone; one extra LN after the last layer
In post-LN the residual path itself passes through a LayerNorm in every sublayer, so the input of layer 1 reaches the output only through normalisations. In pre-LN the path from to is a pure sum, , and the gradient always contains an identity term. Xiong et al., "On Layer Normalization in the Transformer Architecture" (ICML 2020), showed that at initialisation the post-LN gradients of the parameters near the output are large. This is why post-LN needs a learning-rate warm-up. Pre-LN gradients are well behaved, and pre-LN models trained without warm-up in their experiments. Wang et al. (ACL 2019) had earlier used pre-norm to train 30-layer translation encoders.
Pre-LN has its own costs. The residual stream is never normalised inside the stack, so its scale grows with depth. Each later layer then changes it relatively less, and the final LayerNorm is mandatory. When post-LN trains successfully it often reaches slightly better quality. Liu et al. (EMNLP 2020) traced post-LN's instability to its strong dependence on the residual branch and proposed a re-scaled initialisation (Admin) to keep post-LN's quality with pre-LN's stability.
Parameter count
- the four projections; the number of heads does not change it, because heads of width together span
- the FFN with , twice the attention parameters
- biases and LayerNorm gains and biases; negligible for in the hundreds
Compute follows the same split. Per token and layer, the matrix multiplies cost about FLOPs, independent of . The attention scores and the weighted sum add FLOPs per token, so the quadratic part matters only once is comparable to : 3072 tokens for . Long-input encoders change exactly this term (see Sparse attention, Longformer, BigBird).
| model | N | d | d_ff | heads | per layer | N layers | total |
|---|---|---|---|---|---|---|---|
| Transformer base encoder | 6 | 512 | 2048 | 8 | 3,152,384 | 18.91M | + shared 37K × 512 embedding |
| Transformer big encoder | 6 | 1024 | 4096 | 16 | 12,596,224 | 75.58M | + shared embedding |
| BERT-base | 12 | 768 | 3072 | 12 | 7,087,872 | 85.05M | 109.48M (paper: 110M) |
| BERT-large | 24 | 1024 | 4096 | 16 | 12,596,224 | 302.31M | 335.14M (paper: 340M) |
The BERT totals add the 30,522-token WordPiece table, 512 learned position embeddings, 2 segment embeddings, the embedding LayerNorm and the pooler. For BERT-base the embeddings are 23.84M of the 109.48M, more than 20%.
BERT (Devlin et al. 2019)
- Sizes
- BERT-base , , , 110M; BERT-large , , , 340M; FFN width , GELU
- Input
- WordPiece, 30,000-token vocabulary; [CLS] first, [SEP] between segments; token + segment + learned position embeddings; at most 512 tokens
- Pre-training tasks
- masked LM: 15% of tokens chosen, of which 80% become [MASK], 10% a random token, 10% unchanged; next-sentence prediction on [CLS]
- Data
- BooksCorpus (800M words) and English Wikipedia (2,500M words)
- Schedule
- 1,000,000 steps of 256 sequences × 512 tokens (128,000 tokens per batch), about 40 epochs; Adam, learning rate , 10,000 warm-up steps
- Results (BERT-large)
- GLUE 80.5 (+7.7 points), MultiNLI 86.7%, SQuAD v1.1 test F1 93.2, SQuAD v2.0 test F1 83.1
- LayerNorm
- post-LN, as in the 2017 Transformer
Worked example: one sublayer, post-LN vs pre-LN
Take one token with , , and a toy sublayer (a cyclic shift scaled by one half). LayerNorm has gain 1 and bias 0, and is ignored. Both orders use the same .
| quantity | post-LN | pre-LN |
|---|---|---|
| sublayer input | x = (2, 0, 1, −1) | LN(x) = (1.3416, −0.4472, 0.4472, −1.3416) |
| F(input) | (0, 0.5, −0.5, 1) | (−0.2236, 0.2236, −0.6708, 0.6708) |
| residual sum | (2, 0.5, 0.5, 0) | (1.7764, 0.2236, 0.3292, −0.3292) |
| layer output | LN(sum) = (1.6667, −0.3333, −0.3333, −1) | the sum itself |
| output mean / RMS | 0 / 1 | 0.5 / 0.925 |
The arithmetic
- 1Post-LN: , so . Its mean is and its variance , so its standard deviation is . Then .
- 2Pre-LN: has mean , variance , standard deviation , so .
- 3, and .
- 4Post-LN always returns a zero-mean, unit-variance vector, so the input's mean and scale are lost at every sublayer. Pre-LN passes through unchanged and only adds a bounded update: the mean of survives. Over many layers these updates accumulate, which is the growth in the figure.
1import math, torch, torch.nn as nn23def sinusoid(L, d):4 pos = torch.arange(L)[:, None]5 ang = pos / 10000 ** (torch.arange(0, d, 2) / d)6 pe = torch.zeros(L, d)7 pe[:, 0::2], pe[:, 1::2] = torch.sin(ang), torch.cos(ang)8 return pe910class EncoderLayer(nn.Module):11 def __init__(self, d=512, h=8, d_ff=2048, p=0.1, pre_ln=False):12 super().__init__()13 self.attn = nn.MultiheadAttention(d, h, dropout=p, batch_first=True)14 self.ff = nn.Sequential(nn.Linear(d, d_ff), nn.ReLU(), nn.Dropout(p), nn.Linear(d_ff, d))15 self.n1, self.n2 = nn.LayerNorm(d), nn.LayerNorm(d)16 self.drop, self.pre_ln = nn.Dropout(p), pre_ln1718 def _sa(self, x, pad):19 return self.drop(self.attn(x, x, x, key_padding_mask=pad, need_weights=False)[0])2021 def forward(self, x, pad): # x (N, n, d); pad (N, n), True = padding22 if self.pre_ln:23 x = x + self._sa(self.n1(x), pad)24 return x + self.drop(self.ff(self.n2(x)))25 x = self.n1(x + self._sa(x, pad))26 return self.n2(x + self.drop(self.ff(x)))2728class Encoder(nn.Module):29 def __init__(self, vocab, N=6, d=512, h=8, d_ff=2048, p=0.1, pre_ln=False, max_len=512):30 super().__init__()31 self.emb, self.d = nn.Embedding(vocab, d, padding_idx=0), d32 self.register_buffer("pe", sinusoid(max_len, d), persistent=False)33 self.layers = nn.ModuleList(EncoderLayer(d, h, d_ff, p, pre_ln) for _ in range(N))34 self.final = nn.LayerNorm(d) if pre_ln else nn.Identity() # pre-LN needs a last LN35 self.drop = nn.Dropout(p)3637 def forward(self, ids):38 pad = ids == 039 x = self.drop(self.emb(ids) * math.sqrt(self.d) + self.pe[: ids.size(1)])40 for layer in self.layers:41 x = layer(x, pad)42 return self.final(x), pad4344enc = Encoder(vocab=37000).eval()45ids = torch.randint(1, 37000, (2, 10)); ids[1, 7:] = 0 # second sentence has 3 pads46out, pad = enc(ids)47print(out.shape) # torch.Size([2, 10, 512])48print(sum(p.numel() for p in enc.layers[0].parameters())) # 3152384 = 12*512**2 + 13*51249print(sum(p.numel() for p in nn.TransformerEncoderLayer(512, 8, 2048).parameters())) # 315238450print(sum(p.numel() for p in enc.parameters())) # 37858304 (18.94M of it embeddings)5152# sentence vector: mean over real tokens only53sent = (out * ~pad[..., None]).sum(1) / (~pad).sum(1, keepdim=True)54print(sent.shape) # torch.Size([2, 512])Use when
- +Every token sees the whole input in one layer: bidirectional context, constant path length between any two positions
- +All positions are computed in parallel in training and at inference; no recurrence
- +Simple, uniform block: parameters per layer, easy to scale by and
- +The same block works for text, image patches (ViT), audio frames and other token sequences
Avoid when
- −Attention cost and memory grow as per layer; long inputs need sparse or linear attention
- −Not generative on its own: there is no causal factorisation to sample from
- −Post-LN needs warm-up and careful learning rates; pre-LN grows the residual stream and needs a final LN
- −No notion of order without positional encodings
Check yourself
A Transformer encoder layer has d = 768 and d_ff = 3072. How many parameters does it have, including biases and LayerNorms?
Check yourself
Why does a pre-LN Transformer typically train without learning-rate warm-up while the original post-LN one does not?
Remember
- ✓Encoder layer (post-LN): , ; self-attention is bidirectional, and only padding is masked.
- ✓Parameters per layer: attention FFN () ; 3,152,384 for , 7,087,872 for BERT-base.
- ✓Pre-LN keeps an identity residual path and trains without warm-up, but needs a final LN; post-LN is the 2017/BERT choice.
- ✓Attention is the only place tokens exchange information; the FFN works per token and holds two thirds of the weights.
- ✓Without positional information the encoder is permutation-equivariant.
- ✓BERT = the encoder alone, pre-trained with masked-token prediction: base 12×768 (110M), large 24×1024 (340M).