Encoder-only models

Encoder-only models (BERT, RoBERTa, ELECTRA, DeBERTa, ModernBERT) run a transformer encoder with no attention mask, so every output sees the whole input. Pretrained by masked- or replaced-token objectives, they feed [CLS], per-token or pooled vectors to task heads, but do not generate.

15 min read1493 wordscore0 viewsupdated 2026-10-13

Definition

Encoder-only model is a language model made of the encoder half of the Transformer (Vaswani et al., 2017) alone: a stack of self-attention and feed-forward blocks with no attention mask, so the output at every position depends on the entire input. It maps nn tokens to nn contextual vectors h1,…,hnh_1,\dots,h_n and has no decoder. It is pretrained with a denoising objective, such as recovering masked tokens or detecting replaced ones, and fine-tuned by attaching small heads to those vectors. The family began with BERT (Devlin, Chang, Lee and Toutanova, "BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding", NAACL 2019). It includes RoBERTa, ALBERT, DistilBERT, ELECTRA, DeBERTa and ModernBERT.

In 2018 there were two ways to pretrain contextual representations. GPT trained a left-to-right transformer, so each token's vector saw only its left context. ELMo (Peters et al., 2018) trained separate left-to-right and right-to-left LSTMs and concatenated their states. It saw both sides, but only shallowly: no layer combined the two directions. The obstacle was the objective. A language model that predicts xtx_t cannot also attend to xtx_t, or it would just copy it. BERT's fix was to corrupt the input instead of masking attention. It replaces some tokens with [MASK], lets every layer attend everywhere, and predicts only the corrupted positions. The result was a deeply bidirectional encoder. On release it set the state of the art on GLUE, SQuAD and SWAG.

The family is defined by what it gives up. Without a causal mask there is no factorisation p(x)=∏tp(xt∣x<t)p(x)=\prod_t p(x_t\mid x_{<t}), so the model cannot be sampled left to right. In return, each output sees the full context. That is what classification, tagging, extractive QA and retrieval need, because the whole input is available before any decision is made. The other two shapes are covered in Decoder-only models and Encoder–decoder models.

Bidirectional attention

H(ℓ)=LN(A(ℓ)+FFN(A(ℓ))),A(ℓ)=LN(H(ℓ−1)+softmax(QK⊤dh+M)V),Mij=0  ∀i,jH^{(\ell)}=\mathrm{LN}\Big(A^{(\ell)}+\mathrm{FFN}\big(A^{(\ell)}\big)\Big),\quad A^{(\ell)}=\mathrm{LN}\Big(H^{(\ell-1)}+\mathrm{softmax}\Big(\frac{QK^\top}{\sqrt{d_h}}+M\Big)V\Big),\qquad M_{ij}=0\ \ \forall i,j
MM
attention mask; an encoder uses M=0M=0 (only padding is masked), a decoder uses Mij=−∞M_{ij}=-\infty for j>ij>i (see Causal attention)
H(0)H^{(0)}
token + position (+ segment) embeddings; BERT uses a 30,522-token WordPiece vocabulary and 512 learned positions
LN\mathrm{LN}
post-norm in BERT/RoBERTa; ModernBERT uses pre-norm
hi=Hi(L)h_i=H^{(L)}_i
the contextual vector of token ii, a function of all nn tokens

The mask is the whole difference from a decoder block. The parameter count, FFN and norms are the same, and Transformer encoder covers the block itself. The consequence is easy to state and easy to check. Under a causal mask, hih_i depends only on x≤ix_{\le i}. Two sentences with the same prefix therefore produce identical vectors for that prefix, and the first token sees nothing but itself. Under the encoder mask, every hih_i changes when any token changes. The interactive figure below runs both masks on the same weights.

sequence classificationh_CLS → tanh pooler (d × d) → linear → softmaxsentiment, NLI, GLUE; fine-tune the whole stackh_CLSh₁h₂h₃h₄h₅h_SEPencoder block × L (residual + LayerNorm around each)multi-head self-attention, no maskevery position i attends to every j, left and rightposition-wise FFN (GELU; GeGLU in ModernBERT)BERT-base 12 layers, d = 768, 12 heads, 110MBERT-large 24 layers, d = 1024, 16 heads, 340Mtoken + position + segment embeddings (WordPiece, 30,522)[CLS]thebankoftheriver[SEP]one output vector per input token; nothing is generatedModernBERT (2024): RoPE, GeGLU, local 128-token windows with a globallayer every third layer, 8192 context, 2T training tokenstask heads (click)sequence class.token class. (NER)span QAsentence embeddingmasked LMELECTRA RTDencoderall pairsdecoderj ≤ iwhy it does not generate:MLM learns p(x_i | rest),not p(x_t | x_<t); noleft-to-right factorisationand no KV cache, so eachnew token re-encodes all
fig.An encoder-only model and its heads (click a head on the right). Tokens, with [CLS] first and [SEP] last, are embedded and passed through L blocks of unmasked self-attention and FFN. The model returns one vector per token. Sequence classification reads only h_CLS through a tanh pooler. Token classification and extractive QA read every h_i. Sentence embeddings mean-pool all of them. The masked-LM head (pretraining) reads the [MASK] position. ELECTRA's discriminator classifies every position as original or replaced. Right: the encoder's all-pairs mask next to a decoder's j ≤ i mask.

Pretraining objectives

LMLM=−∑i∈Mlog⁡ softmax(E g(hi)+b)xi,∣M∣≈0.15 n,x~i={[MASK]80%random token10%xi10%\mathcal L_{\text{MLM}}=-\sum_{i\in\mathcal M}\log\,\mathrm{softmax}\big(E\,g(h_i)+b\big)_{x_i},\qquad |\mathcal M|\approx0.15\,n,\quad \tilde x_i=\begin{cases}\texttt{[MASK]}&80\%\\\text{random token}&10\%\\x_i&10\%\end{cases}
M\mathcal M
the positions chosen for prediction, 15% of tokens in BERT, 30% in ModernBERT
gg
a dense + GELU + LayerNorm transform; the output matrix is tied to the input embedding EE
80/10/1080/10/10
so the model cannot rely on [MASK] appearing, since [MASK] never appears at fine-tuning time
hih_i
computed from the corrupted sequence x~\tilde x, using context on both sides

BERT added a next-sentence prediction (NSP) loss on hCLSh_{\text{CLS}}: was segment B the true continuation of segment A? RoBERTa (Liu et al., 2019) found that NSP did not help. It dropped NSP, masked dynamically instead of once, trained longer on 160 GB of text with larger batches, and beat BERT clearly. Since then, most encoders train on MLM alone. ALBERT replaced NSP with sentence-order prediction and shared weights across layers. DistilBERT (Sanh et al., 2019) distilled BERT into 6 layers, 40% smaller and 60% faster while keeping 97% of its GLUE performance.

L=LMLM(G)+λ LDisc(D),LDisc=−∑t=1n[1(x^t=xt)log⁡D(x^,t)+1(x^t≠xt)log⁡(1−D(x^,t))],λ=50\mathcal L=\mathcal L_{\text{MLM}}(G)+\lambda\,\mathcal L_{\text{Disc}}(D),\quad \mathcal L_{\text{Disc}}=-\sum_{t=1}^{n}\Big[\mathbb 1(\hat x_t=x_t)\log D(\hat x,t)+\mathbb 1(\hat x_t\neq x_t)\log\big(1-D(\hat x,t)\big)\Big],\quad \lambda=50
GG
a small MLM generator (1/4 to 1/2 of the discriminator's size) that fills the masked slots by sampling
x^\hat x
the input with masked slots replaced by generator samples
D(x^,t)D(\hat x,t)
σ(w⊤ht)\sigma(w^\top h_t), the discriminator's probability that token tt is original; a correct generator guess counts as original
∑t=1n\sum_{t=1}^{n}
the loss covers all positions, not only the 15% that were masked

MLM learns from only about 15% of positions per sequence. ELECTRA (Clark, Luong, Le and Manning, "ELECTRA: Pre-training Text Encoders as Discriminators Rather Than Generators", ICLR 2020) turned pretraining into replaced-token detection (RTD) over every position. The generator is trained by maximum likelihood, not adversarially, and only the discriminator is kept. ELECTRA-Small, trained for 4 days on one GPU, beat GPT on GLUE, although GPT used about 30× more compute. ELECTRA-Large matched RoBERTa and XLNet with about a quarter of their compute. DeBERTa (He et al., ICLR 2021) instead changed attention. It keeps separate content and relative-position vectors, which interact through disentangled attention. A 1.5B version was the first single model to pass the human baseline on SuperGLUE.

Task heads: token outputs vs pooled outputs

y=softmax(Wctanh⁡(WphCLS+bp))⏟sequence,yi=softmax(Wthi)⏟token,Pistart=softmaxi(s⊤hi), Piend=softmaxi(e⊤hi)⏟span\underbrace{y=\mathrm{softmax}\big(W_c\tanh(W_p h_{\text{CLS}}+b_p)\big)}_{\text{sequence}},\quad \underbrace{y_i=\mathrm{softmax}(W_t h_i)}_{\text{token}},\quad \underbrace{P^{\text{start}}_i=\mathrm{softmax}_i(s^\top h_i),\ P^{\text{end}}_i=\mathrm{softmax}_i(e^\top h_i)}_{\text{span}}
hCLSh_{\text{CLS}}
the first token's output; it is a sentence summary only because fine-tuning makes it one
WpW_p
BERT's pooler, a d × d dense layer with tanh, pretrained through NSP
WtW_t
tag classifier applied to each token, e.g. 9 BIO tags for CoNLL-2003 NER
s,e∈Rds,e\in\mathbb R^d
start and end vectors for extractive QA (SQuAD); the answer is the best span with start ≤ end

All three heads are tiny, and fine-tuning updates the whole encoder with them. The heads differ in which outputs they read. Sentence-level tasks read one pooled vector. Token-level tasks read every hih_i, so their labels must be aligned to sub-word tokens; the usual choice is to label the first sub-token of each word. For pairs, such as entailment or a query and a passage, both texts go into one sequence, [CLS] A [SEP] B [SEP]. Attention then compares them token by token. This is a cross-encoder: accurate, but it must re-run for every pair.

Sentence embeddings and retrieval

Reimers and Gurevych ("Sentence-BERT", EMNLP 2019) pointed out that cross-encoding is hopeless for search. Finding the most similar pair among 10,000 sentences takes (100002)=49,995,000\binom{10000}{2}=49{,}995{,}000 forward passes, about 65 hours with BERT. Encoding each sentence once and comparing vectors takes seconds. Raw BERT vectors are poor for this, though. Averaged BERT outputs, or hCLSh_{\text{CLS}}, scored below averaged GloVe vectors on semantic textual similarity. SBERT fine-tunes a siamese BERT. Each sentence is mean-pooled to uu and vv, and a classifier on (u,v,∣u−v∣)(u,v,|u-v|) is trained on NLI pairs. At test time, cos⁡(u,v)\cos(u,v) is the similarity.

u=∑imihi∑imi,train: softmax(W[u; v; ∣u−v∣]),use: sim(a,b)=u⊤v∥u∥∥v∥u=\frac{\sum_i m_i h_i}{\sum_i m_i},\qquad \text{train: } \mathrm{softmax}\big(W[u;\,v;\,|u-v|]\big),\qquad \text{use: } \mathrm{sim}(a,b)=\frac{u^\top v}{\|u\|\|v\|}
mim_i
attention mask (1 for real tokens, 0 for padding); mean pooling was SBERT's default
[u;v;∣u−v∣][u;v;|u-v|]
concatenation fed to a softmax classifier over entailment / neutral / contradiction
sim\mathrm{sim}
cosine similarity; vectors are pre-computed and indexed for nearest-neighbour search

The same split organises neural retrieval. Bi-encoders such as DPR (Karpukhin et al., 2020, two BERT-base encoders compared by hCLSh_{\text{CLS}} dot product) index a corpus once. Late-interaction models such as ColBERT (Khattab and Zaharia, 2020) keep every token vector and score with a sum of max-similarities, a token-level head used for retrieval. Cross-encoders re-rank the top candidates. Most retrieval-augmented systems use an encoder-only model for at least one of these stages.

Representative configurations

BERT-base
12 layers, d=768d=768, 12 heads, FFN 3072, 512 positions, WordPiece 30,522; 110M parameters (109,482,240 with the pooler); BooksCorpus + English Wikipedia, about 3.3B words
BERT-large
24 layers, d=1024d=1024, 16 heads, FFN 4096; 340M parameters
RoBERTa
BERT architecture with byte-level BPE (50K), no NSP, dynamic masking, 160 GB of text, large batches
ELECTRA
generator 1/4 to 1/2 the discriminator's size, shared embeddings, λ=50\lambda=50, RTD loss on all tokens
ModernBERT
Warner et al. (2024): base 149M / 22 layers, large 395M / 28 layers; 8192-token context; 2T training tokens including code; RoPE, GeGLU, pre-norm, no biases; local 128-token attention with global attention every third layer; 30% masking, no NSP
Typical heads
sequence: pooler + d×kd\times k; token: d×d\times tags; QA: d×2d\times2; embedding: mean pooling, no parameters
h ← LN(h + softmax(QKᵀ/√d + M)V) → LN(h + MLP(h)) · d = 16, 2 layers, untrainedlayer 2 attention, sentence A, M = none (bidirectional)[CLS].12.13.11.21.18.16.09the.09.21.05.11.26.15.12bank.12.07.22.24.10.13.11of.23.07.19.13.08.11.20the.12.15.11.10.22.13.18river.20.17.02.13.21.15.12[SEP].11.09.26.21.12.12.10CLSthebankoftheriverSEProws: queries · columns: keysposition 2 (“bank”), sentence A vs Bafter layer 1: cos = 0.9651after layer 2: cos = 0.8918sentence vectors, cos(A, B)■ [CLS] pooling (BERT)0.9521■ mean pooling (SBERT)0.8094■ last token (decoder embedders)0.8714bidirectional: every output depends onevery token, so “bank” already differsafter one layer and [CLS] can summarisethe whole sentence.AB[CLS][CLS]0.048thethe0.031bankbank0.108ofraised0.616thethe0.054riverrates0.767[SEP][SEP]0.1291 − cos(h_i in A, h_i in B) after layer 2, per position
mask
input
fig.A seeded, untrained 16-dimensional encoder run on "[CLS] the bank of the river [SEP]" (A) and "[CLS] the bank raised the rates [SEP]" (B), which share the first three tokens. The weights are random, so the numbers measure which tokens can influence which outputs, not meaning. At the default (bidirectional, 2 layers), the attention row for "bank" spreads over all seven keys (.12 .07 .22 .24 .10 .13 .11 in A), and the two "bank" vectors drift apart: cos 0.9651 after layer 1 and 0.8918 after layer 2. Pooled sentence vectors differ too: [CLS] 0.9521, mean 0.8094, last token 0.8714. Per-position 1 − cos is 0.048 even for [CLS] and 0.767 at river/rates. Switch to the causal mask and positions 0–2 become exactly identical in A and B: 1 − cos is 0.000, and the [CLS] and "bank" cosines are 1.0000. [CLS] pooling is useless there, while the last token (0.8928) and mean pooling (0.9280) still see the difference. Replacing "bank" with [MASK] makes the context matter more: bidirectional cos 0.8529 and then 0.7027. Under the causal mask the masked slot stays at 1.0000, so it cannot use "river" or "rates" to guess the word.

Why encoders do not generate

MLM teaches conditionals p(xi∣x∖M)p(x_i\mid x_{\setminus\mathcal M}), the distribution of a few missing tokens given everything else. Generating text needs the joint distribution, and a set of conditionals with all context is not a left-to-right factorisation. You can sample from an MLM by repeatedly masking and resampling positions. Wang and Cho, "BERT has a Mouth, and It Must Speak" (2019), did this as Gibbs-style sampling. It is slow, the length must be fixed in advance, and quality lagged behind left-to-right models.

The cost structure also works against generation. Appending a token changes every earlier hih_i, because each position attends to the new token. Nothing can be cached, so each step re-encodes the whole sequence: O(n2)O(n^2) attention per new token, against O(n)O(n) with a KV cache in a decoder. The objectives that generate, causal LM or span corruption with a decoder (T5, BART), therefore live in other architectures. In the other direction, decoders can be turned into embedders by pooling the last token, which has seen everything under the causal mask. E5-Mistral (Wang et al., 2024) does this. LLM2Vec (BehnamGhader et al., 2024) instead re-enables bidirectional attention and trains briefly with masked next-token prediction.

Worked example: heads and masking on BERT-base

headreadsparametersexample
sequence classification (3 classes)h_CLS590,592 pooler + 2,307MNLI entailment
token classification (9 tags)every h_i6,921CoNLL-2003 NER
span extractionevery h_i1,538SQuAD
mean-pooled embeddingall h_i0SBERT, retrieval
MLM head (tied output)masked h_i622,650pretraining
ELECTRA discriminator headevery h_i591,361replaced-token detection
Head sizes on BERT-base (d = 768); embeddings + 12 layers are 108,891,648 parameters, 109,482,240 with the pooler (computed in Python)

One 512-token training sequence

  1. 1MLM picks 0.15⋅512=76.80.15\cdot512=76.8, about 77 positions. Of those, about 0.8⋅76.8=610.8\cdot76.8=61 become [MASK], about 8 become random tokens, and about 8 stay unchanged. The loss is computed at these 77 positions only.
  2. 2ELECTRA uses the same masking for the generator, but the discriminator's loss covers all 512 positions, about 512/77≈6.7×512/77\approx6.7\times as many targets per sequence. That is where its compute efficiency comes from.
  3. 3A 3-way classifier on hCLSh_{\text{CLS}} adds 768⋅768+768=590,592768\cdot768+768=590{,}592 (pooler) plus 768⋅3+3=2,307768\cdot3+3=2{,}307 parameters, about 0.54% of the encoder. A 9-tag NER head adds 768⋅9+9=6,921768\cdot9+9=6{,}921.
  4. 4The MLM head is a 768×768768\times768 dense layer with bias (590,592590{,}592), a LayerNorm (1,5361{,}536) and an output bias over the vocabulary (30,52230{,}522). Its 30,522×76830{,}522\times768 output matrix is the tied embedding, so it adds only 622,650622{,}650 parameters.
  5. 5Searching 10,000 sentences for the closest pair takes 10000⋅9999/2=49,995,00010000\cdot9999/2=49{,}995{,}000 cross-encoder passes. A bi-encoder needs 10,000 passes plus cheap dot products.
python
1import numpy as np
2
3rng = np.random.default_rng(0)
4V, MASK, CLS, SEP = 30522, 103, 101, 102 # bert-base-uncased ids for [MASK], [CLS], [SEP]
5
6def mlm_mask(ids, p=0.15):
7 """BERT's rule: pick 15% of non-special positions; 80% -> [MASK], 10% -> random, 10% unchanged."""
8 ids = ids.copy()
9 labels = np.full_like(ids, -100) # -100 = ignored by the loss
10 cand = np.where((ids != CLS) & (ids != SEP))[0]
11 pick = rng.choice(cand, size=round(p * len(cand)), replace=False)
12 labels[pick] = ids[pick]
13 u = rng.random(len(pick))
14 ids[pick[u < 0.8]] = MASK
15 rnd = pick[(u >= 0.8) & (u < 0.9)]
16 ids[rnd] = rng.integers(1000, V, len(rnd))
17 return ids, labels
18
19seq = np.concatenate([[CLS], rng.integers(1000, V, 510), [SEP]]) # 512 tokens
20x, y = mlm_mask(seq)
21picked = y != -100
22print("predicted positions:", picked.sum(), # 76 of 510
23 "| [MASK]:", (x == MASK).sum(), # 61
24 "| random:", (picked & (x != MASK) & (x != seq)).sum(), # 7
25 "| kept:", (picked & (x == seq)).sum()) # 8
26
27# ELECTRA: a generator fills the masked slots; the discriminator labels EVERY position.
28gen = x.copy()
29gen[x == MASK] = np.where(rng.random((x == MASK).sum()) < 0.3, # pretend the generator is right 30% of the time
30 seq[x == MASK], rng.integers(1000, V, (x == MASK).sum()))
31rtd = (gen != seq).astype(int) # 1 = replaced, 0 = original (a correct guess counts as original)
32print("RTD targets:", len(rtd), "positions,", rtd.sum(), "replaced") # 512 positions, 54 replaced
33
34# Sentence embeddings: mean-pool token vectors under the attention mask, then cosine.
35def mean_pool(h, attn_mask): # h: (B, T, d), attn_mask: (B, T)
36 m = attn_mask[..., None]
37 return (h * m).sum(1) / m.sum(1)
38
39h = rng.standard_normal((2, 6, 768)) # stand-in for encoder outputs
40attn = np.array([[1, 1, 1, 1, 1, 1], [1, 1, 1, 1, 0, 0]]) # second sentence is padded
41u, v = mean_pool(h, attn)
42print("cos(u, v) =", round(float(u @ v / np.linalg.norm(u) / np.linalg.norm(v)), 4)) # 0.0475 (random vectors)
BERT's 80/10/10 masking, ELECTRA's all-position targets and masked mean pooling (numpy; run as shown)
python
1import torch
2from transformers import AutoModel, AutoModelForSequenceClassification, AutoModelForTokenClassification, AutoTokenizer, pipeline
3
4# masked-LM head: fill in a blank using context on both sides
5fill = pipeline("fill-mask", model="bert-base-uncased")
6print([p["token_str"] for p in fill("He sat on the [MASK] of the river.")[:3]])
7
8# sequence and token heads: freshly initialised on top of the pretrained encoder, then fine-tuned
9clf = AutoModelForSequenceClassification.from_pretrained("bert-base-uncased", num_labels=3) # reads the pooled [CLS]
10ner = AutoModelForTokenClassification.from_pretrained("bert-base-uncased", num_labels=9) # reads every h_i
11
12# sentence embeddings: mean pooling over the attention mask
13name = "sentence-transformers/all-MiniLM-L6-v2"
14tok, enc = AutoTokenizer.from_pretrained(name), AutoModel.from_pretrained(name)
15batch = tok(["the bank of the river", "the bank raised the rates"], padding=True, return_tensors="pt")
16with torch.no_grad():
17 h = enc(**batch).last_hidden_state # (2, T, 384)
18m = batch["attention_mask"].unsqueeze(-1).float()
19emb = torch.nn.functional.normalize((h * m).sum(1) / m.sum(1), dim=-1)
20print("cosine:", (emb[0] @ emb[1]).item())
The same heads with Hugging Face transformers (not run here; needs torch and model downloads)

Use when

  • +Every output sees the full input: the best accuracy per parameter on classification, tagging, extractive QA and retrieval
  • +Small and fast: one forward pass per input, easy to batch, cheap to fine-tune
  • +Per-token outputs support token-level tasks and late-interaction retrieval directly
  • +Bi-encoder embeddings can be pre-computed and indexed for search over millions of documents

Avoid when

  • −Cannot generate text: no left-to-right factorisation and no KV cache
  • −MLM learns from only about 15% of positions per sequence (ELECTRA fixes this with RTD)
  • −Needs a task-specific head and labelled fine-tuning data; no zero-shot prompting
  • −Classic models are limited to 512 tokens; long-context encoders such as ModernBERT are recent

Check yourself

Two inputs share the prefix "[CLS] the bank". In which model are the output vectors at positions 0–2 guaranteed identical for both inputs?

Check yourself

Why does ELECTRA learn more per sequence than BERT's MLM at the same compute?

Remember

  • ✓Encoder-only = transformer encoder with mask M=0M=0: nn tokens in, nn contextual vectors out, each depending on the whole input.
  • ✓Pretraining: MLM on about 15% of tokens (80% [MASK] / 10% random / 10% kept), or ELECTRA's replaced-token detection on all tokens with LMLM(G)+50 LDisc(D)\mathcal L_{\text{MLM}}(G)+50\,\mathcal L_{\text{Disc}}(D).
  • ✓Heads: [CLS] + tanh pooler for sequences, WthiW_t h_i per token for NER, start/end vectors for QA, mean pooling + cosine for SBERT embeddings.
  • ✓Cross-encoders compare pairs jointly; bi-encoders embed once and index. Retrieval pipelines use both, plus late interaction (ColBERT) on token vectors.
  • ✓They do not generate: MLM gives conditionals, not a left-to-right factorisation, and every new token would change every earlier output, so nothing can be cached.
← back to Deep Learning roadmap