# Encoder-only models

> Encoder-only models (BERT, RoBERTa, ELECTRA, DeBERTa, ModernBERT) run a transformer encoder with no attention mask, so every output sees the whole input. Pretrained by masked- or replaced-token objectives, they feed [CLS], per-token or pooled vectors to task heads, but do not generate.

Source: https://www.mlfoundry.in/learn/encoder-only-models  
Section: Deep Learning › Large Language Models (LLMs)  
Level: core · 15 min read · updated 2026-10-13

**Encoder-only model:** is a language model made of the encoder half of the Transformer (Vaswani et al., 2017) alone: a stack of self-attention and feed-forward blocks with **no attention mask**, so the output at every position depends on the entire input. It maps $n$ tokens to $n$ contextual vectors $h_1,\dots,h_n$ and has no decoder. It is pretrained with a **denoising** objective, such as recovering masked tokens or detecting replaced ones, and fine-tuned by attaching small heads to those vectors. The family began with **BERT** (Devlin, Chang, Lee and Toutanova, "BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding", NAACL 2019). It includes **RoBERTa**, **ALBERT**, **DistilBERT**, ELECTRA, DeBERTa and ModernBERT.

In 2018 there were two ways to pretrain contextual representations. **GPT** trained a left-to-right transformer, so each token's vector saw only its left context. ELMo (Peters et al., 2018) trained separate left-to-right and right-to-left LSTMs and concatenated their states. It saw both sides, but only shallowly: no layer combined the two directions. The obstacle was the objective. A language model that predicts $x_t$ cannot also attend to $x_t$, or it would just copy it. BERT's fix was to *corrupt the input* instead of masking attention. It replaces some tokens with `[MASK]`, lets every layer attend everywhere, and predicts only the corrupted positions. The result was a deeply bidirectional encoder. On release it set the state of the art on GLUE, SQuAD and SWAG.

The family is defined by what it gives up. Without a causal mask there is no factorisation $p(x)=\prod_t p(x_t\mid x_{<t})$, so the model cannot be sampled left to right. In return, each output sees the full context. That is what classification, tagging, extractive QA and retrieval need, because the whole input is available before any decision is made. The other two shapes are covered in **Decoder-only models** and **Encoder–decoder models**.

## Bidirectional attention

$$
H^{(\ell)}=\mathrm{LN}\Big(A^{(\ell)}+\mathrm{FFN}\big(A^{(\ell)}\big)\Big),\quad A^{(\ell)}=\mathrm{LN}\Big(H^{(\ell-1)}+\mathrm{softmax}\Big(\frac{QK^\top}{\sqrt{d_h}}+M\Big)V\Big),\qquad M_{ij}=0\ \ \forall i,j
$$

- $M$: attention mask; an encoder uses $M=0$ (only padding is masked), a decoder uses $M_{ij}=-\infty$ for $j>i$ (see **Causal attention**)
- $H^{(0)}$: token + position (+ segment) embeddings; BERT uses a 30,522-token **WordPiece** vocabulary and 512 learned positions
- $\mathrm{LN}$: post-norm in BERT/RoBERTa; ModernBERT uses pre-norm
- $h_i=H^{(L)}_i$: the contextual vector of token $i$, a function of **all** $n$ tokens

The mask is the whole difference from a decoder block. The parameter count, FFN and norms are the same, and **Transformer encoder** covers the block itself. The consequence is easy to state and easy to check. Under a causal mask, $h_i$ depends only on $x_{\le i}$. Two sentences with the same prefix therefore produce *identical* vectors for that prefix, and the first token sees nothing but itself. Under the encoder mask, every $h_i$ changes when any token changes. The interactive figure below runs both masks on the same weights.

> Figure: An encoder-only model and its heads (click a head on the right). Tokens, with [CLS] first and [SEP] last, are embedded and passed through L blocks of unmasked self-attention and FFN. The model returns one vector per token. Sequence classification reads only h_CLS through a tanh pooler. Token classification and extractive QA read every h_i. Sentence embeddings mean-pool all of them. The masked-LM head (pretraining) reads the [MASK] position. ELECTRA's discriminator classifies every position as original or replaced. Right: the encoder's all-pairs mask next to a decoder's j ≤ i mask.

## Pretraining objectives

$$
\mathcal L_{\text{MLM}}=-\sum_{i\in\mathcal M}\log\,\mathrm{softmax}\big(E\,g(h_i)+b\big)_{x_i},\qquad |\mathcal M|\approx0.15\,n,\quad \tilde x_i=\begin{cases}\texttt{[MASK]}&80\%\\\text{random token}&10\%\\x_i&10\%\end{cases}
$$

- $\mathcal M$: the positions chosen for prediction, 15% of tokens in BERT, 30% in ModernBERT
- $g$: a dense + GELU + LayerNorm transform; the output matrix is tied to the input embedding $E$
- $80/10/10$: so the model cannot rely on `[MASK]` appearing, since `[MASK]` never appears at fine-tuning time
- $h_i$: computed from the *corrupted* sequence $\tilde x$, using context on both sides

BERT added a **next-sentence prediction** (NSP) loss on $h_{\text{CLS}}$: was segment B the true continuation of segment A? **RoBERTa** (Liu et al., 2019) found that NSP did not help. It dropped NSP, masked dynamically instead of once, trained longer on 160 GB of text with larger batches, and beat BERT clearly. Since then, most encoders train on MLM alone. **ALBERT** replaced NSP with sentence-order prediction and shared weights across layers. **DistilBERT** (Sanh et al., 2019) distilled BERT into 6 layers, 40% smaller and 60% faster while keeping 97% of its GLUE performance.

$$
\mathcal L=\mathcal L_{\text{MLM}}(G)+\lambda\,\mathcal L_{\text{Disc}}(D),\quad \mathcal L_{\text{Disc}}=-\sum_{t=1}^{n}\Big[\mathbb 1(\hat x_t=x_t)\log D(\hat x,t)+\mathbb 1(\hat x_t\neq x_t)\log\big(1-D(\hat x,t)\big)\Big],\quad \lambda=50
$$

- $G$: a small MLM **generator** (1/4 to 1/2 of the discriminator's size) that fills the masked slots by sampling
- $\hat x$: the input with masked slots replaced by generator samples
- $D(\hat x,t)$: $\sigma(w^\top h_t)$, the discriminator's probability that token $t$ is original; a correct generator guess counts as original
- $\sum_{t=1}^{n}$: the loss covers all positions, not only the 15% that were masked

MLM learns from only about 15% of positions per sequence. **ELECTRA** (Clark, Luong, Le and Manning, "ELECTRA: Pre-training Text Encoders as Discriminators Rather Than Generators", ICLR 2020) turned pretraining into **replaced-token detection** (RTD) over every position. The generator is trained by maximum likelihood, not adversarially, and only the discriminator is kept. ELECTRA-Small, trained for 4 days on one GPU, beat GPT on GLUE, although GPT used about 30× more compute. ELECTRA-Large matched RoBERTa and XLNet with about a quarter of their compute. **DeBERTa** (He et al., ICLR 2021) instead changed attention. It keeps separate content and relative-position vectors, which interact through disentangled attention. A 1.5B version was the first single model to pass the human baseline on SuperGLUE.

## Task heads: token outputs vs pooled outputs

$$
\underbrace{y=\mathrm{softmax}\big(W_c\tanh(W_p h_{\text{CLS}}+b_p)\big)}_{\text{sequence}},\quad \underbrace{y_i=\mathrm{softmax}(W_t h_i)}_{\text{token}},\quad \underbrace{P^{\text{start}}_i=\mathrm{softmax}_i(s^\top h_i),\ P^{\text{end}}_i=\mathrm{softmax}_i(e^\top h_i)}_{\text{span}}
$$

- $h_{\text{CLS}}$: the first token's output; it is a sentence summary only because fine-tuning makes it one
- $W_p$: BERT's pooler, a d × d dense layer with tanh, pretrained through NSP
- $W_t$: tag classifier applied to each token, e.g. 9 BIO tags for CoNLL-2003 NER
- $s,e\in\mathbb R^d$: start and end vectors for extractive QA (SQuAD); the answer is the best span with start ≤ end

All three heads are tiny, and fine-tuning updates the whole encoder with them. The heads differ in *which outputs they read*. Sentence-level tasks read one pooled vector. Token-level tasks read every $h_i$, so their labels must be aligned to sub-word tokens; the usual choice is to label the first sub-token of each word. For pairs, such as entailment or a query and a passage, both texts go into one sequence, `[CLS] A [SEP] B [SEP]`. Attention then compares them token by token. This is a **cross-encoder**: accurate, but it must re-run for every pair.

## Sentence embeddings and retrieval

Reimers and Gurevych ("Sentence-BERT", EMNLP 2019) pointed out that cross-encoding is hopeless for search. Finding the most similar pair among 10,000 sentences takes $\binom{10000}{2}=49{,}995{,}000$ forward passes, about 65 hours with BERT. Encoding each sentence once and comparing vectors takes seconds. Raw BERT vectors are poor for this, though. Averaged BERT outputs, or $h_{\text{CLS}}$, scored *below* averaged GloVe vectors on semantic textual similarity. SBERT fine-tunes a siamese BERT. Each sentence is mean-pooled to $u$ and $v$, and a classifier on $(u,v,|u-v|)$ is trained on NLI pairs. At test time, $\cos(u,v)$ is the similarity.

$$
u=\frac{\sum_i m_i h_i}{\sum_i m_i},\qquad \text{train: } \mathrm{softmax}\big(W[u;\,v;\,|u-v|]\big),\qquad \text{use: } \mathrm{sim}(a,b)=\frac{u^\top v}{\|u\|\|v\|}
$$

- $m_i$: attention mask (1 for real tokens, 0 for padding); mean pooling was SBERT's default
- $[u;v;|u-v|]$: concatenation fed to a softmax classifier over entailment / neutral / contradiction
- $\mathrm{sim}$: cosine similarity; vectors are pre-computed and indexed for nearest-neighbour search

The same split organises neural retrieval. **Bi-encoders** such as DPR (Karpukhin et al., 2020, two BERT-base encoders compared by $h_{\text{CLS}}$ dot product) index a corpus once. **Late-interaction** models such as ColBERT (Khattab and Zaharia, 2020) keep every token vector and score with a sum of max-similarities, a token-level head used for retrieval. **Cross-encoders** re-rank the top candidates. Most retrieval-augmented systems use an encoder-only model for at least one of these stages.

**Representative configurations**

- **BERT-base:** 12 layers, $d=768$, 12 heads, FFN 3072, 512 positions, WordPiece 30,522; 110M parameters (109,482,240 with the pooler); BooksCorpus + English Wikipedia, about 3.3B words
- **BERT-large:** 24 layers, $d=1024$, 16 heads, FFN 4096; 340M parameters
- **RoBERTa:** BERT architecture with byte-level BPE (50K), no NSP, dynamic masking, 160 GB of text, large batches
- **ELECTRA:** generator 1/4 to 1/2 the discriminator's size, shared embeddings, $\lambda=50$, RTD loss on all tokens
- **ModernBERT:** Warner et al. (2024): base 149M / 22 layers, large 395M / 28 layers; 8192-token context; 2T training tokens including code; RoPE, GeGLU, pre-norm, no biases; local 128-token attention with global attention every third layer; 30% masking, no NSP
- **Typical heads:** sequence: pooler + $d\times k$; token: $d\times$ tags; QA: $d\times2$; embedding: mean pooling, no parameters

> Figure: A seeded, untrained 16-dimensional encoder run on "[CLS] the bank of the river [SEP]" (A) and "[CLS] the bank raised the rates [SEP]" (B), which share the first three tokens. The weights are random, so the numbers measure which tokens can influence which outputs, not meaning. At the default (bidirectional, 2 layers), the attention row for "bank" spreads over all seven keys (.12 .07 .22 .24 .10 .13 .11 in A), and the two "bank" vectors drift apart: cos 0.9651 after layer 1 and 0.8918 after layer 2. Pooled sentence vectors differ too: [CLS] 0.9521, mean 0.8094, last token 0.8714. Per-position 1 − cos is 0.048 even for [CLS] and 0.767 at river/rates. Switch to the causal mask and positions 0–2 become exactly identical in A and B: 1 − cos is 0.000, and the [CLS] and "bank" cosines are 1.0000. [CLS] pooling is useless there, while the last token (0.8928) and mean pooling (0.9280) still see the difference. Replacing "bank" with [MASK] makes the context matter more: bidirectional cos 0.8529 and then 0.7027. Under the causal mask the masked slot stays at 1.0000, so it cannot use "river" or "rates" to guess the word.

## Why encoders do not generate

MLM teaches conditionals $p(x_i\mid x_{\setminus\mathcal M})$, the distribution of a few missing tokens given everything else. Generating text needs the joint distribution, and a set of conditionals with all context is not a left-to-right factorisation. You can sample from an MLM by repeatedly masking and resampling positions. Wang and Cho, "BERT has a Mouth, and It Must Speak" (2019), did this as Gibbs-style sampling. It is slow, the length must be fixed in advance, and quality lagged behind left-to-right models.

The cost structure also works against generation. Appending a token changes every earlier $h_i$, because each position attends to the new token. Nothing can be cached, so each step re-encodes the whole sequence: $O(n^2)$ attention per new token, against $O(n)$ with a **KV cache** in a decoder. The objectives that generate, causal LM or span corruption with a decoder (**T5**, **BART**), therefore live in other architectures. In the other direction, decoders can be turned into embedders by pooling the last token, which has seen everything under the causal mask. E5-Mistral (Wang et al., 2024) does this. LLM2Vec (BehnamGhader et al., 2024) instead re-enables bidirectional attention and trains briefly with masked next-token prediction.

## Worked example: heads and masking on BERT-base

Head sizes on BERT-base (d = 768); embeddings + 12 layers are 108,891,648 parameters, 109,482,240 with the pooler (computed in Python)

| head | reads | parameters | example |
| --- | --- | --- | --- |
| sequence classification (3 classes) | h_CLS | 590,592 pooler + 2,307 | MNLI entailment |
| token classification (9 tags) | every h_i | 6,921 | CoNLL-2003 NER |
| span extraction | every h_i | 1,538 | SQuAD |
| mean-pooled embedding | all h_i | 0 | SBERT, retrieval |
| MLM head (tied output) | masked h_i | 622,650 | pretraining |
| ELECTRA discriminator head | every h_i | 591,361 | replaced-token detection |

**One 512-token training sequence**

1. MLM picks $0.15\cdot512=76.8$, about 77 positions. Of those, about $0.8\cdot76.8=61$ become `[MASK]`, about 8 become random tokens, and about 8 stay unchanged. The loss is computed at these 77 positions only.
2. ELECTRA uses the same masking for the generator, but the discriminator's loss covers all 512 positions, about $512/77\approx6.7\times$ as many targets per sequence. That is where its compute efficiency comes from.
3. A 3-way classifier on $h_{\text{CLS}}$ adds $768\cdot768+768=590{,}592$ (pooler) plus $768\cdot3+3=2{,}307$ parameters, about 0.54% of the encoder. A 9-tag NER head adds $768\cdot9+9=6{,}921$.
4. The MLM head is a $768\times768$ dense layer with bias ($590{,}592$), a LayerNorm ($1{,}536$) and an output bias over the vocabulary ($30{,}522$). Its $30{,}522\times768$ output matrix is the tied embedding, so it adds only $622{,}650$ parameters.
5. Searching 10,000 sentences for the closest pair takes $10000\cdot9999/2=49{,}995{,}000$ cross-encoder passes. A bi-encoder needs 10,000 passes plus cheap dot products.

```python
import numpy as np

rng = np.random.default_rng(0)
V, MASK, CLS, SEP = 30522, 103, 101, 102           # bert-base-uncased ids for [MASK], [CLS], [SEP]

def mlm_mask(ids, p=0.15):
    """BERT's rule: pick 15% of non-special positions; 80% -> [MASK], 10% -> random, 10% unchanged."""
    ids = ids.copy()
    labels = np.full_like(ids, -100)                  # -100 = ignored by the loss
    cand = np.where((ids != CLS) & (ids != SEP))[0]
    pick = rng.choice(cand, size=round(p * len(cand)), replace=False)
    labels[pick] = ids[pick]
    u = rng.random(len(pick))
    ids[pick[u < 0.8]] = MASK
    rnd = pick[(u >= 0.8) & (u < 0.9)]
    ids[rnd] = rng.integers(1000, V, len(rnd))
    return ids, labels

seq = np.concatenate([[CLS], rng.integers(1000, V, 510), [SEP]])   # 512 tokens
x, y = mlm_mask(seq)
picked = y != -100
print("predicted positions:", picked.sum(),                         # 76 of 510
      "| [MASK]:", (x == MASK).sum(),                               # 61
      "| random:", (picked & (x != MASK) & (x != seq)).sum(),       # 7
      "| kept:", (picked & (x == seq)).sum())                       # 8

# ELECTRA: a generator fills the masked slots; the discriminator labels EVERY position.
gen = x.copy()
gen[x == MASK] = np.where(rng.random((x == MASK).sum()) < 0.3,  # pretend the generator is right 30% of the time
                          seq[x == MASK], rng.integers(1000, V, (x == MASK).sum()))
rtd = (gen != seq).astype(int)                      # 1 = replaced, 0 = original (a correct guess counts as original)
print("RTD targets:", len(rtd), "positions,", rtd.sum(), "replaced")  # 512 positions, 54 replaced

# Sentence embeddings: mean-pool token vectors under the attention mask, then cosine.
def mean_pool(h, attn_mask):                        # h: (B, T, d), attn_mask: (B, T)
    m = attn_mask[..., None]
    return (h * m).sum(1) / m.sum(1)

h = rng.standard_normal((2, 6, 768))                # stand-in for encoder outputs
attn = np.array([[1, 1, 1, 1, 1, 1], [1, 1, 1, 1, 0, 0]])           # second sentence is padded
u, v = mean_pool(h, attn)
print("cos(u, v) =", round(float(u @ v / np.linalg.norm(u) / np.linalg.norm(v)), 4))  # 0.0475 (random vectors)
```

BERT's 80/10/10 masking, ELECTRA's all-position targets and masked mean pooling (numpy; run as shown)

```python
import torch
from transformers import AutoModel, AutoModelForSequenceClassification, AutoModelForTokenClassification, AutoTokenizer, pipeline

# masked-LM head: fill in a blank using context on both sides
fill = pipeline("fill-mask", model="bert-base-uncased")
print([p["token_str"] for p in fill("He sat on the [MASK] of the river.")[:3]])

# sequence and token heads: freshly initialised on top of the pretrained encoder, then fine-tuned
clf = AutoModelForSequenceClassification.from_pretrained("bert-base-uncased", num_labels=3)    # reads the pooled [CLS]
ner = AutoModelForTokenClassification.from_pretrained("bert-base-uncased", num_labels=9)       # reads every h_i

# sentence embeddings: mean pooling over the attention mask
name = "sentence-transformers/all-MiniLM-L6-v2"
tok, enc = AutoTokenizer.from_pretrained(name), AutoModel.from_pretrained(name)
batch = tok(["the bank of the river", "the bank raised the rates"], padding=True, return_tensors="pt")
with torch.no_grad():
    h = enc(**batch).last_hidden_state                       # (2, T, 384)
m = batch["attention_mask"].unsqueeze(-1).float()
emb = torch.nn.functional.normalize((h * m).sum(1) / m.sum(1), dim=-1)
print("cosine:", (emb[0] @ emb[1]).item())
```

The same heads with Hugging Face transformers (not run here; needs torch and model downloads)

> **[CLS] is not a sentence embedding out of the box:** Pretrained BERT's $h_{\text{CLS}}$ was trained only for NSP. It is a good *starting point* for a fine-tuned classifier, but as a frozen similarity vector it scores worse than averaged GloVe on STS. Use a model trained for embeddings (SBERT-style contrastive or NLI training), and use the pooling it was trained with: mean, [CLS] or last token. Mixing them up silently ruins retrieval quality.

> **The mask decides what a position can know:** The weights of an encoder and a decoder block are interchangeable. Only $M$ differs. In the figure the same random weights give identical prefix vectors under the causal mask and different ones without it. A causal [CLS] at position 0 sees only itself. A causal `[MASK]` cannot see the word that disambiguates it. That one matrix explains why encoders win on understanding tasks at equal size, and why decoder embedders pool the last token.

> **Encoders are still the workhorse:** Encoder-only models are small, fast and non-generative. That makes them the default for high-volume classification, PII and content filtering, NER, reranking and embedding-based retrieval, where every input must be scored cheaply. ModernBERT's 8192-token context and code-aware training answer the old 512-token limit. A 149M model can encode a whole document in one pass.

**Strengths**

- Every output sees the full input: the best accuracy per parameter on classification, tagging, extractive QA and retrieval
- Small and fast: one forward pass per input, easy to batch, cheap to fine-tune
- Per-token outputs support token-level tasks and late-interaction retrieval directly
- Bi-encoder embeddings can be pre-computed and indexed for search over millions of documents

**Limitations**

- Cannot generate text: no left-to-right factorisation and no KV cache
- MLM learns from only about 15% of positions per sequence (ELECTRA fixes this with RTD)
- Needs a task-specific head and labelled fine-tuning data; no zero-shot prompting
- Classic models are limited to 512 tokens; long-context encoders such as ModernBERT are recent

**Question:** Two inputs share the prefix "[CLS] the bank". In which model are the output vectors at positions 0–2 guaranteed identical for both inputs?

- A. A BERT-style encoder, because the weights are shared
- B. A causal decoder, because position i attends only to positions ≤ i
- C. Both, because the tokens are the same
- D. Neither, because dropout makes them differ

**Answer:** B. A causal decoder, because position i attends only to positions ≤ i — Under the causal mask, $h_i$ is a function of $x_{\le i}$ only, so equal prefixes give equal vectors (in eval mode). In an encoder every position attends to the whole sequence, so "river" versus "rates" changes even $h_{\text{CLS}}$. The figure shows 1 − cos = 0.000 at positions 0–2 under the causal mask and 0.048, 0.031 and 0.108 without it.

**Question:** Why does ELECTRA learn more per sequence than BERT's MLM at the same compute?

- A. It uses a larger vocabulary
- B. Its discriminator gets a loss at every position (original vs replaced), not only at the 15% that were masked
- C. It trains the generator adversarially
- D. It removes the [CLS] token

**Answer:** B. Its discriminator gets a loss at every position (original vs replaced), not only at the 15% that were masked — Replaced-token detection turns all $n$ positions into binary targets, about $512/77\approx6.7\times$ more targets per 512-token sequence. The generator is trained by maximum likelihood, not adversarially. ELECTRA-Small beat GPT on GLUE with about 1/30 of its compute.

**Key points**

- Encoder-only = transformer encoder with mask $M=0$: $n$ tokens in, $n$ contextual vectors out, each depending on the whole input.
- Pretraining: MLM on about 15% of tokens (80% [MASK] / 10% random / 10% kept), or ELECTRA's replaced-token detection on all tokens with $\mathcal L_{\text{MLM}}(G)+50\,\mathcal L_{\text{Disc}}(D)$.
- Heads: [CLS] + tanh pooler for sequences, $W_t h_i$ per token for NER, start/end vectors for QA, mean pooling + cosine for SBERT embeddings.
- Cross-encoders compare pairs jointly; bi-encoders embed once and index. Retrieval pipelines use both, plus late interaction (ColBERT) on token vectors.
- They do not generate: MLM gives conditionals, not a left-to-right factorisation, and every new token would change every earlier output, so nothing can be cached.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
