Definition
Encoder-only model is a language model made of the encoder half of the Transformer (Vaswani et al., 2017) alone: a stack of self-attention and feed-forward blocks with no attention mask, so the output at every position depends on the entire input. It maps tokens to contextual vectors and has no decoder. It is pretrained with a denoising objective, such as recovering masked tokens or detecting replaced ones, and fine-tuned by attaching small heads to those vectors. The family began with BERT (Devlin, Chang, Lee and Toutanova, "BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding", NAACL 2019). It includes RoBERTa, ALBERT, DistilBERT, ELECTRA, DeBERTa and ModernBERT.
In 2018 there were two ways to pretrain contextual representations. GPT trained a left-to-right transformer, so each token's vector saw only its left context. ELMo (Peters et al., 2018) trained separate left-to-right and right-to-left LSTMs and concatenated their states. It saw both sides, but only shallowly: no layer combined the two directions. The obstacle was the objective. A language model that predicts cannot also attend to , or it would just copy it. BERT's fix was to corrupt the input instead of masking attention. It replaces some tokens with [MASK], lets every layer attend everywhere, and predicts only the corrupted positions. The result was a deeply bidirectional encoder. On release it set the state of the art on GLUE, SQuAD and SWAG.
The family is defined by what it gives up. Without a causal mask there is no factorisation , so the model cannot be sampled left to right. In return, each output sees the full context. That is what classification, tagging, extractive QA and retrieval need, because the whole input is available before any decision is made. The other two shapes are covered in Decoder-only models and Encoder–decoder models.
Bidirectional attention
- attention mask; an encoder uses (only padding is masked), a decoder uses for (see Causal attention)
- token + position (+ segment) embeddings; BERT uses a 30,522-token WordPiece vocabulary and 512 learned positions
- post-norm in BERT/RoBERTa; ModernBERT uses pre-norm
- the contextual vector of token , a function of all tokens
The mask is the whole difference from a decoder block. The parameter count, FFN and norms are the same, and Transformer encoder covers the block itself. The consequence is easy to state and easy to check. Under a causal mask, depends only on . Two sentences with the same prefix therefore produce identical vectors for that prefix, and the first token sees nothing but itself. Under the encoder mask, every changes when any token changes. The interactive figure below runs both masks on the same weights.
Pretraining objectives
- the positions chosen for prediction, 15% of tokens in BERT, 30% in ModernBERT
- a dense + GELU + LayerNorm transform; the output matrix is tied to the input embedding
- so the model cannot rely on
[MASK]appearing, since[MASK]never appears at fine-tuning time - computed from the corrupted sequence , using context on both sides
BERT added a next-sentence prediction (NSP) loss on : was segment B the true continuation of segment A? RoBERTa (Liu et al., 2019) found that NSP did not help. It dropped NSP, masked dynamically instead of once, trained longer on 160 GB of text with larger batches, and beat BERT clearly. Since then, most encoders train on MLM alone. ALBERT replaced NSP with sentence-order prediction and shared weights across layers. DistilBERT (Sanh et al., 2019) distilled BERT into 6 layers, 40% smaller and 60% faster while keeping 97% of its GLUE performance.
- a small MLM generator (1/4 to 1/2 of the discriminator's size) that fills the masked slots by sampling
- the input with masked slots replaced by generator samples
- , the discriminator's probability that token is original; a correct generator guess counts as original
- the loss covers all positions, not only the 15% that were masked
MLM learns from only about 15% of positions per sequence. ELECTRA (Clark, Luong, Le and Manning, "ELECTRA: Pre-training Text Encoders as Discriminators Rather Than Generators", ICLR 2020) turned pretraining into replaced-token detection (RTD) over every position. The generator is trained by maximum likelihood, not adversarially, and only the discriminator is kept. ELECTRA-Small, trained for 4 days on one GPU, beat GPT on GLUE, although GPT used about 30× more compute. ELECTRA-Large matched RoBERTa and XLNet with about a quarter of their compute. DeBERTa (He et al., ICLR 2021) instead changed attention. It keeps separate content and relative-position vectors, which interact through disentangled attention. A 1.5B version was the first single model to pass the human baseline on SuperGLUE.
Task heads: token outputs vs pooled outputs
- the first token's output; it is a sentence summary only because fine-tuning makes it one
- BERT's pooler, a d × d dense layer with tanh, pretrained through NSP
- tag classifier applied to each token, e.g. 9 BIO tags for CoNLL-2003 NER
- start and end vectors for extractive QA (SQuAD); the answer is the best span with start ≤ end
All three heads are tiny, and fine-tuning updates the whole encoder with them. The heads differ in which outputs they read. Sentence-level tasks read one pooled vector. Token-level tasks read every , so their labels must be aligned to sub-word tokens; the usual choice is to label the first sub-token of each word. For pairs, such as entailment or a query and a passage, both texts go into one sequence, [CLS] A [SEP] B [SEP]. Attention then compares them token by token. This is a cross-encoder: accurate, but it must re-run for every pair.
Sentence embeddings and retrieval
Reimers and Gurevych ("Sentence-BERT", EMNLP 2019) pointed out that cross-encoding is hopeless for search. Finding the most similar pair among 10,000 sentences takes forward passes, about 65 hours with BERT. Encoding each sentence once and comparing vectors takes seconds. Raw BERT vectors are poor for this, though. Averaged BERT outputs, or , scored below averaged GloVe vectors on semantic textual similarity. SBERT fine-tunes a siamese BERT. Each sentence is mean-pooled to and , and a classifier on is trained on NLI pairs. At test time, is the similarity.
- attention mask (1 for real tokens, 0 for padding); mean pooling was SBERT's default
- concatenation fed to a softmax classifier over entailment / neutral / contradiction
- cosine similarity; vectors are pre-computed and indexed for nearest-neighbour search
The same split organises neural retrieval. Bi-encoders such as DPR (Karpukhin et al., 2020, two BERT-base encoders compared by dot product) index a corpus once. Late-interaction models such as ColBERT (Khattab and Zaharia, 2020) keep every token vector and score with a sum of max-similarities, a token-level head used for retrieval. Cross-encoders re-rank the top candidates. Most retrieval-augmented systems use an encoder-only model for at least one of these stages.
Representative configurations
- BERT-base
- 12 layers, , 12 heads, FFN 3072, 512 positions, WordPiece 30,522; 110M parameters (109,482,240 with the pooler); BooksCorpus + English Wikipedia, about 3.3B words
- BERT-large
- 24 layers, , 16 heads, FFN 4096; 340M parameters
- RoBERTa
- BERT architecture with byte-level BPE (50K), no NSP, dynamic masking, 160 GB of text, large batches
- ELECTRA
- generator 1/4 to 1/2 the discriminator's size, shared embeddings, , RTD loss on all tokens
- ModernBERT
- Warner et al. (2024): base 149M / 22 layers, large 395M / 28 layers; 8192-token context; 2T training tokens including code; RoPE, GeGLU, pre-norm, no biases; local 128-token attention with global attention every third layer; 30% masking, no NSP
- Typical heads
- sequence: pooler + ; token: tags; QA: ; embedding: mean pooling, no parameters
Why encoders do not generate
MLM teaches conditionals , the distribution of a few missing tokens given everything else. Generating text needs the joint distribution, and a set of conditionals with all context is not a left-to-right factorisation. You can sample from an MLM by repeatedly masking and resampling positions. Wang and Cho, "BERT has a Mouth, and It Must Speak" (2019), did this as Gibbs-style sampling. It is slow, the length must be fixed in advance, and quality lagged behind left-to-right models.
The cost structure also works against generation. Appending a token changes every earlier , because each position attends to the new token. Nothing can be cached, so each step re-encodes the whole sequence: attention per new token, against with a KV cache in a decoder. The objectives that generate, causal LM or span corruption with a decoder (T5, BART), therefore live in other architectures. In the other direction, decoders can be turned into embedders by pooling the last token, which has seen everything under the causal mask. E5-Mistral (Wang et al., 2024) does this. LLM2Vec (BehnamGhader et al., 2024) instead re-enables bidirectional attention and trains briefly with masked next-token prediction.
Worked example: heads and masking on BERT-base
| head | reads | parameters | example |
|---|---|---|---|
| sequence classification (3 classes) | h_CLS | 590,592 pooler + 2,307 | MNLI entailment |
| token classification (9 tags) | every h_i | 6,921 | CoNLL-2003 NER |
| span extraction | every h_i | 1,538 | SQuAD |
| mean-pooled embedding | all h_i | 0 | SBERT, retrieval |
| MLM head (tied output) | masked h_i | 622,650 | pretraining |
| ELECTRA discriminator head | every h_i | 591,361 | replaced-token detection |
One 512-token training sequence
- 1MLM picks , about 77 positions. Of those, about become
[MASK], about 8 become random tokens, and about 8 stay unchanged. The loss is computed at these 77 positions only. - 2ELECTRA uses the same masking for the generator, but the discriminator's loss covers all 512 positions, about as many targets per sequence. That is where its compute efficiency comes from.
- 3A 3-way classifier on adds (pooler) plus parameters, about 0.54% of the encoder. A 9-tag NER head adds .
- 4The MLM head is a dense layer with bias (), a LayerNorm () and an output bias over the vocabulary (). Its output matrix is the tied embedding, so it adds only parameters.
- 5Searching 10,000 sentences for the closest pair takes cross-encoder passes. A bi-encoder needs 10,000 passes plus cheap dot products.
1import numpy as np23rng = np.random.default_rng(0)4V, MASK, CLS, SEP = 30522, 103, 101, 102 # bert-base-uncased ids for [MASK], [CLS], [SEP]56def mlm_mask(ids, p=0.15):7 """BERT's rule: pick 15% of non-special positions; 80% -> [MASK], 10% -> random, 10% unchanged."""8 ids = ids.copy()9 labels = np.full_like(ids, -100) # -100 = ignored by the loss10 cand = np.where((ids != CLS) & (ids != SEP))[0]11 pick = rng.choice(cand, size=round(p * len(cand)), replace=False)12 labels[pick] = ids[pick]13 u = rng.random(len(pick))14 ids[pick[u < 0.8]] = MASK15 rnd = pick[(u >= 0.8) & (u < 0.9)]16 ids[rnd] = rng.integers(1000, V, len(rnd))17 return ids, labels1819seq = np.concatenate([[CLS], rng.integers(1000, V, 510), [SEP]]) # 512 tokens20x, y = mlm_mask(seq)21picked = y != -10022print("predicted positions:", picked.sum(), # 76 of 51023 "| [MASK]:", (x == MASK).sum(), # 6124 "| random:", (picked & (x != MASK) & (x != seq)).sum(), # 725 "| kept:", (picked & (x == seq)).sum()) # 82627# ELECTRA: a generator fills the masked slots; the discriminator labels EVERY position.28gen = x.copy()29gen[x == MASK] = np.where(rng.random((x == MASK).sum()) < 0.3, # pretend the generator is right 30% of the time30 seq[x == MASK], rng.integers(1000, V, (x == MASK).sum()))31rtd = (gen != seq).astype(int) # 1 = replaced, 0 = original (a correct guess counts as original)32print("RTD targets:", len(rtd), "positions,", rtd.sum(), "replaced") # 512 positions, 54 replaced3334# Sentence embeddings: mean-pool token vectors under the attention mask, then cosine.35def mean_pool(h, attn_mask): # h: (B, T, d), attn_mask: (B, T)36 m = attn_mask[..., None]37 return (h * m).sum(1) / m.sum(1)3839h = rng.standard_normal((2, 6, 768)) # stand-in for encoder outputs40attn = np.array([[1, 1, 1, 1, 1, 1], [1, 1, 1, 1, 0, 0]]) # second sentence is padded41u, v = mean_pool(h, attn)42print("cos(u, v) =", round(float(u @ v / np.linalg.norm(u) / np.linalg.norm(v)), 4)) # 0.0475 (random vectors)1import torch2from transformers import AutoModel, AutoModelForSequenceClassification, AutoModelForTokenClassification, AutoTokenizer, pipeline34# masked-LM head: fill in a blank using context on both sides5fill = pipeline("fill-mask", model="bert-base-uncased")6print([p["token_str"] for p in fill("He sat on the [MASK] of the river.")[:3]])78# sequence and token heads: freshly initialised on top of the pretrained encoder, then fine-tuned9clf = AutoModelForSequenceClassification.from_pretrained("bert-base-uncased", num_labels=3) # reads the pooled [CLS]10ner = AutoModelForTokenClassification.from_pretrained("bert-base-uncased", num_labels=9) # reads every h_i1112# sentence embeddings: mean pooling over the attention mask13name = "sentence-transformers/all-MiniLM-L6-v2"14tok, enc = AutoTokenizer.from_pretrained(name), AutoModel.from_pretrained(name)15batch = tok(["the bank of the river", "the bank raised the rates"], padding=True, return_tensors="pt")16with torch.no_grad():17 h = enc(**batch).last_hidden_state # (2, T, 384)18m = batch["attention_mask"].unsqueeze(-1).float()19emb = torch.nn.functional.normalize((h * m).sum(1) / m.sum(1), dim=-1)20print("cosine:", (emb[0] @ emb[1]).item())Use when
- +Every output sees the full input: the best accuracy per parameter on classification, tagging, extractive QA and retrieval
- +Small and fast: one forward pass per input, easy to batch, cheap to fine-tune
- +Per-token outputs support token-level tasks and late-interaction retrieval directly
- +Bi-encoder embeddings can be pre-computed and indexed for search over millions of documents
Avoid when
- −Cannot generate text: no left-to-right factorisation and no KV cache
- −MLM learns from only about 15% of positions per sequence (ELECTRA fixes this with RTD)
- −Needs a task-specific head and labelled fine-tuning data; no zero-shot prompting
- −Classic models are limited to 512 tokens; long-context encoders such as ModernBERT are recent
Check yourself
Two inputs share the prefix "[CLS] the bank". In which model are the output vectors at positions 0–2 guaranteed identical for both inputs?
Check yourself
Why does ELECTRA learn more per sequence than BERT's MLM at the same compute?
Remember
- ✓Encoder-only = transformer encoder with mask : tokens in, contextual vectors out, each depending on the whole input.
- ✓Pretraining: MLM on about 15% of tokens (80% [MASK] / 10% random / 10% kept), or ELECTRA's replaced-token detection on all tokens with .
- ✓Heads: [CLS] + tanh pooler for sequences, per token for NER, start/end vectors for QA, mean pooling + cosine for SBERT embeddings.
- ✓Cross-encoders compare pairs jointly; bi-encoders embed once and index. Retrieval pipelines use both, plus late interaction (ColBERT) on token vectors.
- ✓They do not generate: MLM gives conditionals, not a left-to-right factorisation, and every new token would change every earlier output, so nothing can be cached.