Contextual embeddings

A contextual embedding is a token's hidden state in a pretrained network, so "bank" gets a new vector in each sentence. CoVe (2017) used an MT encoder, ELMo (2018) a softmax-weighted mix of biLM layers, BERT any layer; raw vectors are anisotropic, so centre them before comparing.

15 min read1450 wordscore0 viewsupdated 2026-10-13

Definition

Contextual embedding is the vector a pretrained network assigns to one occurrence of a token, computed from the whole sentence around it. A static embedding (Word2Vec, GloVe, FastText) is a lookup table: one vector per word type. A contextual embedding is a function fℓ(s,i)f_\ell(s,i) of the sentence ss, the position ii and the layer ℓ\ell. The idea was popularised by CoVe (McCann, Bradbury, Xiong and Socher, "Learned in Translation: Contextualized Word Vectors", NeurIPS 2017) and ELMo (Peters, Neumann, Iyyer, Gardner, Clark, Lee and Zettlemoyer, "Deep contextualized word representations", NAACL 2018). It became the default with BERT (Devlin et al., NAACL 2019), whose hidden states are contextual embeddings at every layer.

The polysemy problem

A static model gives "bank" one vector, whether the sentence is about a loan or a river. Training pushes that vector towards the contexts of all its senses at once. Arora, Li, Liang, Ma and Risteski ("Linear Algebraic Structure of Word Senses, with Applications to Polysemy", TACL 2018) showed that the result is close to a weighted sum of sense vectors, weighted by how often each sense occurs. The rare sense gets drowned out, and any downstream model has to disambiguate from scratch. The same holds for weaker kinds of context dependence: "play" in a theatre, a game and a football match, or "light" as noun, verb and adjective.

Downstream models in 2016 already ran an LSTM over static vectors, which made the task representation contextual. The new idea was to pretrain the contextualiser on a large unlabeled or cheaply labeled corpus and transfer it. context2vec (Melamud, Goldberger and Dagan, CoNLL 2016) trained a biLSTM to predict a word from its sentence. TagLM (Peters et al., ACL 2017) added pretrained language-model states to a sequence tagger. CoVe and ELMo turned this into a general recipe.

CoVe: vectors from a translation encoder

CoVe(w)=MT-LSTM(GloVe(w)),w~=[GloVe(w); CoVe(w)]\mathrm{CoVe}(w)=\text{MT-LSTM}\big(\mathrm{GloVe}(w)\big),\qquad \tilde w=\big[\mathrm{GloVe}(w);\ \mathrm{CoVe}(w)\big]
MT-LSTM\text{MT-LSTM}
the two-layer bidirectional LSTM encoder of an attentional English→German translation model
GloVe(w)\mathrm{GloVe}(w)
300-d static vectors, the encoder's input
w~\tilde w
what the downstream task model receives: static and contextual vectors concatenated

McCann et al. trained the encoder on English–German translation, with the largest version on the WMT 2017 data of about 7 million sentence pairs. Translation forces the encoder to resolve senses, because "bank" becomes "Bank" or "Ufer" in German. Adding CoVe improved sentiment, question classification, entailment and question answering over GloVe alone, and the gains grew with the size of the MT training set. The limitation was the supervision: parallel text is scarce compared with raw text.

ELMo: a deep biLM and a learned layer mix

∑k=1N[log⁡p(tk∣t1,…,tk−1;Θx,Θ→LSTM,Θs)+log⁡p(tk∣tk+1,…,tN;Θx,Θ←LSTM,Θs)]\sum_{k=1}^{N}\Big[\log p\big(t_k\mid t_1,\dots,t_{k-1};\Theta_x,\overrightarrow{\Theta}_{LSTM},\Theta_s\big)+\log p\big(t_k\mid t_{k+1},\dots,t_N;\Theta_x,\overleftarrow{\Theta}_{LSTM},\Theta_s\big)\Big]
Θ→LSTM,Θ←LSTM\overrightarrow{\Theta}_{LSTM},\overleftarrow{\Theta}_{LSTM}
separate forward and backward two-layer LSTM language models
Θx\Theta_x
shared token representation: a character CNN, so any string has an input vector
Θs\Theta_s
shared softmax layer over the vocabulary

ELMo trains on raw text, so it can use far more data than CoVe. The forward LM reads left to right and the backward LM right to left. Neither sees the token it predicts. For token kk, a biLM with L=2L=2 layers yields 2L+1=32L+1=3 representations: the context-independent token layer hk,0h_{k,0} and, at each LSTM layer, the concatenation hk,j=[h→k,j;h←k,j]h_{k,j}=[\overrightarrow h_{k,j};\overleftarrow h_{k,j}]. The two directions are concatenated, never mixed inside the network. This is "shallow" bidirectionality, which BERT later replaced with a masked objective.

ELMoktask=γtask∑j=0Lsjtask hk,j,stask=softmax(wtask)\mathrm{ELMo}_k^{task}=\gamma^{task}\sum_{j=0}^{L}s_j^{task}\,h_{k,j},\qquad s^{task}=\mathrm{softmax}(w^{task})
hk,0h_{k,0}
char-CNN token vector, duplicated to match the 2×5122\times512 width of the LSTM layers
sjtasks_j^{task}
softmax-normalised scalar weight of layer j, learned by the downstream task with the biLM frozen
γtask\gamma^{task}
scalar scale for the whole vector; the paper notes it helps optimisation

The downstream model learns only L+2=4L+2=4 numbers to choose its mixture. Different tasks choose differently. Peters et al. found that the first biLSTM layer is better for part-of-speech tagging and the second for word-sense disambiguation, and that using all layers beats using the top layer alone. ELMo vectors are concatenated with the task model's usual input, and the paper found it also helps to add them at the output of the task RNN for some tasks.

ELMo (Peters et al. 2018)ELMo(bank) = γ·(s₀h₀ + s₁h₁ + s₂h₂)s = softmax(w), w and γ learned per task→ fwd← bwd→ fwd← bwd→ fwd← bwd→ fwd← bwd→ fwd← bwd→ fwd← bwdchar-CNNchar-CNNchar-CNNtheriverbanks₀ h₀ token layers₁ h₁ biLSTM 1s₂ h₂ biLSTM 22 layers × 4096 LSTM units, 512-d projections,residual link; char-CNN 2048 filters → 512;forward and backward LMs on 1B Word BenchmarkBERT-base as frozen featuresNER dev F1, CoNLL-03 (BERT paper)emb12345678910111212 layers · d = 768 · 12 heads · 110Membeddings only91.0last hidden94.9second-to-last95.6sum all 1295.5sum last four95.9concat last four96.1fine-tuned96.4bar: F1 − 90; hover a rowfrozen top-4 concat: 96.1,0.3 F1 below fine-tuning
fig.Left: ELMo. Each token goes through a character CNN and two biLSTM layers, in which the forward and backward LSTMs run separately and are concatenated. A task combines the three layers of each token with softmax weights s₀, s₁, s₂ and a scale γ. Right: BERT-base used as a frozen feature extractor for CoNLL-2003 NER. Hover a row to see which layers it uses. Concatenating the top four layers (96.1 dev F1) comes within 0.3 of fine-tuning the whole model (96.4), while the embedding layer alone gives 91.0.

Configurations and results from the papers

ELMo biLM
char-CNN with 2048 filters and two highway layers, projected to 512; 2 biLSTM layers with 4096 units and 512-d projections, residual connection between layers; trained on the 1B Word Benchmark
ELMo gains
SQuAD F1 81.1 → 85.8; SNLI 88.0 → 88.7; SRL 81.4 → 84.6; coreference 67.2 → 70.4; NER F1 90.15 → 92.22; SST-5 51.4 → 54.7 (baseline → baseline + ELMo)
CoVe
2-layer biLSTM encoder of an English→German attentional MT model, GloVe input; the largest version used WMT 2017 (about 7M sentence pairs)
BERT-base
12 layers, d=768d=768, 12 heads, 110M parameters, WordPiece vocabulary of 30,000; hidden states of 13 layers (embeddings + 12) available as features
BERT features
CoNLL-2003 NER dev F1 with a frozen BERT-base: embeddings 91.0, last hidden 94.9, second-to-last 95.6, weighted sum of all 12 layers 95.5, weighted sum of last four 95.9, concatenation of last four 96.1; fine-tuning 96.4
Anisotropy
Ethayarajh (EMNLP 2019): in BERT, ELMo and GPT-2 the average cosine between random words is far above 0 in most layers; in GPT-2's last layer two random words have on average almost perfect cosine similarity

BERT-style contextual vectors

BERT replaced the two one-directional LMs with one deep bidirectional encoder trained by masked language modelling (see BERT, Encoder-only models). Every layer of every token now sees the whole sentence through self-attention. Its hidden states can be used in two ways. In feature extraction the network is frozen and the task model reads chosen layers, as with ELMo. In fine-tuning all weights are updated for the task. For most tasks fine-tuning wins, but frozen features are cheaper when many tasks share one encoder or when the representation itself is the object of study.

Two practical details matter. First, a word may be split into several WordPiece tokens ("embeddings" → em, ##bed, ##ding, ##s). The word's vector is then usually the first sub-token or the mean over its sub-tokens. Second, layer 0 is not a pure type vector. It already includes position and segment embeddings, so "bank" at position 2 and at position 6 differ slightly before any attention.

SelfSimℓ(w)=1n2−n∑j∑k≠jcos⁡(fℓ(sj,ij), fℓ(sk,ik))\mathrm{SelfSim}_\ell(w)=\frac{1}{n^2-n}\sum_{j}\sum_{k\ne j}\cos\big(f_\ell(s_j,i_j),\,f_\ell(s_k,i_k)\big)
fℓ(s,i)f_\ell(s,i)
the layer-ℓ\ell vector of the token at position ii of sentence ss
s1,…,sns_1,\dots,s_n
nn sentences that contain ww, at positions i1,…,ini_1,\dots,i_n
SelfSim=1\mathrm{SelfSim}=1
the word is effectively static at that layer; lower means more context-specific

Ethayarajh ("How Contextual are Contextualized Word Representations?", EMNLP 2019) measured this and two related quantities for ELMo, BERT and GPT-2. Self-similarity falls in the upper layers, so upper layers are more context-specific. On average, less than 5% of the variance of a word's contextual vectors is explained by their first principal component, so a single static vector is a poor summary. That static vector is still useful: the first principal component of a word's representations in BERT's lower layers beat GloVe and FastText on many word-similarity benchmarks.

Anisotropy: why raw cosines mislead

The same study found that contextual vectors are anisotropic: they occupy a narrow cone rather than spreading in all directions. If the average cosine between random tokens is 0.8, then a cosine of 0.85 between two occurrences of "bank" says almost nothing. Ethayarajh therefore reports self-similarity minus this anisotropy baseline. Gao et al. ("Representation Degeneration Problem in Training Natural Language Generation Models", ICLR 2019) traced part of the effect to likelihood training, which pushes the embeddings of rare tokens in one common direction. Later work found that a handful of "rogue" dimensions with huge magnitude dominate cosine similarity in many transformer models.

Simple post-processing helps. Subtracting the mean vector and projecting out the top principal directions ("All-but-the-Top", Mu and Viswanath, ICLR 2018) or fully whitening the vectors spreads them out again. BERT-flow (Li et al., EMNLP 2020) maps sentence vectors to a Gaussian with a normalising flow. Contrastive fine-tuning such as SimCSE (Gao, Yao and Chen, EMNLP 2021) makes the space more uniform by training. The figure below shows the cheapest fix, mean-centring, on a toy encoder.

Layer-wise behaviour is consistent across studies. Tenney, Das and Pavlick ("BERT Rediscovers the Classical NLP Pipeline", ACL 2019) probed BERT and found the pipeline in order of depth: part-of-speech information peaks early, then constituents and dependencies, then semantic roles and coreference higher up. The top layers of a pretrained model are the most specialised to its pretraining objective, which is one reason the second-to-last layer or a mix of upper layers often beats the last one as features.

H ← normalise(H + λ·softmax(HHᵀ/τ)·H), no learned weights · layer 0 = static vectorscos(bank_i, bank_j), layer 6F1F2F3R1R2R3F1 cash1.87.96.79.70.73F2 loan.871.89.76.80.81F3 account.96.891.81.75.78R1 river.79.76.811.93.94R2 flooded.70.80.75.931.90R3 muddy.73.81.78.94.901mean cosine by layer-1-0.500.510123456same sensecross senserandom pairslayer 6: same sense 0.915 · cross sense 0.770 · gap 0.146 · random pairs 0.835self-similarity of bank (all 15 pairs) 0.828 · minus random-pair baseline -0.007attention from "bank" building layer 6 (F1 cash)0.15she0.18deposited0.18cash0.15at0.16the0.18bankclick a heatmap row to pick the sentence
vectors
fig.A toy contextual encoder with no learned weights: each layer adds λ times a dot-product self-attention average to every token (temperature τ) and renormalises. "bank" starts exactly between a finance axis and a river axis, and appears in three financial (F1–F3) and three river (R1–R3) sentences. At layer 0 all six vectors are identical: every cosine is 1 and random pairs of different words average 0.184. Building layer 1, "bank" in F1 puts 0.229 of its attention on "cash" and 0.141 on "deposited", and in R1 it puts 0.248 on "river". At the defaults (layer 6, λ = 1, τ = 0.5) the raw cosines give same-sense 0.915 and cross-sense 0.770, a gap of only 0.146. But random pairs of different words now average 0.835: repeated averaging has squeezed every token into a narrow cone, and the self-similarity of "bank" (0.828) is below that baseline. Switch to mean-centred and the senses split cleanly: same-sense 0.446, cross-sense −0.524, gap 0.970, random pairs −0.028. At layer 3 the centred values are 0.866 and 0.402. Raise τ to 1 and attention at layer 6 is almost uniform, so the vectors lose word identity too. λ = 0 keeps every layer static.

Worked example: an ELMo layer mix

Take a frozen biLM with three layers and 3-d vectors for "bank" in two sentences. The token layer h0h_0 is the same in both, because it sees only the characters. A task has learned w=(0, 1, 0.5)w=(0,\ 1,\ 0.5) and γ=0.8\gamma=0.8.

layer j"cash at the bank""the river bank"cos
0 (char-CNN)(0.4, 0.4, 0.0)(0.4, 0.4, 0.0)1.000
1 (biLSTM 1)(0.7, 0.1, 0.3)(0.2, 0.6, 0.3)0.539
2 (biLSTM 2)(0.9, −0.2, 0.5)(−0.3, 0.9, 0.4)−0.232
ELMo mix(0.5644, 0.0510, 0.2444)(0.0669, 0.5239, 0.2199)0.335
Layer vectors for "bank" (made-up values; mix computed in Python)

Computing ELMo(bank)

  1. 1Layer weights: s=softmax(0,1,0.5)=(1, 2.718, 1.649)/5.367=(0.1863, 0.5065, 0.3072)s=\mathrm{softmax}(0,1,0.5)=(1,\,2.718,\,1.649)/5.367=(0.1863,\ 0.5065,\ 0.3072).
  2. 2Weighted layers for the financial sentence: 0.1863 h0=(0.0745,0.0745,0)0.1863\,h_0=(0.0745,0.0745,0), 0.5065 h1=(0.3545,0.0506,0.1519)0.5065\,h_1=(0.3545,0.0506,0.1519), 0.3072 h2=(0.2765,−0.0614,0.1536)0.3072\,h_2=(0.2765,-0.0614,0.1536).
  3. 3Sum (0.7055, 0.0637, 0.3055)(0.7055,\ 0.0637,\ 0.3055), times γ=0.8\gamma=0.8: ELMo=(0.5644, 0.0510, 0.2444)\mathrm{ELMo}=(0.5644,\ 0.0510,\ 0.2444).
  4. 4The same mix for the river sentence gives (0.0669, 0.5239, 0.2199)(0.0669,\ 0.5239,\ 0.2199). The cosine between the two is 0.3350.335: between layer 1 (0.539) and layer 2 (−0.232), pulled up by the shared token layer.
  5. 5A task that needs the sense would raise w2w_2; one that needs the word's form (POS tagging, say) would raise w0w_0 or w1w_1. That choice costs 4 parameters.
python
1import torch
2import torch.nn.functional as F
3from transformers import AutoTokenizer, AutoModel
4
5tok = AutoTokenizer.from_pretrained("bert-base-uncased")
6model = AutoModel.from_pretrained("bert-base-uncased", output_hidden_states=True).eval()
7
8sentences = ["she deposited cash at the bank", "the bank approved the loan",
9 "we sat on the river bank", "the bank of the river flooded"]
10sense = torch.tensor([0, 0, 1, 1])
11
12@torch.no_grad()
13def states(sentence):
14 enc = tok(sentence, return_tensors="pt")
15 hs = model(**enc).hidden_states # 13 tensors (embeddings + 12 layers), each (1, T, 768)
16 return enc, torch.stack(hs)[:, 0] # (13, T, 768)
17
18bank, everything = [], []
19for s in sentences:
20 enc, h = states(s)
21 pos = enc.word_ids(0).index(s.split().index("bank")) # first sub-token of the word
22 bank.append(h[:, pos]) # (13, 768)
23 everything.append(h[:, 1:-1]) # all tokens except [CLS]/[SEP]
24bank = torch.stack(bank) # (4, 13, 768)
25mean = torch.cat(everything, dim=1).mean(dim=1) # (13, 768): per-layer mean vector
26
27for layer in range(13):
28 for centre in (False, True):
29 v = bank[:, layer] - (mean[layer] if centre else 0)
30 c = F.cosine_similarity(v[:, None], v[None, :], dim=-1) # (4, 4)
31 iu = torch.triu_indices(4, 4, 1)
32 pairs = c[iu[0], iu[1]]
33 is_same = sense[iu[0]] == sense[iu[1]]
34 print(f"layer {layer:2d} {'centred' if centre else 'raw '} "
35 f"same {pairs[is_same].mean():.3f} cross {pairs[~is_same].mean():.3f}")
36
37class ScalarMix(torch.nn.Module):
38 """ELMo's task-specific layer mix: gamma * sum_j softmax(w)_j * h_j."""
39 def __init__(self, n_layers: int):
40 super().__init__()
41 self.w = torch.nn.Parameter(torch.zeros(n_layers))
42 self.gamma = torch.nn.Parameter(torch.ones(()))
43 def forward(self, hidden_states): # (n_layers, ..., d)
44 s = torch.softmax(self.w, dim=0)
45 return self.gamma * torch.einsum("l,l...->...", s, hidden_states)
46
47mix = ScalarMix(13)
48features = mix(bank.transpose(0, 1)) # (4, 768): one trainable mix over all 13 layers
Layer-wise contextual vectors for "bank" from BERT with Hugging Face transformers, with mean-centring and an ELMo-style scalar mix (not run here: written for torch 2 and transformers 4; no outputs are claimed)

Use when

  • +Resolve polysemy: one vector per occurrence, not per type
  • +Carry syntax and semantics learned from huge unlabeled corpora into small labeled tasks
  • +Frozen features are cheap to share across many tasks; a scalar mix adds only L + 2 parameters
  • +Character or subword inputs mean no word is out of vocabulary

Avoid when

  • −A forward pass of a large network for every sentence, versus a table lookup
  • −No single vector per word for lexicons or word-level search; you must pool over contexts
  • −Anisotropic: raw cosines are inflated and need centring, whitening or contrastive training
  • −Which layer to use depends on the task and the model and has to be chosen

Check yourself

In ELMo, what does a downstream task learn on top of the frozen biLM?

Check yourself

Two occurrences of "bank" in GPT-2's last layer have cosine 0.95. What should you check before concluding that they are used in the same sense?

Remember

  • ✓A contextual embedding is fℓ(s,i)f_\ell(s,i), a hidden state of a pretrained network, so each occurrence of a word gets its own vector; static vectors average over senses.
  • ✓ELMo: a char-CNN and two biLSTM layers trained as forward and backward LMs; tasks learn γ∑jsoftmax(w)jhk,j\gamma\sum_j\mathrm{softmax}(w)_j h_{k,j} over the layers. CoVe used an MT encoder instead.
  • ✓BERT's hidden states are contextual at every layer; frozen top-four-layer features reach 96.1 NER F1 against 96.4 for fine-tuning.
  • ✓Lower layers carry form and syntax, upper layers context and semantics; self-similarity falls with depth.
  • ✓Contextual vectors are anisotropic: compare cosines with the random-pair baseline, or centre and whiten first.
← back to Deep Learning roadmap