Definition
Contextual embedding is the vector a pretrained network assigns to one occurrence of a token, computed from the whole sentence around it. A static embedding (Word2Vec, GloVe, FastText) is a lookup table: one vector per word type. A contextual embedding is a function of the sentence , the position and the layer . The idea was popularised by CoVe (McCann, Bradbury, Xiong and Socher, "Learned in Translation: Contextualized Word Vectors", NeurIPS 2017) and ELMo (Peters, Neumann, Iyyer, Gardner, Clark, Lee and Zettlemoyer, "Deep contextualized word representations", NAACL 2018). It became the default with BERT (Devlin et al., NAACL 2019), whose hidden states are contextual embeddings at every layer.
The polysemy problem
A static model gives "bank" one vector, whether the sentence is about a loan or a river. Training pushes that vector towards the contexts of all its senses at once. Arora, Li, Liang, Ma and Risteski ("Linear Algebraic Structure of Word Senses, with Applications to Polysemy", TACL 2018) showed that the result is close to a weighted sum of sense vectors, weighted by how often each sense occurs. The rare sense gets drowned out, and any downstream model has to disambiguate from scratch. The same holds for weaker kinds of context dependence: "play" in a theatre, a game and a football match, or "light" as noun, verb and adjective.
Downstream models in 2016 already ran an LSTM over static vectors, which made the task representation contextual. The new idea was to pretrain the contextualiser on a large unlabeled or cheaply labeled corpus and transfer it. context2vec (Melamud, Goldberger and Dagan, CoNLL 2016) trained a biLSTM to predict a word from its sentence. TagLM (Peters et al., ACL 2017) added pretrained language-model states to a sequence tagger. CoVe and ELMo turned this into a general recipe.
CoVe: vectors from a translation encoder
- the two-layer bidirectional LSTM encoder of an attentional English→German translation model
- 300-d static vectors, the encoder's input
- what the downstream task model receives: static and contextual vectors concatenated
McCann et al. trained the encoder on English–German translation, with the largest version on the WMT 2017 data of about 7 million sentence pairs. Translation forces the encoder to resolve senses, because "bank" becomes "Bank" or "Ufer" in German. Adding CoVe improved sentiment, question classification, entailment and question answering over GloVe alone, and the gains grew with the size of the MT training set. The limitation was the supervision: parallel text is scarce compared with raw text.
ELMo: a deep biLM and a learned layer mix
- separate forward and backward two-layer LSTM language models
- shared token representation: a character CNN, so any string has an input vector
- shared softmax layer over the vocabulary
ELMo trains on raw text, so it can use far more data than CoVe. The forward LM reads left to right and the backward LM right to left. Neither sees the token it predicts. For token , a biLM with layers yields representations: the context-independent token layer and, at each LSTM layer, the concatenation . The two directions are concatenated, never mixed inside the network. This is "shallow" bidirectionality, which BERT later replaced with a masked objective.
- char-CNN token vector, duplicated to match the width of the LSTM layers
- softmax-normalised scalar weight of layer j, learned by the downstream task with the biLM frozen
- scalar scale for the whole vector; the paper notes it helps optimisation
The downstream model learns only numbers to choose its mixture. Different tasks choose differently. Peters et al. found that the first biLSTM layer is better for part-of-speech tagging and the second for word-sense disambiguation, and that using all layers beats using the top layer alone. ELMo vectors are concatenated with the task model's usual input, and the paper found it also helps to add them at the output of the task RNN for some tasks.
Configurations and results from the papers
- ELMo biLM
- char-CNN with 2048 filters and two highway layers, projected to 512; 2 biLSTM layers with 4096 units and 512-d projections, residual connection between layers; trained on the 1B Word Benchmark
- ELMo gains
- SQuAD F1 81.1 → 85.8; SNLI 88.0 → 88.7; SRL 81.4 → 84.6; coreference 67.2 → 70.4; NER F1 90.15 → 92.22; SST-5 51.4 → 54.7 (baseline → baseline + ELMo)
- CoVe
- 2-layer biLSTM encoder of an English→German attentional MT model, GloVe input; the largest version used WMT 2017 (about 7M sentence pairs)
- BERT-base
- 12 layers, , 12 heads, 110M parameters, WordPiece vocabulary of 30,000; hidden states of 13 layers (embeddings + 12) available as features
- BERT features
- CoNLL-2003 NER dev F1 with a frozen BERT-base: embeddings 91.0, last hidden 94.9, second-to-last 95.6, weighted sum of all 12 layers 95.5, weighted sum of last four 95.9, concatenation of last four 96.1; fine-tuning 96.4
- Anisotropy
- Ethayarajh (EMNLP 2019): in BERT, ELMo and GPT-2 the average cosine between random words is far above 0 in most layers; in GPT-2's last layer two random words have on average almost perfect cosine similarity
BERT-style contextual vectors
BERT replaced the two one-directional LMs with one deep bidirectional encoder trained by masked language modelling (see BERT, Encoder-only models). Every layer of every token now sees the whole sentence through self-attention. Its hidden states can be used in two ways. In feature extraction the network is frozen and the task model reads chosen layers, as with ELMo. In fine-tuning all weights are updated for the task. For most tasks fine-tuning wins, but frozen features are cheaper when many tasks share one encoder or when the representation itself is the object of study.
Two practical details matter. First, a word may be split into several WordPiece tokens ("embeddings" → em, ##bed, ##ding, ##s). The word's vector is then usually the first sub-token or the mean over its sub-tokens. Second, layer 0 is not a pure type vector. It already includes position and segment embeddings, so "bank" at position 2 and at position 6 differ slightly before any attention.
- the layer- vector of the token at position of sentence
- sentences that contain , at positions
- the word is effectively static at that layer; lower means more context-specific
Ethayarajh ("How Contextual are Contextualized Word Representations?", EMNLP 2019) measured this and two related quantities for ELMo, BERT and GPT-2. Self-similarity falls in the upper layers, so upper layers are more context-specific. On average, less than 5% of the variance of a word's contextual vectors is explained by their first principal component, so a single static vector is a poor summary. That static vector is still useful: the first principal component of a word's representations in BERT's lower layers beat GloVe and FastText on many word-similarity benchmarks.
Anisotropy: why raw cosines mislead
The same study found that contextual vectors are anisotropic: they occupy a narrow cone rather than spreading in all directions. If the average cosine between random tokens is 0.8, then a cosine of 0.85 between two occurrences of "bank" says almost nothing. Ethayarajh therefore reports self-similarity minus this anisotropy baseline. Gao et al. ("Representation Degeneration Problem in Training Natural Language Generation Models", ICLR 2019) traced part of the effect to likelihood training, which pushes the embeddings of rare tokens in one common direction. Later work found that a handful of "rogue" dimensions with huge magnitude dominate cosine similarity in many transformer models.
Simple post-processing helps. Subtracting the mean vector and projecting out the top principal directions ("All-but-the-Top", Mu and Viswanath, ICLR 2018) or fully whitening the vectors spreads them out again. BERT-flow (Li et al., EMNLP 2020) maps sentence vectors to a Gaussian with a normalising flow. Contrastive fine-tuning such as SimCSE (Gao, Yao and Chen, EMNLP 2021) makes the space more uniform by training. The figure below shows the cheapest fix, mean-centring, on a toy encoder.
Layer-wise behaviour is consistent across studies. Tenney, Das and Pavlick ("BERT Rediscovers the Classical NLP Pipeline", ACL 2019) probed BERT and found the pipeline in order of depth: part-of-speech information peaks early, then constituents and dependencies, then semantic roles and coreference higher up. The top layers of a pretrained model are the most specialised to its pretraining objective, which is one reason the second-to-last layer or a mix of upper layers often beats the last one as features.
Worked example: an ELMo layer mix
Take a frozen biLM with three layers and 3-d vectors for "bank" in two sentences. The token layer is the same in both, because it sees only the characters. A task has learned and .
| layer j | "cash at the bank" | "the river bank" | cos |
|---|---|---|---|
| 0 (char-CNN) | (0.4, 0.4, 0.0) | (0.4, 0.4, 0.0) | 1.000 |
| 1 (biLSTM 1) | (0.7, 0.1, 0.3) | (0.2, 0.6, 0.3) | 0.539 |
| 2 (biLSTM 2) | (0.9, −0.2, 0.5) | (−0.3, 0.9, 0.4) | −0.232 |
| ELMo mix | (0.5644, 0.0510, 0.2444) | (0.0669, 0.5239, 0.2199) | 0.335 |
Computing ELMo(bank)
- 1Layer weights: .
- 2Weighted layers for the financial sentence: , , .
- 3Sum , times : .
- 4The same mix for the river sentence gives . The cosine between the two is : between layer 1 (0.539) and layer 2 (−0.232), pulled up by the shared token layer.
- 5A task that needs the sense would raise ; one that needs the word's form (POS tagging, say) would raise or . That choice costs 4 parameters.
1import torch2import torch.nn.functional as F3from transformers import AutoTokenizer, AutoModel45tok = AutoTokenizer.from_pretrained("bert-base-uncased")6model = AutoModel.from_pretrained("bert-base-uncased", output_hidden_states=True).eval()78sentences = ["she deposited cash at the bank", "the bank approved the loan",9 "we sat on the river bank", "the bank of the river flooded"]10sense = torch.tensor([0, 0, 1, 1])1112@torch.no_grad()13def states(sentence):14 enc = tok(sentence, return_tensors="pt")15 hs = model(**enc).hidden_states # 13 tensors (embeddings + 12 layers), each (1, T, 768)16 return enc, torch.stack(hs)[:, 0] # (13, T, 768)1718bank, everything = [], []19for s in sentences:20 enc, h = states(s)21 pos = enc.word_ids(0).index(s.split().index("bank")) # first sub-token of the word22 bank.append(h[:, pos]) # (13, 768)23 everything.append(h[:, 1:-1]) # all tokens except [CLS]/[SEP]24bank = torch.stack(bank) # (4, 13, 768)25mean = torch.cat(everything, dim=1).mean(dim=1) # (13, 768): per-layer mean vector2627for layer in range(13):28 for centre in (False, True):29 v = bank[:, layer] - (mean[layer] if centre else 0)30 c = F.cosine_similarity(v[:, None], v[None, :], dim=-1) # (4, 4)31 iu = torch.triu_indices(4, 4, 1)32 pairs = c[iu[0], iu[1]]33 is_same = sense[iu[0]] == sense[iu[1]]34 print(f"layer {layer:2d} {'centred' if centre else 'raw '} "35 f"same {pairs[is_same].mean():.3f} cross {pairs[~is_same].mean():.3f}")3637class ScalarMix(torch.nn.Module):38 """ELMo's task-specific layer mix: gamma * sum_j softmax(w)_j * h_j."""39 def __init__(self, n_layers: int):40 super().__init__()41 self.w = torch.nn.Parameter(torch.zeros(n_layers))42 self.gamma = torch.nn.Parameter(torch.ones(()))43 def forward(self, hidden_states): # (n_layers, ..., d)44 s = torch.softmax(self.w, dim=0)45 return self.gamma * torch.einsum("l,l...->...", s, hidden_states)4647mix = ScalarMix(13)48features = mix(bank.transpose(0, 1)) # (4, 768): one trainable mix over all 13 layersUse when
- +Resolve polysemy: one vector per occurrence, not per type
- +Carry syntax and semantics learned from huge unlabeled corpora into small labeled tasks
- +Frozen features are cheap to share across many tasks; a scalar mix adds only L + 2 parameters
- +Character or subword inputs mean no word is out of vocabulary
Avoid when
- −A forward pass of a large network for every sentence, versus a table lookup
- −No single vector per word for lexicons or word-level search; you must pool over contexts
- −Anisotropic: raw cosines are inflated and need centring, whitening or contrastive training
- −Which layer to use depends on the task and the model and has to be chosen
Check yourself
In ELMo, what does a downstream task learn on top of the frozen biLM?
Check yourself
Two occurrences of "bank" in GPT-2's last layer have cosine 0.95. What should you check before concluding that they are used in the same sense?
Remember
- ✓A contextual embedding is , a hidden state of a pretrained network, so each occurrence of a word gets its own vector; static vectors average over senses.
- ✓ELMo: a char-CNN and two biLSTM layers trained as forward and backward LMs; tasks learn over the layers. CoVe used an MT encoder instead.
- ✓BERT's hidden states are contextual at every layer; frozen top-four-layer features reach 96.1 NER F1 against 96.4 for fine-tuning.
- ✓Lower layers carry form and syntax, upper layers context and semantics; self-similarity falls with depth.
- ✓Contextual vectors are anisotropic: compare cosines with the random-pair baseline, or centre and whiten first.