# Contextual embeddings

> A contextual embedding is a token's hidden state in a pretrained network, so "bank" gets a new vector in each sentence. CoVe (2017) used an MT encoder, ELMo (2018) a softmax-weighted mix of biLM layers, BERT any layer; raw vectors are anisotropic, so centre them before comparing.

Source: https://www.mlfoundry.in/learn/contextual-embeddings  
Section: Deep Learning › Large Language Models (LLMs)  
Level: core · 15 min read · updated 2026-10-13

**Contextual embedding:** is the vector a pretrained network assigns to one *occurrence* of a token, computed from the whole sentence around it. A static embedding (**Word2Vec**, **GloVe**, **FastText**) is a lookup table: one vector per word type. A contextual embedding is a function $f_\ell(s,i)$ of the sentence $s$, the position $i$ and the layer $\ell$. The idea was popularised by CoVe (McCann, Bradbury, Xiong and Socher, "Learned in Translation: Contextualized Word Vectors", NeurIPS 2017) and ELMo (Peters, Neumann, Iyyer, Gardner, Clark, Lee and Zettlemoyer, "Deep contextualized word representations", NAACL 2018). It became the default with BERT (Devlin et al., NAACL 2019), whose hidden states are contextual embeddings at every layer.

## The polysemy problem

A static model gives "bank" one vector, whether the sentence is about a loan or a river. Training pushes that vector towards the contexts of *all* its senses at once. Arora, Li, Liang, Ma and Risteski ("Linear Algebraic Structure of Word Senses, with Applications to Polysemy", TACL 2018) showed that the result is close to a weighted sum of sense vectors, weighted by how often each sense occurs. The rare sense gets drowned out, and any downstream model has to disambiguate from scratch. The same holds for weaker kinds of context dependence: "play" in a theatre, a game and a football match, or "light" as noun, verb and adjective.

Downstream models in 2016 already ran an LSTM over static vectors, which made the *task* representation contextual. The new idea was to **pretrain** the contextualiser on a large unlabeled or cheaply labeled corpus and transfer it. context2vec (Melamud, Goldberger and Dagan, CoNLL 2016) trained a biLSTM to predict a word from its sentence. TagLM (Peters et al., ACL 2017) added pretrained language-model states to a sequence tagger. CoVe and ELMo turned this into a general recipe.

## CoVe: vectors from a translation encoder

$$
\mathrm{CoVe}(w)=\text{MT-LSTM}\big(\mathrm{GloVe}(w)\big),\qquad \tilde w=\big[\mathrm{GloVe}(w);\ \mathrm{CoVe}(w)\big]
$$

- $\text{MT-LSTM}$: the two-layer bidirectional LSTM encoder of an attentional English→German translation model
- $\mathrm{GloVe}(w)$: 300-d static vectors, the encoder's input
- $\tilde w$: what the downstream task model receives: static and contextual vectors concatenated

McCann et al. trained the encoder on English–German translation, with the largest version on the WMT 2017 data of about 7 million sentence pairs. Translation forces the encoder to resolve senses, because "bank" becomes "Bank" or "Ufer" in German. Adding CoVe improved sentiment, question classification, entailment and question answering over GloVe alone, and the gains grew with the size of the MT training set. The limitation was the supervision: parallel text is scarce compared with raw text.

## ELMo: a deep biLM and a learned layer mix

$$
\sum_{k=1}^{N}\Big[\log p\big(t_k\mid t_1,\dots,t_{k-1};\Theta_x,\overrightarrow{\Theta}_{LSTM},\Theta_s\big)+\log p\big(t_k\mid t_{k+1},\dots,t_N;\Theta_x,\overleftarrow{\Theta}_{LSTM},\Theta_s\big)\Big]
$$

- $\overrightarrow{\Theta}_{LSTM},\overleftarrow{\Theta}_{LSTM}$: separate forward and backward two-layer LSTM language models
- $\Theta_x$: shared token representation: a character CNN, so any string has an input vector
- $\Theta_s$: shared softmax layer over the vocabulary

ELMo trains on raw text, so it can use far more data than CoVe. The forward LM reads left to right and the backward LM right to left. Neither sees the token it predicts. For token $k$, a biLM with $L=2$ layers yields $2L+1=3$ representations: the context-independent token layer $h_{k,0}$ and, at each LSTM layer, the concatenation $h_{k,j}=[\overrightarrow h_{k,j};\overleftarrow h_{k,j}]$. The two directions are concatenated, never mixed inside the network. This is "shallow" bidirectionality, which BERT later replaced with a masked objective.

$$
\mathrm{ELMo}_k^{task}=\gamma^{task}\sum_{j=0}^{L}s_j^{task}\,h_{k,j},\qquad s^{task}=\mathrm{softmax}(w^{task})
$$

- $h_{k,0}$: char-CNN token vector, duplicated to match the $2\times512$ width of the LSTM layers
- $s_j^{task}$: softmax-normalised scalar weight of layer j, learned by the downstream task with the biLM frozen
- $\gamma^{task}$: scalar scale for the whole vector; the paper notes it helps optimisation

The downstream model learns only $L+2=4$ numbers to choose its mixture. Different tasks choose differently. Peters et al. found that the first biLSTM layer is better for part-of-speech tagging and the second for word-sense disambiguation, and that using all layers beats using the top layer alone. ELMo vectors are concatenated with the task model's usual input, and the paper found it also helps to add them at the output of the task RNN for some tasks.

> Figure: Left: ELMo. Each token goes through a character CNN and two biLSTM layers, in which the forward and backward LSTMs run separately and are concatenated. A task combines the three layers of each token with softmax weights s₀, s₁, s₂ and a scale γ. Right: BERT-base used as a frozen feature extractor for CoNLL-2003 NER. Hover a row to see which layers it uses. Concatenating the top four layers (96.1 dev F1) comes within 0.3 of fine-tuning the whole model (96.4), while the embedding layer alone gives 91.0.

**Configurations and results from the papers**

- **ELMo biLM:** char-CNN with 2048 filters and two highway layers, projected to 512; 2 biLSTM layers with 4096 units and 512-d projections, residual connection between layers; trained on the 1B Word Benchmark
- **ELMo gains:** SQuAD F1 81.1 → 85.8; SNLI 88.0 → 88.7; SRL 81.4 → 84.6; coreference 67.2 → 70.4; NER F1 90.15 → 92.22; SST-5 51.4 → 54.7 (baseline → baseline + ELMo)
- **CoVe:** 2-layer biLSTM encoder of an English→German attentional MT model, GloVe input; the largest version used WMT 2017 (about 7M sentence pairs)
- **BERT-base:** 12 layers, $d=768$, 12 heads, 110M parameters, WordPiece vocabulary of 30,000; hidden states of 13 layers (embeddings + 12) available as features
- **BERT features:** CoNLL-2003 NER dev F1 with a frozen BERT-base: embeddings 91.0, last hidden 94.9, second-to-last 95.6, weighted sum of all 12 layers 95.5, weighted sum of last four 95.9, concatenation of last four 96.1; fine-tuning 96.4
- **Anisotropy:** Ethayarajh (EMNLP 2019): in BERT, ELMo and GPT-2 the average cosine between random words is far above 0 in most layers; in GPT-2's last layer two random words have on average almost perfect cosine similarity

## BERT-style contextual vectors

BERT replaced the two one-directional LMs with one deep bidirectional encoder trained by masked language modelling (see **BERT**, **Encoder-only models**). Every layer of every token now sees the whole sentence through self-attention. Its hidden states can be used in two ways. In **feature extraction** the network is frozen and the task model reads chosen layers, as with ELMo. In **fine-tuning** all weights are updated for the task. For most tasks fine-tuning wins, but frozen features are cheaper when many tasks share one encoder or when the representation itself is the object of study.

Two practical details matter. First, a word may be split into several WordPiece tokens ("embeddings" → `em`, `##bed`, `##ding`, `##s`). The word's vector is then usually the first sub-token or the mean over its sub-tokens. Second, layer 0 is not a pure type vector. It already includes position and segment embeddings, so "bank" at position 2 and at position 6 differ slightly before any attention.

$$
\mathrm{SelfSim}_\ell(w)=\frac{1}{n^2-n}\sum_{j}\sum_{k\ne j}\cos\big(f_\ell(s_j,i_j),\,f_\ell(s_k,i_k)\big)
$$

- $f_\ell(s,i)$: the layer-$\ell$ vector of the token at position $i$ of sentence $s$
- $s_1,\dots,s_n$: $n$ sentences that contain $w$, at positions $i_1,\dots,i_n$
- $\mathrm{SelfSim}=1$: the word is effectively static at that layer; lower means more context-specific

Ethayarajh ("How Contextual are Contextualized Word Representations?", EMNLP 2019) measured this and two related quantities for ELMo, BERT and GPT-2. Self-similarity falls in the upper layers, so upper layers are more context-specific. On average, less than 5% of the variance of a word's contextual vectors is explained by their first principal component, so a single static vector is a poor summary. That static vector is still useful: the first principal component of a word's representations in BERT's lower layers beat GloVe and FastText on many word-similarity benchmarks.

## Anisotropy: why raw cosines mislead

The same study found that contextual vectors are **anisotropic**: they occupy a narrow cone rather than spreading in all directions. If the average cosine between *random* tokens is 0.8, then a cosine of 0.85 between two occurrences of "bank" says almost nothing. Ethayarajh therefore reports self-similarity minus this anisotropy baseline. Gao et al. ("Representation Degeneration Problem in Training Natural Language Generation Models", ICLR 2019) traced part of the effect to likelihood training, which pushes the embeddings of rare tokens in one common direction. Later work found that a handful of "rogue" dimensions with huge magnitude dominate cosine similarity in many transformer models.

Simple post-processing helps. Subtracting the mean vector and projecting out the top principal directions ("All-but-the-Top", Mu and Viswanath, ICLR 2018) or fully whitening the vectors spreads them out again. BERT-flow (Li et al., EMNLP 2020) maps sentence vectors to a Gaussian with a normalising flow. Contrastive fine-tuning such as SimCSE (Gao, Yao and Chen, EMNLP 2021) makes the space more uniform by training. The figure below shows the cheapest fix, mean-centring, on a toy encoder.

Layer-wise behaviour is consistent across studies. Tenney, Das and Pavlick ("BERT Rediscovers the Classical NLP Pipeline", ACL 2019) probed BERT and found the pipeline in order of depth: part-of-speech information peaks early, then constituents and dependencies, then semantic roles and coreference higher up. The top layers of a pretrained model are the most specialised to its pretraining objective, which is one reason the second-to-last layer or a mix of upper layers often beats the last one as features.

> Figure: A toy contextual encoder with no learned weights: each layer adds λ times a dot-product self-attention average to every token (temperature τ) and renormalises. "bank" starts exactly between a finance axis and a river axis, and appears in three financial (F1–F3) and three river (R1–R3) sentences. At layer 0 all six vectors are identical: every cosine is 1 and random pairs of different words average 0.184. Building layer 1, "bank" in F1 puts 0.229 of its attention on "cash" and 0.141 on "deposited", and in R1 it puts 0.248 on "river". At the defaults (layer 6, λ = 1, τ = 0.5) the raw cosines give same-sense 0.915 and cross-sense 0.770, a gap of only 0.146. But random pairs of different words now average 0.835: repeated averaging has squeezed every token into a narrow cone, and the self-similarity of "bank" (0.828) is below that baseline. Switch to mean-centred and the senses split cleanly: same-sense 0.446, cross-sense −0.524, gap 0.970, random pairs −0.028. At layer 3 the centred values are 0.866 and 0.402. Raise τ to 1 and attention at layer 6 is almost uniform, so the vectors lose word identity too. λ = 0 keeps every layer static.

## Worked example: an ELMo layer mix

Take a frozen biLM with three layers and 3-d vectors for "bank" in two sentences. The token layer $h_0$ is the same in both, because it sees only the characters. A task has learned $w=(0,\ 1,\ 0.5)$ and $\gamma=0.8$.

Layer vectors for "bank" (made-up values; mix computed in Python)

| layer j | "cash at the bank" | "the river bank" | cos |
| --- | --- | --- | --- |
| 0 (char-CNN) | (0.4, 0.4, 0.0) | (0.4, 0.4, 0.0) | 1.000 |
| 1 (biLSTM 1) | (0.7, 0.1, 0.3) | (0.2, 0.6, 0.3) | 0.539 |
| 2 (biLSTM 2) | (0.9, −0.2, 0.5) | (−0.3, 0.9, 0.4) | −0.232 |
| ELMo mix | (0.5644, 0.0510, 0.2444) | (0.0669, 0.5239, 0.2199) | 0.335 |

**Computing ELMo(bank)**

1. Layer weights: $s=\mathrm{softmax}(0,1,0.5)=(1,\,2.718,\,1.649)/5.367=(0.1863,\ 0.5065,\ 0.3072)$.
2. Weighted layers for the financial sentence: $0.1863\,h_0=(0.0745,0.0745,0)$, $0.5065\,h_1=(0.3545,0.0506,0.1519)$, $0.3072\,h_2=(0.2765,-0.0614,0.1536)$.
3. Sum $(0.7055,\ 0.0637,\ 0.3055)$, times $\gamma=0.8$: $\mathrm{ELMo}=(0.5644,\ 0.0510,\ 0.2444)$.
4. The same mix for the river sentence gives $(0.0669,\ 0.5239,\ 0.2199)$. The cosine between the two is $0.335$: between layer 1 (0.539) and layer 2 (−0.232), pulled up by the shared token layer.
5. A task that needs the sense would raise $w_2$; one that needs the word's form (POS tagging, say) would raise $w_0$ or $w_1$. That choice costs 4 parameters.

```python
import torch
import torch.nn.functional as F
from transformers import AutoTokenizer, AutoModel

tok = AutoTokenizer.from_pretrained("bert-base-uncased")
model = AutoModel.from_pretrained("bert-base-uncased", output_hidden_states=True).eval()

sentences = ["she deposited cash at the bank", "the bank approved the loan",
             "we sat on the river bank", "the bank of the river flooded"]
sense = torch.tensor([0, 0, 1, 1])

@torch.no_grad()
def states(sentence):
    enc = tok(sentence, return_tensors="pt")
    hs = model(**enc).hidden_states                 # 13 tensors (embeddings + 12 layers), each (1, T, 768)
    return enc, torch.stack(hs)[:, 0]              # (13, T, 768)

bank, everything = [], []
for s in sentences:
    enc, h = states(s)
    pos = enc.word_ids(0).index(s.split().index("bank"))   # first sub-token of the word
    bank.append(h[:, pos])                                  # (13, 768)
    everything.append(h[:, 1:-1])                           # all tokens except [CLS]/[SEP]
bank = torch.stack(bank)                                    # (4, 13, 768)
mean = torch.cat(everything, dim=1).mean(dim=1)             # (13, 768): per-layer mean vector

for layer in range(13):
    for centre in (False, True):
        v = bank[:, layer] - (mean[layer] if centre else 0)
        c = F.cosine_similarity(v[:, None], v[None, :], dim=-1)          # (4, 4)
        iu = torch.triu_indices(4, 4, 1)
        pairs = c[iu[0], iu[1]]
        is_same = sense[iu[0]] == sense[iu[1]]
        print(f"layer {layer:2d} {'centred' if centre else 'raw    '} "
              f"same {pairs[is_same].mean():.3f}  cross {pairs[~is_same].mean():.3f}")

class ScalarMix(torch.nn.Module):
    """ELMo's task-specific layer mix: gamma * sum_j softmax(w)_j * h_j."""
    def __init__(self, n_layers: int):
        super().__init__()
        self.w = torch.nn.Parameter(torch.zeros(n_layers))
        self.gamma = torch.nn.Parameter(torch.ones(()))
    def forward(self, hidden_states):              # (n_layers, ..., d)
        s = torch.softmax(self.w, dim=0)
        return self.gamma * torch.einsum("l,l...->...", s, hidden_states)

mix = ScalarMix(13)
features = mix(bank.transpose(0, 1))              # (4, 768): one trainable mix over all 13 layers
```

Layer-wise contextual vectors for "bank" from BERT with Hugging Face transformers, with mean-centring and an ELMo-style scalar mix (not run here: written for torch 2 and transformers 4; no outputs are claimed)

> **Raw cosine similarity is inflated:** Because contextual vectors share a few dominant directions, almost any two tokens have a high cosine in upper layers. A cosine of 0.9 between two uses of a word can be *below* the random-pair baseline, as in the figure at layer 6 (self-similarity 0.828 against a baseline of 0.835). Before comparing vectors, subtract the mean, whiten, or at least report the baseline next to the number.

> **Context-specific does not mean sense-specific:** Upper layers make "bank" depend on its context, but also on everything else in the sentence: topic, position, neighbouring words. In the figure, the centred same-sense cosine falls from 0.866 at layer 3 to 0.446 at layer 6 while the senses still separate. The vectors become sentence-specific as well as sense-specific. Word-sense tasks therefore often do best on middle or upper-middle layers, not the last.

> **Contextual embeddings in today's LLMs:** In a decoder-only LLM every hidden state is contextual but sees only the left context (see **Decoder-only models**). Embedding models used for retrieval are usually encoders or LLMs fine-tuned with a contrastive objective, which pools the token vectors into one sentence vector and fixes much of the anisotropy. The token-level ideas here (layer choice, sub-token pooling, centring) still apply when you probe or reuse hidden states.

**Strengths**

- Resolve polysemy: one vector per occurrence, not per type
- Carry syntax and semantics learned from huge unlabeled corpora into small labeled tasks
- Frozen features are cheap to share across many tasks; a scalar mix adds only L + 2 parameters
- Character or subword inputs mean no word is out of vocabulary

**Limitations**

- A forward pass of a large network for every sentence, versus a table lookup
- No single vector per word for lexicons or word-level search; you must pool over contexts
- Anisotropic: raw cosines are inflated and need centring, whitening or contrastive training
- Which layer to use depends on the task and the model and has to be chosen

**Question:** In ELMo, what does a downstream task learn on top of the frozen biLM?

- A. A new character CNN
- B. Softmax-normalised scalar weights over the L + 1 layers and one scale γ
- C. A full attention layer over the biLM states
- D. Only which layer to pick, as a hard choice

**Answer:** B. Softmax-normalised scalar weights over the L + 1 layers and one scale γ — $\mathrm{ELMo}_k^{task}=\gamma\sum_j s_j h_{k,j}$ with $s=\mathrm{softmax}(w)$. For a two-layer biLM that is $L+2=4$ parameters, and different tasks learn different mixtures: lower layers for syntax, upper layers for word sense.

**Question:** Two occurrences of "bank" in GPT-2's last layer have cosine 0.95. What should you check before concluding that they are used in the same sense?

- A. Nothing: 0.95 is very high
- B. The cosine between random pairs of tokens in that layer, because the representations are anisotropic
- C. Whether the sentences have the same length
- D. The cosine at the embedding layer

**Answer:** B. The cosine between random pairs of tokens in that layer, because the representations are anisotropic — Ethayarajh (2019) found that random words in GPT-2's last layer have on average almost perfect cosine similarity. A high cosine is only informative relative to that baseline, or after mean-centring or whitening.

**Key points**

- A contextual embedding is $f_\ell(s,i)$, a hidden state of a pretrained network, so each occurrence of a word gets its own vector; static vectors average over senses.
- ELMo: a char-CNN and two biLSTM layers trained as forward and backward LMs; tasks learn $\gamma\sum_j\mathrm{softmax}(w)_j h_{k,j}$ over the layers. CoVe used an MT encoder instead.
- BERT's hidden states are contextual at every layer; frozen top-four-layer features reach 96.1 NER F1 against 96.4 for fine-tuning.
- Lower layers carry form and syntax, upper layers context and semantics; self-similarity falls with depth.
- Contextual vectors are anisotropic: compare cosines with the random-pair baseline, or centre and whiten first.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
