Word embeddings

A word embedding maps each word to a dense vector of d ≈ 50–300 numbers so that words used in similar contexts lie close in cosine similarity. Count methods factor a PPMI co-occurrence matrix with truncated SVD; Word2Vec and GloVe learn much the same geometry by optimisation.

15 min read1298 wordscore0 viewsupdated 2026-10-13

Definition

Word embedding is a map w↦ew∈Rdw\mapsto e_w\in\mathbb R^d from a vocabulary of VV words to dense vectors, with dd usually 50–300, far smaller than VV. It is learned from unlabelled text so that geometry reflects usage: words that occur in similar contexts get vectors with high cosine similarity. The idea rests on the distributional hypothesis. Harris ("Distributional Structure", 1954) argued that differences in meaning go with differences in distribution, and Firth (1957) put it as "you shall know a word by the company it keeps". In practice the embeddings are the rows of a V×dV\times d matrix EE, and looking a word up is multiplying a one-hot vector by E⊤E^\top.

A model that reads words needs to turn each one into numbers. The simplest choice is a one-hot vector: a length-VV vector with a single 1. It is exact and needs no training, but it is useless as a representation. Every pair of distinct words is orthogonal, so "king" is as far from "queen" as from "fish". It has VV dimensions, often 10510^5 to 10610^6. Nothing learned about one word transfers to another. A dense embedding fixes all three: dd is a few hundred, similar words share directions, and a classifier trained on "cat" already knows something about "dog".

The history has two lines. Count-based methods build a word–context matrix and compress it. Latent Semantic Analysis (Deerwester et al., 1990) applied SVD to a term–document matrix. HAL (Lund and Burgess, 1996) used word–word counts from a sliding window. Church and Hanks (1990) introduced pointwise mutual information (PMI) as a word-association measure. Predictive methods learn vectors as parameters of a model that predicts words. Bengio et al. ("A Neural Probabilistic Language Model", 2003) learned an embedding table jointly with a neural language model. Collobert and Weston (2008) trained embeddings with a ranking objective and reused them across tasks. Word2Vec (Mikolov et al., 2013) removed the hidden layer and scaled to billions of tokens. GloVe (Pennington et al., 2014) fit global counts directly, and FastText (Bojanowski et al., 2017) added subword vectors. All of these assign one vector per word type. Contextual embeddings (ELMo, BERT, 2018) replaced that with a vector per occurrence.

From one-hot to dense

ew=E⊤xw,xw∈{0,1}V,  ∑ixw,i=1,E∈RV×d,cos⁡(u,v)=u⊤v∥u∥ ∥v∥e_w=E^\top x_w,\qquad x_w\in\{0,1\}^V,\ \ \textstyle\sum_i x_{w,i}=1,\qquad E\in\mathbb R^{V\times d},\qquad \cos(u,v)=\dfrac{u^\top v}{\|u\|\,\|v\|}
xwx_w
one-hot vector of word ww; xw⊤xw′=0x_w^\top x_{w'}=0 for every w≠w′w\neq w'
EE
embedding matrix; row ww is ewe_w, so the product is a table lookup
dd
embedding size: 50–300 for classic word vectors, 768 in GPT-2 small, 4096 in Llama 2 7B
cos⁡(u,v)\cos(u,v)
the standard similarity; it ignores vector length, which mostly tracks word frequency

The input layer of every language model, including the Decoder-only transformer, is exactly this lookup. What differs is where EE comes from. Classic word embeddings learn EE alone from co-occurrence statistics, then hand it to other models. An LLM learns EE jointly with the rest of the network from next-token prediction.

From a corpus to a V × d matrix of word vectors (hover a route)corpusthe king worehis crown …N tokenscounts #(w, c)window ±LV × V, sparsecount-based (LSA, HAL, PPMI)PPMI + truncated SVDM ≈ U_k Σ_k V_kᵀhybrid (GloVe, 2014)fit log X_ijweighted least squarespredictive (Word2Vec, 2013)predict context cσ(w·c), SGD on pairsword vectorsE ∈ ℝ^(V×d)d ≈ 50–300one row per wordcos(u, v)b − a + c ≈ dstreams (w, c) pairs,never stores the matrixone-hot x_queen ∈ {0,1}^VV ≈ 10⁵–10⁶ dims, one 1; every pair of wordsis orthogonal: cos(king, queen) = cos(king, fish) = 0Eᵀx = row of Edense e_queen ∈ ℝ^d0.21-0.470.080.62-0.150.33-0.710.40d ≈ 50–300 real numbers; geometry carries meaning:cos(king, queen) high, cos(king, fish) lowDistributional hypothesis (Harris 1954; Firth 1957): similar contexts, similar meanings.
fig.Three routes from a corpus to the same object, a V × d matrix of word vectors. Count-based methods build the sparse V × V co-occurrence matrix, reweight it with PPMI and keep the top k singular directions. GloVe builds the same matrix and fits its logarithm by weighted least squares. Word2Vec never stores the matrix: it streams (word, context) pairs and trains a classifier, which Levy and Goldberg (2014) showed implicitly factorises a shifted PMI matrix. Hover a route to highlight it. Below, a one-hot vector has V dimensions and makes every pair of words orthogonal, while a dense vector has a few hundred dimensions whose geometry carries similarity.

Count-based: co-occurrence, PMI and SVD

Slide a window of ±L\pm L words over the corpus and count how often word ww appears with context word cc. The result is a V×VV\times V matrix of counts #(w,c)\#(w,c). A row is already a vector for ww, but raw counts are dominated by frequency. "the" co-occurs with everything, so two unrelated nouns look similar because both appear next to "the". PMI corrects this by comparing the observed co-occurrence with what independence would predict.

PMI(w,c)=log⁡P(w,c)P(w) P(c)=log⁡#(w,c)⋅∣D∣#(w)⋅#(c),PPMI(w,c)=max⁡(0, PMI(w,c))\mathrm{PMI}(w,c)=\log\frac{P(w,c)}{P(w)\,P(c)}=\log\frac{\#(w,c)\cdot|D|}{\#(w)\cdot\#(c)},\qquad \mathrm{PPMI}(w,c)=\max\big(0,\ \mathrm{PMI}(w,c)\big)
#(w,c)\#(w,c)
number of times cc falls in the window around ww; ∣D∣|D| is the total number of pairs
#(w), #(c)\#(w),\ \#(c)
row and column sums: how often each appears in any pair
PMI>0\mathrm{PMI}>0
ww and cc co-occur more than chance; PMI =0=0 means independent
max⁡(0,⋅)\max(0,\cdot)
unseen pairs have PMI =−∞=-\infty and negative values are unreliable, so PPMI clips them to 0

PMI over-rewards rare contexts: one co-occurrence with a word seen twice gives a huge ratio. A standard fix is context distribution smoothing: raise the context counts to α=0.75\alpha=0.75 before normalising, Pα(c)=#(c)α/∑c′#(c′)αP_\alpha(c)=\#(c)^\alpha/\sum_{c'}\#(c')^\alpha. This raises the probability of rare contexts and so lowers their PMI. It is the same exponent Word2Vec uses for its negative-sampling distribution, and that is no coincidence (see below).

M≈Uk Σk Vk⊤,W=Uk Σk p,p∈{1,12,0}M\approx U_k\,\Sigma_k\,V_k^\top,\qquad W=U_k\,\Sigma_k^{\,p},\qquad p\in\{1,\tfrac12,0\}
MM
the PPMI matrix, V×VV\times V, sparse and high-dimensional
UkΣkVk⊤U_k\Sigma_kV_k^\top
truncated SVD: by Eckart–Young, the best rank-kk approximation of MM in Frobenius norm
WW
the kk-dimensional word vectors, one row per word
pp
eigenvalue weighting; Levy, Goldberg and Dagan (2015) found p=12p=\tfrac12 or 00 often better than the textbook p=1p=1

SVD does two jobs. It compresses VV sparse columns into kk dense ones. It also generalises: two words that never share a context, but whose contexts share contexts, end up close. That is second-order similarity, and it is why "king" and "queen" become neighbours even though they rarely appear together.

Predictive: learn the vectors by predicting context

Word2Vec turns the counts into a classification problem. Skip-gram with negative sampling (SGNS) learns a word vector ww and a context vector cc so that σ(w⋅c)\sigma(w\cdot c) is high for pairs seen in the corpus and low for kk random pairs drawn from the unigram distribution raised to 3/43/4. GloVe builds the count matrix and fits wi⊤w~j+bi+b~j≈log⁡Xijw_i^\top\tilde w_j+b_i+\tilde b_j\approx\log X_{ij} with a weighted least-squares loss. Both were reported to beat count-based vectors on analogy and similarity benchmarks, which led to the claim that "predict" beats "count" (Baroni, Dinu and Kruszewski, 2014).

SGNS optimum:w⋅c=log⁡#(w,c)⋅∣D∣#(w)⋅#(c)−log⁡k=PMI(w,c)−log⁡k\text{SGNS optimum:}\quad w\cdot c=\log\frac{\#(w,c)\cdot|D|}{\#(w)\cdot\#(c)}-\log k=\mathrm{PMI}(w,c)-\log k
w⋅cw\cdot c
the learned dot product of word and context vectors for one pair
kk
negatives per positive pair; the shift log⁡k\log k comes from the ratio of negative to positive examples
#(c)\#(c)
with Word2Vec's smoothed noise distribution this becomes #(c)0.75\#(c)^{0.75}, exactly context distribution smoothing

Levy and Goldberg ("Neural Word Embedding as Implicit Matrix Factorization", NeurIPS 2014) showed that when the dimension is large enough to fit each pair independently, SGNS reaches this optimum. It is therefore factorising a shifted PMI matrix. Levy, Goldberg and Dagan ("Improving Distributional Similarity with Lessons Learned from Word Embeddings", TACL 2015) then moved Word2Vec's hyperparameters into the count pipeline: context smoothing α=0.75\alpha=0.75, eigenvalue weighting pp, dynamic windows and subsampling. With matched settings, PPMI + SVD performed comparably to SGNS and GloVe. Much of the reported advantage of predictive methods came from these choices, not from prediction itself.

Similarity, analogies and evaluation

3CosAdd:d∗=arg⁡max⁡d∉{a,b,c} cos⁡ ⁣(ed, eb−ea+ec),e.g. eking−eman+ewoman≈equeen\text{3CosAdd:}\quad d^\ast=\arg\max_{d\notin\{a,b,c\}}\ \cos\!\big(e_d,\ e_b-e_a+e_c\big),\qquad \text{e.g. } e_{\text{king}}-e_{\text{man}}+e_{\text{woman}}\approx e_{\text{queen}}
a:b::c:da:b::c:d
"aa is to bb as cc is to dd"; vectors are unit-normalised first
d∉{a,b,c}d\notin\{a,b,c\}
the three question words are excluded; without this, the answer is very often bb or cc itself
3CosMul\text{3CosMul}
Levy and Goldberg (2014) multiply the cosines instead of adding them, which stops one large term dominating

Analogies work because a consistent relation, such as gender or tense, shows up as a roughly constant offset between vectors. Mikolov, Yih and Zweig (2013) documented this in recurrent-LM vectors. The Google analogy set from the Word2Vec paper has 19,544 questions, 8,869 semantic and 10,675 syntactic. Accuracy counts exact top-1 matches.

Word similarity benchmarks compare model cosines with human ratings by Spearman rank correlation ρ\rho. RG-65 (Rubenstein and Goodenough, 1965) has 65 noun pairs. WordSim-353 (Finkelstein et al., 2002) has 353 pairs and mixes similarity with relatedness ("cup"–"coffee"). SimLex-999 (Hill, Reichart and Korhonen, 2015) rates strict similarity only, so related but dissimilar pairs score low. Intrinsic scores do not always predict extrinsic performance, which means accuracy on a downstream task such as tagging, NER or classification when the vectors are used as input features. Report both.

Widely used pretrained word vectors

word2vec GoogleNews
300-d skip-gram vectors for 3M words and phrases, trained on about 100B words of Google News (released with Mikolov et al., 2013)
GloVe 6B
Wikipedia 2014 + Gigaword 5, 6B tokens, 400K uncased vocabulary, d∈{50,100,200,300}d\in\{50,100,200,300\}
GloVe 42B / 840B
Common Crawl, 300-d; 1.9M uncased and 2.2M cased vocabulary
GloVe Twitter
2B tweets, 27B tokens, 1.2M vocabulary, d∈{25,50,100,200}d\in\{25,50,100,200\}
fastText crawl-300d-2M
2M word vectors, 300-d, trained on 600B tokens of Common Crawl (Mikolov et al., 2018)
Typical training
window 2–10, dd = 100–300, rare words (count below 5) dropped, frequent words subsampled
400 sentences · 1860 tokens · 21 words · window ±1 · PPMIthekingqueenmanwomanworeruledworkedwalkedatechasedhishercrownlandfarmdogcatbreadmeatfishthekingqueenmanwomanworeruledworkedwalkedatechasedhishercrownlandfarmdogcatbreadmeatfishrows w, columns c · stronger colour = larger · max 2.34rows of U₂Σ₂ (first two SVD dimensions)thekingqueenmanwomanworeruledworkedwalkedatechasedhishercrownlandfarmdogcatbreadmeatfishhover a cell for its count and PMI · click a word to select itnearest to king (cosine, k = 4): queen 0.999 · her 0.949 · his 0.947king − man + woman → queen 1.000 (next her 0.932)his − king + queen → her 0.999 (next man 0.681)toy similarity benchmark (16 rated pairs): Spearman ρ = 0.715σ₁…σ₅ = 6.21, 6.14, 4.53, 4.51, 3.43 · top 4 keep 70.6% of ‖M‖²● people ● animals ● objects ● his/her ● verbs, the · the map shows 2 dims, the readouts use k
matrix
fig.The full count-based pipeline, computed live on a toy corpus of 400 template sentences (1,860 tokens, 21 words): royals wear crowns and rule land, commoners work farms, everyone walks a dog and eats bread, and his/her carries gender. The left panel is the word × context matrix and the right panel the first two SVD dimensions. The readouts use the top k. At the default (PPMI, window ±1, k = 4), the singular values are 6.21, 6.14, 4.53, 4.51 and 3.43, and the top 4 keep 70.6% of the squared norm. The nearest words to king are queen 0.999, her 0.949 and his 0.947. king − man + woman gives queen (1.000, next her 0.932), his − king + queen gives her (0.999), and the 16-pair toy benchmark scores Spearman ρ = 0.715. Hover king/wore: 24 co-occurrences, PMI 1.652. Switch to raw counts and the largest cell is #(the, man) = 85. king's neighbours become woman, man and queen, and ρ drops to 0.613. Widening the window to ±2 mixes topic into similarity: king's neighbours become farm 0.997, crown 0.993 and land 0.991, his − king + queen gives wore, and ρ falls to 0.579. At ±3, king − man + woman fails too (ruled) and ρ is 0.481. With k = 8 dimensions the top 8 keep 93.1%. king stays next to queen (0.998), but the pronouns drop to 0.434.

Worked example: why PMI beats raw counts

Three words and three contexts with made-up counts. "the" is the frequent, uninformative context.

wordthepetdriverow sum
cat206026
dog225128
car250631
column sum6711785
Co-occurrence counts #(w, c); |D| = 85 pairs

PMI, PPMI and cosine (computed in numpy)

  1. 1Raw-count cosines: cos⁡(cat,dog)=0.997\cos(\text{cat},\text{dog})=0.997 but also cos⁡(cat,car)=0.931\cos(\text{cat},\text{car})=0.931. The shared "the" column dominates both.
  2. 2PMI(cat,pet)=log⁡6⋅8526⋅11=log⁡1.783=0.578\mathrm{PMI}(\text{cat},\text{pet})=\log\frac{6\cdot85}{26\cdot11}=\log1.783=0.578. Cats appear with "pet" 1.8 times more than chance.
  3. 3PMI(cat,the)=log⁡20⋅8526⋅67=log⁡0.976=−0.024\mathrm{PMI}(\text{cat},\text{the})=\log\frac{20\cdot85}{26\cdot67}=\log0.976=-0.024. "the" is independent of "cat", so PPMI sets it to 0. The same happens for dog (−0.003-0.003), and car keeps only 0.0230.023.
  4. 4PMI(car,drive)=log⁡6⋅8531⋅7=0.855\mathrm{PMI}(\text{car},\text{drive})=\log\frac{6\cdot85}{31\cdot7}=0.855; PMI(dog,drive)=log⁡1⋅8528⋅7=−0.835\mathrm{PMI}(\text{dog},\text{drive})=\log\frac{1\cdot85}{28\cdot7}=-0.835, clipped to 0.
  5. 5PPMI rows: cat (0, 0.578, 0)(0,\,0.578,\,0), dog (0, 0.322, 0)(0,\,0.322,\,0), car (0.023, 0, 0.855)(0.023,\,0,\,0.855). Now cos⁡(cat,dog)=1.000\cos(\text{cat},\text{dog})=1.000 and cos⁡(cat,car)=0.000\cos(\text{cat},\text{car})=0.000.
  6. 6With context smoothing α=0.75\alpha=0.75: P0.75(drive)=0.128P_{0.75}(\text{drive})=0.128 instead of 0.0820.082, so PMI(dog,drive)\mathrm{PMI}(\text{dog},\text{drive}) falls further to −1.272-1.272. Rare contexts are penalised.
python
1import numpy as np
2
3corpus = """the king wore his crown . the queen wore her crown . the king ruled his land .
4the queen ruled her land . the man worked his farm . the woman worked her farm .
5the man walked his dog . the woman walked her dog . the dog ate meat . the cat ate fish""".split()
6
7vocab = sorted(set(corpus) - {"."})
8idx = {w: i for i, w in enumerate(vocab)}
9
10def cooccurrence(tokens, window=2):
11 C = np.zeros((len(vocab), len(vocab)))
12 for i, w in enumerate(tokens):
13 if w == ".":
14 continue
15 for j in range(max(0, i - window), min(len(tokens), i + window + 1)):
16 if j != i and tokens[j] != ".":
17 C[idx[w], idx[tokens[j]]] += 1
18 return C
19
20def ppmi(C, alpha=0.75):
21 """PPMI with context-distribution smoothing (Levy, Goldberg & Dagan 2015)."""
22 pw = C.sum(1, keepdims=True) / C.sum()
23 pc = C.sum(0, keepdims=True) ** alpha
24 pc /= pc.sum()
25 with np.errstate(divide="ignore"):
26 pmi = np.log(C / C.sum() / (pw * pc))
27 return np.maximum(pmi, 0) # log 0 = -inf is clipped to 0 as well
28
29C = cooccurrence(corpus, window=1)
30M = ppmi(C)
31U, S, Vt = np.linalg.svd(M)
32k = 8
33E = U[:, :k] * np.sqrt(S[:k]) # W = U_k Σ_k^p with p = 0.5
34E /= np.linalg.norm(E, axis=1, keepdims=True)
35
36def neighbours(w, n=3):
37 sims = E @ E[idx[w]]
38 return [(vocab[j], round(float(sims[j]), 3)) for j in np.argsort(-sims) if vocab[j] != w][:n]
39
40def analogy(a, b, c): # a : b :: c : ? (3CosAdd)
41 q = E[idx[b]] - E[idx[a]] + E[idx[c]]
42 sims = E @ q / np.linalg.norm(q)
43 return next(vocab[j] for j in np.argsort(-sims) if vocab[j] not in (a, b, c))
44
45print(len(vocab), "words; top singular values", np.round(S[:4], 2)) # 19 words; [5.76 5.25 4.88 4.5 ]
46print("king:", neighbours("king")) # [('queen', 1.0), ('cat', 0.31), ('dog', 0.185)]
47print("man : king :: woman :", analogy("man", "king", "woman")) # queen
Count-based word vectors with numpy: windowed counts, smoothed PPMI, truncated SVD with p = 0.5, neighbours and 3CosAdd (run; outputs in comments)

Use when

  • +Dense, low-dimensional, and cheap to look up: one row of a table per word
  • +Learned from raw text with no labels; transfer what is known about one word to similar words
  • +Simple geometry: cosine for similarity, vector offsets for many relations
  • +Count-based and predictive methods are well understood and closely related

Avoid when

  • −One vector per word type: no polysemy, no dependence on the sentence
  • −Out-of-vocabulary words get no vector (FastText's subwords partly fix this)
  • −Antonyms share contexts and end up close ('hot' vs 'cold')
  • −Inherit corpus biases; intrinsic benchmarks track downstream accuracy only loosely

Where it's used

Feature input

Initialising the embedding layer of CNN/RNN taggers and classifiers before contextual models

Lexical tasks

Synonym discovery, query expansion, lexicon induction and cross-lingual dictionary mapping

Analysis

Measuring semantic change over time and auditing social biases in corpora

LLM input layer

Every model in Transformer-based language models starts with a learned V×dV\times d token embedding of the same form

Check yourself

Why does PPMI usually give better word vectors than raw co-occurrence counts?

Check yourself

Levy and Goldberg (2014) showed that skip-gram with k negative samples implicitly factorises which matrix?

Remember

  • ✓A word embedding is a row of a V×dV\times d matrix, with d≈50d\approx50–300≪V300\ll V. Lookup is E⊤xwE^\top x_w, and similarity is cosine.
  • ✓Distributional hypothesis: words in similar contexts have similar meanings (Harris 1954; Firth 1957).
  • ✓Count-based: window counts → PPMI=max⁡(0,log⁡P(w,c)P(w)P(c))\mathrm{PPMI}=\max(0,\log\frac{P(w,c)}{P(w)P(c)}) → truncated SVD W=UkΣkpW=U_k\Sigma_k^p.
  • ✓Predictive: Word2Vec (SGNS) and GloVe. SGNS implicitly factorises PMI−log⁡k\mathrm{PMI}-\log k, and with matched hyperparameters (α=0.75\alpha=0.75, pp, windows) count methods perform comparably.
  • ✓Evaluate with similarity benchmarks (Spearman ρ\rho on WordSim-353, SimLex-999) and with 3CosAdd analogies that exclude the query words. Check downstream tasks too.
  • ✓Limits: one vector per word type, no out-of-vocabulary words, antonyms close together, corpus biases. Contextual embeddings address the first.
← back to Deep Learning roadmap