# Word embeddings

> A word embedding maps each word to a dense vector of d ≈ 50–300 numbers so that words used in similar contexts lie close in cosine similarity. Count methods factor a PPMI co-occurrence matrix with truncated SVD; Word2Vec and GloVe learn much the same geometry by optimisation.

Source: https://www.mlfoundry.in/learn/word-embeddings  
Section: Deep Learning › Large Language Models (LLMs)  
Level: core · 15 min read · updated 2026-10-13

**Word embedding:** is a map $w\mapsto e_w\in\mathbb R^d$ from a vocabulary of $V$ words to dense vectors, with $d$ usually 50–300, far smaller than $V$. It is learned from unlabelled text so that geometry reflects usage: words that occur in similar contexts get vectors with high cosine similarity. The idea rests on the **distributional hypothesis**. Harris ("Distributional Structure", 1954) argued that differences in meaning go with differences in distribution, and Firth (1957) put it as "you shall know a word by the company it keeps". In practice the embeddings are the rows of a $V\times d$ matrix $E$, and looking a word up is multiplying a one-hot vector by $E^\top$.

A model that reads words needs to turn each one into numbers. The simplest choice is a **one-hot** vector: a length-$V$ vector with a single 1. It is exact and needs no training, but it is useless as a representation. Every pair of distinct words is orthogonal, so "king" is as far from "queen" as from "fish". It has $V$ dimensions, often $10^5$ to $10^6$. Nothing learned about one word transfers to another. A dense embedding fixes all three: $d$ is a few hundred, similar words share directions, and a classifier trained on "cat" already knows something about "dog".

The history has two lines. **Count-based** methods build a word–context matrix and compress it. Latent Semantic Analysis (Deerwester et al., 1990) applied SVD to a term–document matrix. HAL (Lund and Burgess, 1996) used word–word counts from a sliding window. Church and Hanks (1990) introduced pointwise mutual information (PMI) as a word-association measure. **Predictive** methods learn vectors as parameters of a model that predicts words. Bengio et al. ("A Neural Probabilistic Language Model", 2003) learned an embedding table jointly with a neural language model. Collobert and Weston (2008) trained embeddings with a ranking objective and reused them across tasks. **Word2Vec** (Mikolov et al., 2013) removed the hidden layer and scaled to billions of tokens. **GloVe** (Pennington et al., 2014) fit global counts directly, and **FastText** (Bojanowski et al., 2017) added subword vectors. All of these assign *one vector per word type*. **Contextual embeddings** (ELMo, BERT, 2018) replaced that with a vector per occurrence.

## From one-hot to dense

$$
e_w=E^\top x_w,\qquad x_w\in\{0,1\}^V,\ \ \textstyle\sum_i x_{w,i}=1,\qquad E\in\mathbb R^{V\times d},\qquad \cos(u,v)=\dfrac{u^\top v}{\|u\|\,\|v\|}
$$

- $x_w$: one-hot vector of word $w$; $x_w^\top x_{w'}=0$ for every $w\neq w'$
- $E$: embedding matrix; row $w$ is $e_w$, so the product is a table lookup
- $d$: embedding size: 50–300 for classic word vectors, 768 in GPT-2 small, 4096 in Llama 2 7B
- $\cos(u,v)$: the standard similarity; it ignores vector length, which mostly tracks word frequency

The input layer of every language model, including the **Decoder-only transformer**, is exactly this lookup. What differs is where $E$ comes from. Classic word embeddings learn $E$ alone from co-occurrence statistics, then hand it to other models. An LLM learns $E$ jointly with the rest of the network from next-token prediction.

> Figure: Three routes from a corpus to the same object, a V × d matrix of word vectors. Count-based methods build the sparse V × V co-occurrence matrix, reweight it with PPMI and keep the top k singular directions. GloVe builds the same matrix and fits its logarithm by weighted least squares. Word2Vec never stores the matrix: it streams (word, context) pairs and trains a classifier, which Levy and Goldberg (2014) showed implicitly factorises a shifted PMI matrix. Hover a route to highlight it. Below, a one-hot vector has V dimensions and makes every pair of words orthogonal, while a dense vector has a few hundred dimensions whose geometry carries similarity.

## Count-based: co-occurrence, PMI and SVD

Slide a window of $\pm L$ words over the corpus and count how often word $w$ appears with context word $c$. The result is a $V\times V$ matrix of counts $\#(w,c)$. A row is already a vector for $w$, but raw counts are dominated by frequency. "the" co-occurs with everything, so two unrelated nouns look similar because both appear next to "the". PMI corrects this by comparing the observed co-occurrence with what independence would predict.

$$
\mathrm{PMI}(w,c)=\log\frac{P(w,c)}{P(w)\,P(c)}=\log\frac{\#(w,c)\cdot|D|}{\#(w)\cdot\#(c)},\qquad \mathrm{PPMI}(w,c)=\max\big(0,\ \mathrm{PMI}(w,c)\big)
$$

- $\#(w,c)$: number of times $c$ falls in the window around $w$; $|D|$ is the total number of pairs
- $\#(w),\ \#(c)$: row and column sums: how often each appears in any pair
- $\mathrm{PMI}>0$: $w$ and $c$ co-occur more than chance; PMI $=0$ means independent
- $\max(0,\cdot)$: unseen pairs have PMI $=-\infty$ and negative values are unreliable, so PPMI clips them to 0

PMI over-rewards rare contexts: one co-occurrence with a word seen twice gives a huge ratio. A standard fix is **context distribution smoothing**: raise the context counts to $\alpha=0.75$ before normalising, $P_\alpha(c)=\#(c)^\alpha/\sum_{c'}\#(c')^\alpha$. This raises the probability of rare contexts and so lowers their PMI. It is the same exponent Word2Vec uses for its negative-sampling distribution, and that is no coincidence (see below).

$$
M\approx U_k\,\Sigma_k\,V_k^\top,\qquad W=U_k\,\Sigma_k^{\,p},\qquad p\in\{1,\tfrac12,0\}
$$

- $M$: the PPMI matrix, $V\times V$, sparse and high-dimensional
- $U_k\Sigma_kV_k^\top$: truncated SVD: by Eckart–Young, the best rank-$k$ approximation of $M$ in Frobenius norm
- $W$: the $k$-dimensional word vectors, one row per word
- $p$: eigenvalue weighting; Levy, Goldberg and Dagan (2015) found $p=\tfrac12$ or $0$ often better than the textbook $p=1$

SVD does two jobs. It compresses $V$ sparse columns into $k$ dense ones. It also *generalises*: two words that never share a context, but whose contexts share contexts, end up close. That is second-order similarity, and it is why "king" and "queen" become neighbours even though they rarely appear together.

## Predictive: learn the vectors by predicting context

**Word2Vec** turns the counts into a classification problem. Skip-gram with negative sampling (SGNS) learns a word vector $w$ and a context vector $c$ so that $\sigma(w\cdot c)$ is high for pairs seen in the corpus and low for $k$ random pairs drawn from the unigram distribution raised to $3/4$. **GloVe** builds the count matrix and fits $w_i^\top\tilde w_j+b_i+\tilde b_j\approx\log X_{ij}$ with a weighted least-squares loss. Both were reported to beat count-based vectors on analogy and similarity benchmarks, which led to the claim that "predict" beats "count" (Baroni, Dinu and Kruszewski, 2014).

$$
\text{SGNS optimum:}\quad w\cdot c=\log\frac{\#(w,c)\cdot|D|}{\#(w)\cdot\#(c)}-\log k=\mathrm{PMI}(w,c)-\log k
$$

- $w\cdot c$: the learned dot product of word and context vectors for one pair
- $k$: negatives per positive pair; the shift $\log k$ comes from the ratio of negative to positive examples
- $\#(c)$: with Word2Vec's smoothed noise distribution this becomes $\#(c)^{0.75}$, exactly context distribution smoothing

Levy and Goldberg ("Neural Word Embedding as Implicit Matrix Factorization", NeurIPS 2014) showed that when the dimension is large enough to fit each pair independently, SGNS reaches this optimum. It is therefore factorising a shifted PMI matrix. Levy, Goldberg and Dagan ("Improving Distributional Similarity with Lessons Learned from Word Embeddings", TACL 2015) then moved Word2Vec's hyperparameters into the count pipeline: context smoothing $\alpha=0.75$, eigenvalue weighting $p$, dynamic windows and subsampling. With matched settings, PPMI + SVD performed comparably to SGNS and GloVe. Much of the reported advantage of predictive methods came from these choices, not from prediction itself.

## Similarity, analogies and evaluation

$$
\text{3CosAdd:}\quad d^\ast=\arg\max_{d\notin\{a,b,c\}}\ \cos\!\big(e_d,\ e_b-e_a+e_c\big),\qquad \text{e.g. } e_{\text{king}}-e_{\text{man}}+e_{\text{woman}}\approx e_{\text{queen}}
$$

- $a:b::c:d$: "$a$ is to $b$ as $c$ is to $d$"; vectors are unit-normalised first
- $d\notin\{a,b,c\}$: the three question words are excluded; without this, the answer is very often $b$ or $c$ itself
- $\text{3CosMul}$: Levy and Goldberg (2014) multiply the cosines instead of adding them, which stops one large term dominating

Analogies work because a consistent relation, such as gender or tense, shows up as a roughly constant offset between vectors. Mikolov, Yih and Zweig (2013) documented this in recurrent-LM vectors. The Google analogy set from the Word2Vec paper has 19,544 questions, 8,869 semantic and 10,675 syntactic. Accuracy counts exact top-1 matches.

**Word similarity** benchmarks compare model cosines with human ratings by Spearman rank correlation $\rho$. RG-65 (Rubenstein and Goodenough, 1965) has 65 noun pairs. WordSim-353 (Finkelstein et al., 2002) has 353 pairs and mixes similarity with relatedness ("cup"–"coffee"). SimLex-999 (Hill, Reichart and Korhonen, 2015) rates strict similarity only, so related but dissimilar pairs score low. Intrinsic scores do not always predict **extrinsic** performance, which means accuracy on a downstream task such as tagging, NER or classification when the vectors are used as input features. Report both.

**Widely used pretrained word vectors**

- **word2vec GoogleNews:** 300-d skip-gram vectors for 3M words and phrases, trained on about 100B words of Google News (released with Mikolov et al., 2013)
- **GloVe 6B:** Wikipedia 2014 + Gigaword 5, 6B tokens, 400K uncased vocabulary, $d\in\{50,100,200,300\}$
- **GloVe 42B / 840B:** Common Crawl, 300-d; 1.9M uncased and 2.2M cased vocabulary
- **GloVe Twitter:** 2B tweets, 27B tokens, 1.2M vocabulary, $d\in\{25,50,100,200\}$
- **fastText crawl-300d-2M:** 2M word vectors, 300-d, trained on 600B tokens of Common Crawl (Mikolov et al., 2018)
- **Typical training:** window 2–10, $d$ = 100–300, rare words (count below 5) dropped, frequent words subsampled

> Figure: The full count-based pipeline, computed live on a toy corpus of 400 template sentences (1,860 tokens, 21 words): royals wear crowns and rule land, commoners work farms, everyone walks a dog and eats bread, and his/her carries gender. The left panel is the word × context matrix and the right panel the first two SVD dimensions. The readouts use the top k. At the default (PPMI, window ±1, k = 4), the singular values are 6.21, 6.14, 4.53, 4.51 and 3.43, and the top 4 keep 70.6% of the squared norm. The nearest words to king are queen 0.999, her 0.949 and his 0.947. king − man + woman gives queen (1.000, next her 0.932), his − king + queen gives her (0.999), and the 16-pair toy benchmark scores Spearman ρ = 0.715. Hover king/wore: 24 co-occurrences, PMI 1.652. Switch to raw counts and the largest cell is #(the, man) = 85. king's neighbours become woman, man and queen, and ρ drops to 0.613. Widening the window to ±2 mixes topic into similarity: king's neighbours become farm 0.997, crown 0.993 and land 0.991, his − king + queen gives wore, and ρ falls to 0.579. At ±3, king − man + woman fails too (ruled) and ρ is 0.481. With k = 8 dimensions the top 8 keep 93.1%. king stays next to queen (0.998), but the pronouns drop to 0.434.

## Worked example: why PMI beats raw counts

Three words and three contexts with made-up counts. "the" is the frequent, uninformative context.

Co-occurrence counts #(w, c); |D| = 85 pairs

| word | the | pet | drive | row sum |
| --- | --- | --- | --- | --- |
| cat | 20 | 6 | 0 | 26 |
| dog | 22 | 5 | 1 | 28 |
| car | 25 | 0 | 6 | 31 |
| column sum | 67 | 11 | 7 | 85 |

**PMI, PPMI and cosine (computed in numpy)**

1. Raw-count cosines: $\cos(\text{cat},\text{dog})=0.997$ but also $\cos(\text{cat},\text{car})=0.931$. The shared "the" column dominates both.
2. $\mathrm{PMI}(\text{cat},\text{pet})=\log\frac{6\cdot85}{26\cdot11}=\log1.783=0.578$. Cats appear with "pet" 1.8 times more than chance.
3. $\mathrm{PMI}(\text{cat},\text{the})=\log\frac{20\cdot85}{26\cdot67}=\log0.976=-0.024$. "the" is independent of "cat", so PPMI sets it to 0. The same happens for dog ($-0.003$), and car keeps only $0.023$.
4. $\mathrm{PMI}(\text{car},\text{drive})=\log\frac{6\cdot85}{31\cdot7}=0.855$; $\mathrm{PMI}(\text{dog},\text{drive})=\log\frac{1\cdot85}{28\cdot7}=-0.835$, clipped to 0.
5. PPMI rows: cat $(0,\,0.578,\,0)$, dog $(0,\,0.322,\,0)$, car $(0.023,\,0,\,0.855)$. Now $\cos(\text{cat},\text{dog})=1.000$ and $\cos(\text{cat},\text{car})=0.000$.
6. With context smoothing $\alpha=0.75$: $P_{0.75}(\text{drive})=0.128$ instead of $0.082$, so $\mathrm{PMI}(\text{dog},\text{drive})$ falls further to $-1.272$. Rare contexts are penalised.

```python
import numpy as np

corpus = """the king wore his crown . the queen wore her crown . the king ruled his land .
the queen ruled her land . the man worked his farm . the woman worked her farm .
the man walked his dog . the woman walked her dog . the dog ate meat . the cat ate fish""".split()

vocab = sorted(set(corpus) - {"."})
idx = {w: i for i, w in enumerate(vocab)}

def cooccurrence(tokens, window=2):
    C = np.zeros((len(vocab), len(vocab)))
    for i, w in enumerate(tokens):
        if w == ".":
            continue
        for j in range(max(0, i - window), min(len(tokens), i + window + 1)):
            if j != i and tokens[j] != ".":
                C[idx[w], idx[tokens[j]]] += 1
    return C

def ppmi(C, alpha=0.75):
    """PPMI with context-distribution smoothing (Levy, Goldberg & Dagan 2015)."""
    pw = C.sum(1, keepdims=True) / C.sum()
    pc = C.sum(0, keepdims=True) ** alpha
    pc /= pc.sum()
    with np.errstate(divide="ignore"):
        pmi = np.log(C / C.sum() / (pw * pc))
    return np.maximum(pmi, 0)          # log 0 = -inf is clipped to 0 as well

C = cooccurrence(corpus, window=1)
M = ppmi(C)
U, S, Vt = np.linalg.svd(M)
k = 8
E = U[:, :k] * np.sqrt(S[:k])          # W = U_k Σ_k^p with p = 0.5
E /= np.linalg.norm(E, axis=1, keepdims=True)

def neighbours(w, n=3):
    sims = E @ E[idx[w]]
    return [(vocab[j], round(float(sims[j]), 3)) for j in np.argsort(-sims) if vocab[j] != w][:n]

def analogy(a, b, c):                  # a : b :: c : ?   (3CosAdd)
    q = E[idx[b]] - E[idx[a]] + E[idx[c]]
    sims = E @ q / np.linalg.norm(q)
    return next(vocab[j] for j in np.argsort(-sims) if vocab[j] not in (a, b, c))

print(len(vocab), "words; top singular values", np.round(S[:4], 2))  # 19 words; [5.76 5.25 4.88 4.5 ]
print("king:", neighbours("king"))    # [('queen', 1.0), ('cat', 0.31), ('dog', 0.185)]
print("man : king :: woman :", analogy("man", "king", "woman"))   # queen
```

Count-based word vectors with numpy: windowed counts, smoothed PPMI, truncated SVD with p = 0.5, neighbours and 3CosAdd (run; outputs in comments)

> **Analogy scores hide an exclusion rule:** $e_{\text{king}}-e_{\text{man}}+e_{\text{woman}}$ is usually closest to $e_{\text{king}}$ itself. Standard evaluation removes $a$, $b$ and $c$ from the candidates before taking the argmax. Without that rule, the top answer is often one of the query words. Several widely shared "biased analogy" examples depended on this rule or on choosing which outputs to show (Nissim, van Noord and van der Goot, 2020). Report the candidate set you used.

> **Predict and count are one family:** SGNS factorises $\mathrm{PMI}-\log k$, and GloVe fits $\log X_{ij}$, which is PMI up to the bias terms. Both use a fractional power of about $3/4$ on frequencies. Once the hyperparameters are matched, the three methods give similar vectors. Choose by corpus size, memory and tooling.

> **One vector per word type:** "bank" gets a single vector that averages its river and money senses, weighted by how often each occurs. Embeddings also absorb the biases of their corpus. Bolukbasi et al. (2016) found gender stereotypes in occupation words in the GoogleNews vectors. **Contextual embeddings** solve the polysemy problem by computing a vector for every occurrence. Bias needs explicit measurement either way.

**Strengths**

- Dense, low-dimensional, and cheap to look up: one row of a table per word
- Learned from raw text with no labels; transfer what is known about one word to similar words
- Simple geometry: cosine for similarity, vector offsets for many relations
- Count-based and predictive methods are well understood and closely related

**Limitations**

- One vector per word type: no polysemy, no dependence on the sentence
- Out-of-vocabulary words get no vector (FastText's subwords partly fix this)
- Antonyms share contexts and end up close ('hot' vs 'cold')
- Inherit corpus biases; intrinsic benchmarks track downstream accuracy only loosely

- **Feature input:** Initialising the embedding layer of CNN/RNN taggers and classifiers before contextual models
- **Lexical tasks:** Synonym discovery, query expansion, lexicon induction and cross-lingual dictionary mapping
- **Analysis:** Measuring semantic change over time and auditing social biases in corpora
- **LLM input layer:** Every model in **Transformer-based language models** starts with a learned $V\times d$ token embedding of the same form

**Question:** Why does PPMI usually give better word vectors than raw co-occurrence counts?

- A. It makes the matrix symmetric
- B. It removes the effect of frequency: shared frequent contexts like 'the' get PMI ≈ 0 and stop dominating the similarity
- C. It reduces the matrix to d dimensions
- D. It turns counts into probabilities that sum to 1 per row

**Answer:** B. It removes the effect of frequency: shared frequent contexts like 'the' get PMI ≈ 0 and stop dominating the similarity — PMI compares $P(w,c)$ with $P(w)P(c)$. A context that co-occurs with every word at its base rate has PMI near 0, so it no longer makes unrelated words look similar. In the worked example, $\cos(\text{cat},\text{car})$ falls from $0.931$ to $0$.

**Question:** Levy and Goldberg (2014) showed that skip-gram with k negative samples implicitly factorises which matrix?

- A. The raw co-occurrence count matrix
- B. The log of the co-occurrence counts plus two bias vectors
- C. The PMI matrix shifted by −log k
- D. The term–document matrix used by LSA

**Answer:** C. The PMI matrix shifted by −log k — At the optimum, each word–context dot product equals $\mathrm{PMI}(w,c)-\log k$. The positive pairs push $w\cdot c$ up in proportion to $\#(w,c)$, the $k$ negatives push it down in proportion to $k\,\#(w)\,P_n(c)$, and the log-ratio is the shifted PMI.

**Key points**

- A word embedding is a row of a $V\times d$ matrix, with $d\approx50$–$300\ll V$. Lookup is $E^\top x_w$, and similarity is cosine.
- Distributional hypothesis: words in similar contexts have similar meanings (Harris 1954; Firth 1957).
- Count-based: window counts → $\mathrm{PPMI}=\max(0,\log\frac{P(w,c)}{P(w)P(c)})$ → truncated SVD $W=U_k\Sigma_k^p$.
- Predictive: Word2Vec (SGNS) and GloVe. SGNS implicitly factorises $\mathrm{PMI}-\log k$, and with matched hyperparameters ($\alpha=0.75$, $p$, windows) count methods perform comparably.
- Evaluate with similarity benchmarks (Spearman $\rho$ on WordSim-353, SimLex-999) and with 3CosAdd analogies that exclude the query words. Check downstream tasks too.
- Limits: one vector per word type, no out-of-vocabulary words, antonyms close together, corpus biases. Contextual embeddings address the first.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
