Definition
Word embedding is a map from a vocabulary of words to dense vectors, with usually 50–300, far smaller than . It is learned from unlabelled text so that geometry reflects usage: words that occur in similar contexts get vectors with high cosine similarity. The idea rests on the distributional hypothesis. Harris ("Distributional Structure", 1954) argued that differences in meaning go with differences in distribution, and Firth (1957) put it as "you shall know a word by the company it keeps". In practice the embeddings are the rows of a matrix , and looking a word up is multiplying a one-hot vector by .
A model that reads words needs to turn each one into numbers. The simplest choice is a one-hot vector: a length- vector with a single 1. It is exact and needs no training, but it is useless as a representation. Every pair of distinct words is orthogonal, so "king" is as far from "queen" as from "fish". It has dimensions, often to . Nothing learned about one word transfers to another. A dense embedding fixes all three: is a few hundred, similar words share directions, and a classifier trained on "cat" already knows something about "dog".
The history has two lines. Count-based methods build a word–context matrix and compress it. Latent Semantic Analysis (Deerwester et al., 1990) applied SVD to a term–document matrix. HAL (Lund and Burgess, 1996) used word–word counts from a sliding window. Church and Hanks (1990) introduced pointwise mutual information (PMI) as a word-association measure. Predictive methods learn vectors as parameters of a model that predicts words. Bengio et al. ("A Neural Probabilistic Language Model", 2003) learned an embedding table jointly with a neural language model. Collobert and Weston (2008) trained embeddings with a ranking objective and reused them across tasks. Word2Vec (Mikolov et al., 2013) removed the hidden layer and scaled to billions of tokens. GloVe (Pennington et al., 2014) fit global counts directly, and FastText (Bojanowski et al., 2017) added subword vectors. All of these assign one vector per word type. Contextual embeddings (ELMo, BERT, 2018) replaced that with a vector per occurrence.
From one-hot to dense
- one-hot vector of word ; for every
- embedding matrix; row is , so the product is a table lookup
- embedding size: 50–300 for classic word vectors, 768 in GPT-2 small, 4096 in Llama 2 7B
- the standard similarity; it ignores vector length, which mostly tracks word frequency
The input layer of every language model, including the Decoder-only transformer, is exactly this lookup. What differs is where comes from. Classic word embeddings learn alone from co-occurrence statistics, then hand it to other models. An LLM learns jointly with the rest of the network from next-token prediction.
Count-based: co-occurrence, PMI and SVD
Slide a window of words over the corpus and count how often word appears with context word . The result is a matrix of counts . A row is already a vector for , but raw counts are dominated by frequency. "the" co-occurs with everything, so two unrelated nouns look similar because both appear next to "the". PMI corrects this by comparing the observed co-occurrence with what independence would predict.
- number of times falls in the window around ; is the total number of pairs
- row and column sums: how often each appears in any pair
- and co-occur more than chance; PMI means independent
- unseen pairs have PMI and negative values are unreliable, so PPMI clips them to 0
PMI over-rewards rare contexts: one co-occurrence with a word seen twice gives a huge ratio. A standard fix is context distribution smoothing: raise the context counts to before normalising, . This raises the probability of rare contexts and so lowers their PMI. It is the same exponent Word2Vec uses for its negative-sampling distribution, and that is no coincidence (see below).
- the PPMI matrix, , sparse and high-dimensional
- truncated SVD: by Eckart–Young, the best rank- approximation of in Frobenius norm
- the -dimensional word vectors, one row per word
- eigenvalue weighting; Levy, Goldberg and Dagan (2015) found or often better than the textbook
SVD does two jobs. It compresses sparse columns into dense ones. It also generalises: two words that never share a context, but whose contexts share contexts, end up close. That is second-order similarity, and it is why "king" and "queen" become neighbours even though they rarely appear together.
Predictive: learn the vectors by predicting context
Word2Vec turns the counts into a classification problem. Skip-gram with negative sampling (SGNS) learns a word vector and a context vector so that is high for pairs seen in the corpus and low for random pairs drawn from the unigram distribution raised to . GloVe builds the count matrix and fits with a weighted least-squares loss. Both were reported to beat count-based vectors on analogy and similarity benchmarks, which led to the claim that "predict" beats "count" (Baroni, Dinu and Kruszewski, 2014).
- the learned dot product of word and context vectors for one pair
- negatives per positive pair; the shift comes from the ratio of negative to positive examples
- with Word2Vec's smoothed noise distribution this becomes , exactly context distribution smoothing
Levy and Goldberg ("Neural Word Embedding as Implicit Matrix Factorization", NeurIPS 2014) showed that when the dimension is large enough to fit each pair independently, SGNS reaches this optimum. It is therefore factorising a shifted PMI matrix. Levy, Goldberg and Dagan ("Improving Distributional Similarity with Lessons Learned from Word Embeddings", TACL 2015) then moved Word2Vec's hyperparameters into the count pipeline: context smoothing , eigenvalue weighting , dynamic windows and subsampling. With matched settings, PPMI + SVD performed comparably to SGNS and GloVe. Much of the reported advantage of predictive methods came from these choices, not from prediction itself.
Similarity, analogies and evaluation
- " is to as is to "; vectors are unit-normalised first
- the three question words are excluded; without this, the answer is very often or itself
- Levy and Goldberg (2014) multiply the cosines instead of adding them, which stops one large term dominating
Analogies work because a consistent relation, such as gender or tense, shows up as a roughly constant offset between vectors. Mikolov, Yih and Zweig (2013) documented this in recurrent-LM vectors. The Google analogy set from the Word2Vec paper has 19,544 questions, 8,869 semantic and 10,675 syntactic. Accuracy counts exact top-1 matches.
Word similarity benchmarks compare model cosines with human ratings by Spearman rank correlation . RG-65 (Rubenstein and Goodenough, 1965) has 65 noun pairs. WordSim-353 (Finkelstein et al., 2002) has 353 pairs and mixes similarity with relatedness ("cup"–"coffee"). SimLex-999 (Hill, Reichart and Korhonen, 2015) rates strict similarity only, so related but dissimilar pairs score low. Intrinsic scores do not always predict extrinsic performance, which means accuracy on a downstream task such as tagging, NER or classification when the vectors are used as input features. Report both.
Widely used pretrained word vectors
- word2vec GoogleNews
- 300-d skip-gram vectors for 3M words and phrases, trained on about 100B words of Google News (released with Mikolov et al., 2013)
- GloVe 6B
- Wikipedia 2014 + Gigaword 5, 6B tokens, 400K uncased vocabulary,
- GloVe 42B / 840B
- Common Crawl, 300-d; 1.9M uncased and 2.2M cased vocabulary
- GloVe Twitter
- 2B tweets, 27B tokens, 1.2M vocabulary,
- fastText crawl-300d-2M
- 2M word vectors, 300-d, trained on 600B tokens of Common Crawl (Mikolov et al., 2018)
- Typical training
- window 2–10, = 100–300, rare words (count below 5) dropped, frequent words subsampled
Worked example: why PMI beats raw counts
Three words and three contexts with made-up counts. "the" is the frequent, uninformative context.
| word | the | pet | drive | row sum |
|---|---|---|---|---|
| cat | 20 | 6 | 0 | 26 |
| dog | 22 | 5 | 1 | 28 |
| car | 25 | 0 | 6 | 31 |
| column sum | 67 | 11 | 7 | 85 |
PMI, PPMI and cosine (computed in numpy)
- 1Raw-count cosines: but also . The shared "the" column dominates both.
- 2. Cats appear with "pet" 1.8 times more than chance.
- 3. "the" is independent of "cat", so PPMI sets it to 0. The same happens for dog (), and car keeps only .
- 4; , clipped to 0.
- 5PPMI rows: cat , dog , car . Now and .
- 6With context smoothing : instead of , so falls further to . Rare contexts are penalised.
1import numpy as np23corpus = """the king wore his crown . the queen wore her crown . the king ruled his land .4the queen ruled her land . the man worked his farm . the woman worked her farm .5the man walked his dog . the woman walked her dog . the dog ate meat . the cat ate fish""".split()67vocab = sorted(set(corpus) - {"."})8idx = {w: i for i, w in enumerate(vocab)}910def cooccurrence(tokens, window=2):11 C = np.zeros((len(vocab), len(vocab)))12 for i, w in enumerate(tokens):13 if w == ".":14 continue15 for j in range(max(0, i - window), min(len(tokens), i + window + 1)):16 if j != i and tokens[j] != ".":17 C[idx[w], idx[tokens[j]]] += 118 return C1920def ppmi(C, alpha=0.75):21 """PPMI with context-distribution smoothing (Levy, Goldberg & Dagan 2015)."""22 pw = C.sum(1, keepdims=True) / C.sum()23 pc = C.sum(0, keepdims=True) ** alpha24 pc /= pc.sum()25 with np.errstate(divide="ignore"):26 pmi = np.log(C / C.sum() / (pw * pc))27 return np.maximum(pmi, 0) # log 0 = -inf is clipped to 0 as well2829C = cooccurrence(corpus, window=1)30M = ppmi(C)31U, S, Vt = np.linalg.svd(M)32k = 833E = U[:, :k] * np.sqrt(S[:k]) # W = U_k Σ_k^p with p = 0.534E /= np.linalg.norm(E, axis=1, keepdims=True)3536def neighbours(w, n=3):37 sims = E @ E[idx[w]]38 return [(vocab[j], round(float(sims[j]), 3)) for j in np.argsort(-sims) if vocab[j] != w][:n]3940def analogy(a, b, c): # a : b :: c : ? (3CosAdd)41 q = E[idx[b]] - E[idx[a]] + E[idx[c]]42 sims = E @ q / np.linalg.norm(q)43 return next(vocab[j] for j in np.argsort(-sims) if vocab[j] not in (a, b, c))4445print(len(vocab), "words; top singular values", np.round(S[:4], 2)) # 19 words; [5.76 5.25 4.88 4.5 ]46print("king:", neighbours("king")) # [('queen', 1.0), ('cat', 0.31), ('dog', 0.185)]47print("man : king :: woman :", analogy("man", "king", "woman")) # queenUse when
- +Dense, low-dimensional, and cheap to look up: one row of a table per word
- +Learned from raw text with no labels; transfer what is known about one word to similar words
- +Simple geometry: cosine for similarity, vector offsets for many relations
- +Count-based and predictive methods are well understood and closely related
Avoid when
- −One vector per word type: no polysemy, no dependence on the sentence
- −Out-of-vocabulary words get no vector (FastText's subwords partly fix this)
- −Antonyms share contexts and end up close ('hot' vs 'cold')
- −Inherit corpus biases; intrinsic benchmarks track downstream accuracy only loosely
Where it's used
Feature input
Initialising the embedding layer of CNN/RNN taggers and classifiers before contextual models
Lexical tasks
Synonym discovery, query expansion, lexicon induction and cross-lingual dictionary mapping
Analysis
Measuring semantic change over time and auditing social biases in corpora
LLM input layer
Every model in Transformer-based language models starts with a learned token embedding of the same form
Check yourself
Why does PPMI usually give better word vectors than raw co-occurrence counts?
Check yourself
Levy and Goldberg (2014) showed that skip-gram with k negative samples implicitly factorises which matrix?
Remember
- ✓A word embedding is a row of a matrix, with –. Lookup is , and similarity is cosine.
- ✓Distributional hypothesis: words in similar contexts have similar meanings (Harris 1954; Firth 1957).
- ✓Count-based: window counts → → truncated SVD .
- ✓Predictive: Word2Vec (SGNS) and GloVe. SGNS implicitly factorises , and with matched hyperparameters (, , windows) count methods perform comparably.
- ✓Evaluate with similarity benchmarks (Spearman on WordSim-353, SimLex-999) and with 3CosAdd analogies that exclude the query words. Check downstream tasks too.
- ✓Limits: one vector per word type, no out-of-vocabulary words, antonyms close together, corpus biases. Contextual embeddings address the first.