Definition
GloVe (Global Vectors) is the word-embedding model of Pennington, Socher and Manning, "GloVe: Global Vectors for Word Representation" (EMNLP 2014). It first counts how often each word occurs near each word in the whole corpus, giving a matrix . It then learns a word vector , a context vector and two biases, so that for every non-zero entry. A weighted least-squares loss with weight discounts rare, noisy pairs and caps very frequent ones. The final embedding of word is .
In 2014 there were two families of word vectors. Global matrix factorisation methods, such as LSA and PPMI + SVD, use corpus-wide statistics in one pass. They do well on word similarity but poorly on analogies, and frequent words dominate the factorisation. Local context window methods, such as skip-gram in Word2Vec, scan windows one at a time. They produce strong analogy structure but use the global statistics only implicitly, and they spend most updates on repeated pairs. Pennington et al. set out to design a model whose vector arithmetic is derived from co-occurrence statistics. It is trained on the counts directly, so each distinct pair is visited once per pass, not once per occurrence.
The ratio argument
Let be the count of word in the context of word , , and . A single probability mixes meaning with frequency. The ratio for a probe word isolates what distinguishes from . The paper's example uses ice and steam on a 6-billion-token corpus:
| probe k | P(k | ice) | P(k | steam) | P(k | ice) / P(k | steam) |
|---|---|---|---|
| solid | 1.9 × 10⁻⁴ | 2.2 × 10⁻⁵ | 8.9 |
| gas | 6.6 × 10⁻⁵ | 7.8 × 10⁻⁴ | 8.5 × 10⁻² |
| water | 3.0 × 10⁻³ | 2.2 × 10⁻³ | 1.36 |
| fashion | 1.7 × 10⁻⁵ | 1.8 × 10⁻⁵ | 0.96 |
"solid" relates to ice but not steam, so the ratio is large. "gas" relates to steam, so it is small. "water" relates to both and "fashion" to neither, so their ratios are near 1. The ratio cancels the noise from words like "water" and "fashion" and keeps the relevant contrast. GloVe makes vector differences carry exactly this quantity.
- the information distinguishing from should live in their vector difference
- a dot product keeps a function of a scalar and does not mix vector dimensions
- (a homomorphism from to ) forces the exponential
- does not depend on , so it is absorbed into ; restores the symmetry between words and contexts
Weighted least squares
- zero counts drop out, so never appears; only the non-zero entries of are visited
- rare pairs are noisy and get small weight: ,
- very frequent pairs ("of the") are not allowed to dominate
- gave "a modest improvement" over ; the authors note it is the same power Word2Vec found for its noise distribution
The paper connects to skip-gram. Sum skip-gram's softmax cross-entropy over the corpus and group identical pairs. The global objective becomes , a cross-entropy between the true context distribution and the model's , weighted by . Cross-entropy needs the normalised , which is expensive, and it gives large weight to unlikely events. Replacing it with a squared error on the unnormalised logs, , and replacing the weight with the general gives GloVe. In this sense GloVe and skip-gram fit the same statistics with different losses and weights. Word2Vec's SGNS fits , and GloVe fits minus two biases, which is PMI up to per-word constants.
Cost scales with the number of non-zero entries , not with corpus length. Under a power-law model of co-occurrence counts, the paper estimates for a corpus of tokens. Window methods scale as . Counting is one pass, after which training iterates over a fixed list of pairs.
Setup in the paper
- Corpora
- Wikipedia 2010 (1B tokens), Wikipedia 2014 (1.6B), Gigaword 5 (4.3B), Wikipedia 2014 + Gigaword 5 (6B), Common Crawl (42B); tokenised and lowercased
- Vocabulary
- the 400,000 most frequent words; about 2M for the 42B corpus
- Counting
- 10 words to the left and 10 to the right; a pair words apart contributes
- Weighting
- , in all experiments
- Optimiser
- AdaGrad on randomly ordered non-zero entries, initial learning rate 0.05; 50 iterations for , 100 otherwise
- Output
- , which typically gave a small boost over alone
- Result
- 75% on the Word2Vec analogy task (abstract); the released 6B vectors come in
Worked example: weights and one update
| X_ij | 1 | 5 | 10 | 20 | 50 | 100 | 400 |
|---|---|---|---|---|---|---|---|
| f(X_ij) | 0.0316 | 0.1057 | 0.1778 | 0.2991 | 0.5946 | 1 | 1 |
One pair, X_ij = 20, d = 2
- 1Parameters , , , . Prediction , target , residual .
- 2Weight . Loss term . A pair seen once would get weight , about a tenth as much.
- 3Gradients: , so , , and for each bias.
- 4First AdaGrad step as in glove.c, which drops the factor 2 and starts the accumulators at 1: . Then , , and .
- 5New prediction , residual . The accumulators grow by terms, so later steps on this pair get smaller.
1import numpy as np23rng = np.random.default_rng(0)4topics = [5 ["ice is a cold solid", "the ice is solid", "ice melts into water"],6 ["steam is a hot gas", "the steam is gas", "water boils into steam"],7 ["water is a liquid", "the water is cold", "the water is hot"],8 ["the fashion is a style", "fashion is the style"],9]10stream = []11for _ in range(400): # paragraphs of 3-5 sentences on one topic12 T = topics[rng.integers(len(topics))]13 for _ in range(rng.integers(3, 6)):14 stream += T[rng.integers(len(T))].split()1516vocab = sorted(set(stream))17idx = {w: i for i, w in enumerate(vocab)}18V = len(vocab)1920# 1. co-occurrence counts: symmetric window of 10, a pair at distance d adds 1/d21X = np.zeros((V, V))22ids = [idx[w] for w in stream]23for i, wi in enumerate(ids):24 for dist in range(1, 11):25 if i + dist < len(ids):26 X[wi, ids[i + dist]] += 1 / dist27 X[ids[i + dist], wi] += 1 / dist2829# 2. weighted least squares on log X over the non-zero entries, AdaGrad as in glove.c30x_max, alpha, eta, d, iters = 100.0, 0.75, 0.05, 10, 5031I, J = np.nonzero(X)32logx = np.log(X[I, J])33f = np.minimum(1.0, (X[I, J] / x_max) ** alpha)34W, Wt = (rng.random((V, d)) - 0.5) / d, (rng.random((V, d)) - 0.5) / d35b, bt = (rng.random(V) - 0.5) / d, (rng.random(V) - 0.5) / d36gW, gWt, gb, gbt = np.ones((V, d)), np.ones((V, d)), np.ones(V), np.ones(V)37for it in range(iters):38 cost = 0.039 for n in rng.permutation(len(I)):40 i, j = I[n], J[n]41 diff = W[i] @ Wt[j] + b[i] + bt[j] - logx[n]42 cost += 0.5 * f[n] * diff ** 243 g = eta * f[n] * diff # step size folded in, as in glove.c44 g1, g2 = g * Wt[j], g * W[i]45 W[i] -= g1 / np.sqrt(gW[i]); Wt[j] -= g2 / np.sqrt(gWt[j])46 gW[i] += g1 ** 2; gWt[j] += g2 ** 247 b[i] -= g / np.sqrt(gb[i]); bt[j] -= g / np.sqrt(gbt[j])48 gb[i] += g ** 2; gbt[j] += g ** 249 if it in (0, iters - 1):50 print(f"iter {it + 1}: J/2 = {cost:.2f}") # iter 1: J/2 = 970.13 ... iter 50: J/2 = 3.275152# 3. the ratio argument: P(k|ice) / P(k|steam) in the data and in the fitted model53P = X / X.sum(1, keepdims=True)54Xhat = np.exp(W @ Wt.T + b[:, None] + bt[None, :])55Phat = Xhat / X.sum(1, keepdims=True)56for k in ("solid", "gas", "water", "fashion"):57 i, j, q = idx["ice"], idx["steam"], idx[k]58 print(f"{k:8s} data {P[i, q] / P[j, q]:7.3f} model {Phat[i, q] / Phat[j, q]:7.3f}")59# solid data 24.655 model 33.79260# gas data 0.115 model 0.09161# water data 1.008 model 1.06062# fashion data 1.314 model 1.56763E = W + Wt # the paper's final vectorsUse when
- +Uses global statistics directly; training cost scales with non-zero counts, not corpus length
- +Vector differences are derived to encode co-occurrence ratios, giving strong analogy structure
- +Embarrassingly parallel over a fixed list of pairs; few, stable hyperparameters
- +Widely available pretrained vectors (6B, 42B, 840B, Twitter)
Avoid when
- −Must build and store the co-occurrence matrix: memory grows with vocabulary and window
- −Static, one vector per word type; no out-of-vocabulary handling
- −x_max is an absolute count and must be rescaled for small corpora
- −With tuned hyperparameters, SGNS and PPMI + SVD often match it (Levy et al., 2015)
Check yourself
Why does GloVe model ratios P(k|i) / P(k|j) rather than single probabilities P(k|i)?
Check yourself
With x_max = 100 and α = 3/4, how much weight does a pair with X_ij = 10 get, compared with one at X_ij = 100?
Remember
- ✓Objective: over the non-zero entries of the co-occurrence matrix.
- ✓Weighting: with and ; removes zeros, and the cap stops frequent pairs dominating.
- ✓Derivation: making encode forces and .
- ✓Training: window with weights, AdaGrad at 0.05, 50 or 100 iterations, and final vectors .
- ✓GloVe and skip-gram fit the same co-occurrence statistics with different losses; with tuned settings, neither consistently wins.