GloVe

GloVe (Pennington, Socher & Manning, EMNLP 2014) fits w_iᵀw̃_j + b_i + b̃_j to log X_ij over the non-zero entries of a global co-occurrence matrix, weighting each pair by f(X_ij) = min(1, (X_ij/100)^¾), so vector differences encode ratios of co-occurrence probabilities.

14 min read833 wordscore0 viewsupdated 2026-10-13

Definition

GloVe (Global Vectors) is the word-embedding model of Pennington, Socher and Manning, "GloVe: Global Vectors for Word Representation" (EMNLP 2014). It first counts how often each word jj occurs near each word ii in the whole corpus, giving a matrix XX. It then learns a word vector wiw_i, a context vector w~j\tilde w_j and two biases, so that wi⊤w~j+bi+b~j≈log⁡Xijw_i^\top\tilde w_j+b_i+\tilde b_j\approx\log X_{ij} for every non-zero entry. A weighted least-squares loss with weight f(Xij)f(X_{ij}) discounts rare, noisy pairs and caps very frequent ones. The final embedding of word ii is wi+w~iw_i+\tilde w_i.

In 2014 there were two families of word vectors. Global matrix factorisation methods, such as LSA and PPMI + SVD, use corpus-wide statistics in one pass. They do well on word similarity but poorly on analogies, and frequent words dominate the factorisation. Local context window methods, such as skip-gram in Word2Vec, scan windows one at a time. They produce strong analogy structure but use the global statistics only implicitly, and they spend most updates on repeated pairs. Pennington et al. set out to design a model whose vector arithmetic is derived from co-occurrence statistics. It is trained on the counts directly, so each distinct pair is visited once per pass, not once per occurrence.

The ratio argument

Let XikX_{ik} be the count of word kk in the context of word ii, Xi=∑kXikX_i=\sum_kX_{ik}, and Pik=Xik/XiP_{ik}=X_{ik}/X_i. A single probability mixes meaning with frequency. The ratio Pik/PjkP_{ik}/P_{jk} for a probe word kk isolates what distinguishes ii from jj. The paper's example uses i=i= ice and j=j= steam on a 6-billion-token corpus:

probe kP(k | ice)P(k | steam)P(k | ice) / P(k | steam)
solid1.9 × 10⁻⁴2.2 × 10⁻⁵8.9
gas6.6 × 10⁻⁵7.8 × 10⁻⁴8.5 × 10⁻²
water3.0 × 10⁻³2.2 × 10⁻³1.36
fashion1.7 × 10⁻⁵1.8 × 10⁻⁵0.96
Co-occurrence probabilities from Pennington et al. (2014), Table 1

"solid" relates to ice but not steam, so the ratio is large. "gas" relates to steam, so it is small. "water" relates to both and "fashion" to neither, so their ratios are near 1. The ratio cancels the noise from words like "water" and "fashion" and keeps the relevant contrast. GloVe makes vector differences carry exactly this quantity.

F((wi−wj)⊤w~k)=PikPjk ⇒ F=exp⁡,  wi⊤w~k=log⁡Pik=log⁡Xik−log⁡Xi ⇒ wi⊤w~k+bi+b~k=log⁡XikF\big((w_i-w_j)^\top\tilde w_k\big)=\frac{P_{ik}}{P_{jk}}\ \Rightarrow\ F=\exp,\ \ w_i^\top\tilde w_k=\log P_{ik}=\log X_{ik}-\log X_i\ \Rightarrow\ w_i^\top\tilde w_k+b_i+\tilde b_k=\log X_{ik}
wi−wjw_i-w_j
the information distinguishing ii from jj should live in their vector difference
(⋅)⊤w~k(\cdot)^\top\tilde w_k
a dot product keeps FF a function of a scalar and does not mix vector dimensions
F=exp⁡F=\exp
F(a−b)=F(a)/F(b)F(a-b)=F(a)/F(b) (a homomorphism from (R,+)(\mathbb R,+) to (R>0,×)(\mathbb R_{>0},\times)) forces the exponential
bi, b~kb_i,\ \tilde b_k
log⁡Xi\log X_i does not depend on kk, so it is absorbed into bib_i; b~k\tilde b_k restores the symmetry between words and contexts
GloVe (Pennington, Socher, Manning 2014): one pass to count, then fit the countscorpus1B–42B tokensvocab 400K or 2Mco-occurrence Xwindow ±10 wordspair at distance d: 1/dX ∈ ℝ^(V×V)symmetric, sparsekeep X_ij > 0 onlylog X_ijtarget for eachnon-zero pairJ = Σ_ij f(X_ij) (w_iᵀw̃_j + b_i + b̃_j − log X_ij)²w_i: word vector · w̃_j: context vector · b_i, b̃_j: biasesf down-weights rare pairs and caps frequent onesAdaGradη = 0.0550 or 100iterationsW + W̃summed wordvectorsf(x) = (x / x_max)^α if x < x_max, else 1x_max = 100, α = 3/4, f(0) = 005010015010f(50) = 0.59co-occurrence count xWhy ratios: probe words k, 6B-token corpuskP(k|ice)P(k|steam)ratiosolid1.9×10⁻⁴2.2×10⁻⁵8.9gas6.6×10⁻⁵7.8×10⁻⁴8.5×10⁻²water3.0×10⁻³2.2×10⁻³1.36fashion1.7×10⁻⁵1.8×10⁻⁵0.96large: ice-like · small: steam-like · ≈ 1: neither
fig.The GloVe pipeline. One pass over the corpus builds X with a symmetric 10-word window, where a pair d words apart adds 1/d. The zeros are dropped, and every non-zero entry becomes one regression target log X_ij. The weighted least-squares objective is minimised with AdaGrad (initial rate 0.05, 50 iterations below 300 dimensions and 100 otherwise), and the released vectors are W + W̃. Bottom left: the weighting function with x_max = 100 and α = 3/4, which gives f(50) = 0.59 and f = 1 from 100 up. Bottom right: the paper's ice/steam probe ratios that motivate the model.

Weighted least squares

J=∑i,j=1Vf(Xij) (wi⊤w~j+bi+b~j−log⁡Xij)2,f(x)={(x/xmax⁡)αx<xmax⁡1otherwise,xmax⁡=100, α=34J=\sum_{i,j=1}^{V}f(X_{ij})\,\big(w_i^\top\tilde w_j+b_i+\tilde b_j-\log X_{ij}\big)^2,\qquad f(x)=\begin{cases}(x/x_{\max})^{\alpha}&x<x_{\max}\\1&\text{otherwise}\end{cases},\quad x_{\max}=100,\ \alpha=\tfrac34
f(0)=0f(0)=0
zero counts drop out, so log⁡0\log0 never appears; only the non-zero entries of XX are visited
(x/xmax⁡)3/4(x/x_{\max})^{3/4}
rare pairs are noisy and get small weight: f(1)=0.032f(1)=0.032, f(10)=0.178f(10)=0.178
cap at 1\text{cap at }1
very frequent pairs ("of the") are not allowed to dominate
α=3/4\alpha=3/4
gave "a modest improvement" over α=1\alpha=1; the authors note it is the same power Word2Vec found for its noise distribution

The paper connects JJ to skip-gram. Sum skip-gram's softmax cross-entropy over the corpus and group identical pairs. The global objective becomes −∑iXi∑jPijlog⁡Qij-\sum_iX_i\sum_jP_{ij}\log Q_{ij}, a cross-entropy between the true context distribution PiP_i and the model's QiQ_i, weighted by XiX_i. Cross-entropy needs the normalised QQ, which is expensive, and it gives large weight to unlikely events. Replacing it with a squared error on the unnormalised logs, (wi⊤w~j−log⁡Xij)2(w_i^\top\tilde w_j-\log X_{ij})^2, and replacing the weight XiX_i with the general f(Xij)f(X_{ij}) gives GloVe. In this sense GloVe and skip-gram fit the same statistics with different losses and weights. Word2Vec's SGNS fits PMI−log⁡k\mathrm{PMI}-\log k, and GloVe fits log⁡Xij\log X_{ij} minus two biases, which is PMI up to per-word constants.

Cost scales with the number of non-zero entries ∣X∣|X|, not with corpus length. Under a power-law model of co-occurrence counts, the paper estimates ∣X∣=O(∣C∣0.8)|X|=O(|C|^{0.8}) for a corpus of ∣C∣|C| tokens. Window methods scale as O(∣C∣)O(|C|). Counting is one pass, after which training iterates over a fixed list of pairs.

Setup in the paper

Corpora
Wikipedia 2010 (1B tokens), Wikipedia 2014 (1.6B), Gigaword 5 (4.3B), Wikipedia 2014 + Gigaword 5 (6B), Common Crawl (42B); tokenised and lowercased
Vocabulary
the 400,000 most frequent words; about 2M for the 42B corpus
Counting
10 words to the left and 10 to the right; a pair dd words apart contributes 1/d1/d
Weighting
xmax⁡=100x_{\max}=100, α=3/4\alpha=3/4 in all experiments
Optimiser
AdaGrad on randomly ordered non-zero entries, initial learning rate 0.05; 50 iterations for d<300d<300, 100 otherwise
Output
W+W~W+\tilde W, which typically gave a small boost over WW alone
Result
75% on the Word2Vec analogy task (abstract); the released 6B vectors come in d=50,100,200,300d=50,100,200,300
5106 tokens · 16 words · window 10, 1/distance · 256 X_ij > 0 · d = 5 · iter 100f(x) = min(1, (x/100)^0.75) · 102 pairs at f = 1110100100010X_ij (log scale)w_i·w̃_j + b_i + b̃_j (y) vs log X_ij (x)07-17colour = weight f · dashed y = xThe ratio argument: probe words k for i = ice, j = steamprobe kP(k|ice)P(k|steam)ratio (data)ratio (model)solid0.1520.01113.50311.220gas0.0130.1340.0990.104water0.0770.0800.9650.774fashion0.0120.0120.9710.955model ratio = exp((w_ice − w_steam)·w̃_k + b_ice − b_steam) · X_steam / X_ice; b̃_k cancelsJ = Σ f(X_ij)(w_i·w̃_j + b_i + b̃_j − log X_ij)² = 10.68 (start 3815)RMSE of log X: rare (X < 10, 33) 0.725 · mid (121) 0.327 · frequent (X ≥ 100, 102) 0.203nearest by cosine of W + W̃: ice → cold 0.973, solid 0.948, melts 0.829nearest by cosine of W + W̃: steam → hot 0.931, gas 0.926, boils 0.697AdaGrad, step 0.05, one pass over the 256 pairs per iteration in shuffled order (as in glove.c)weighted: frequent pairs are fitted tightly, rare ones loosely; X = 0 pairs are never in the sum
fig.GloVe trained live with AdaGrad (step 0.05, accumulators started at 1, as in glove.c) on a toy stream of 5,106 tokens. The paragraphs are about ice, steam, water or fashion, and the window is 10 with 1/d weighting. That gives 256 non-zero counts, from 1.78 to 727. At the defaults (d = 5, x_max = 100, α = 0.75, 100 iterations), 102 pairs are at the cap f = 1, and J falls from 3815 at initialisation to 82.63 after 10 iterations, 13.15 after 50 and 10.68 after 100. The fit panel shows the weighting at work. The log-space RMSE is 0.725 on the 33 rare pairs (X < 10), 0.327 on the 121 middle pairs and 0.203 on the 102 frequent ones. The probe table reproduces the paper's pattern: data ratios 13.503 (solid), 0.099 (gas), 0.965 (water) and 0.971 (fashion), against model ratios 11.220, 0.104, 0.774 and 0.955. ice's nearest words in W + W̃ are cold 0.973, solid 0.948 and melts 0.829; steam's are hot 0.931, gas 0.926 and boils 0.697. Switch to uniform weights and the trade-off reverses: rare-pair RMSE 0.171, frequent 0.313. Setting α = 0 gives the same result, since f is then 1 for every observed pair. With d = 10, J reaches 2.42, the frequent-pair RMSE 0.063, and the model ratios 15.798, 0.095, 1.073 and 1.006. With d = 2, J is 24.53, yet the ratios still come out at 11.521, 0.095, 0.883 and 0.969.

Worked example: weights and one update

X_ij15102050100400
f(X_ij)0.03160.10570.17780.29910.594611
f(x) with x_max = 100, α = 3/4 (computed in numpy)

One pair, X_ij = 20, d = 2

  1. 1Parameters wi=(0.3,−0.2)w_i=(0.3,-0.2), w~j=(0.5,0.4)\tilde w_j=(0.5,0.4), bi=0.1b_i=0.1, b~j=0.2\tilde b_j=0.2. Prediction 0.15−0.08+0.1+0.2=0.370.15-0.08+0.1+0.2=0.37, target log⁡20=2.9957\log20=2.9957, residual −2.6257-2.6257.
  2. 2Weight f(20)=0.2 0.75=0.2991f(20)=0.2\,^{0.75}=0.2991. Loss term 0.2991⋅2.62572=2.06190.2991\cdot2.6257^2=2.0619. A pair seen once would get weight 0.03160.0316, about a tenth as much.
  3. 3Gradients: ∂J/∂(pred)=2⋅0.2991⋅(−2.6257)=−1.5706\partial J/\partial(\text{pred})=2\cdot0.2991\cdot(-2.6257)=-1.5706, so ∂J/∂wi=−1.5706 w~j=(−0.7853,−0.6282)\partial J/\partial w_i=-1.5706\,\tilde w_j=(-0.7853,-0.6282), ∂J/∂w~j=−1.5706 wi=(−0.4712,0.3141)\partial J/\partial\tilde w_j=-1.5706\,w_i=(-0.4712,0.3141), and −1.5706-1.5706 for each bias.
  4. 4First AdaGrad step as in glove.c, which drops the factor 2 and starts the accumulators at 1: g=0.05⋅0.2991⋅(−2.6257)=−0.0393g=0.05\cdot0.2991\cdot(-2.6257)=-0.0393. Then wi→(0.3196,−0.1843)w_i\to(0.3196,-0.1843), w~j→(0.5118,0.3921)\tilde w_j\to(0.5118,0.3921), bi→0.1393b_i\to0.1393 and b~j→0.2393\tilde b_j\to0.2393.
  5. 5New prediction 0.46980.4698, residual −2.5259-2.5259. The accumulators grow by g2g^2 terms, so later steps on this pair get smaller.
python
1import numpy as np
2
3rng = np.random.default_rng(0)
4topics = [
5 ["ice is a cold solid", "the ice is solid", "ice melts into water"],
6 ["steam is a hot gas", "the steam is gas", "water boils into steam"],
7 ["water is a liquid", "the water is cold", "the water is hot"],
8 ["the fashion is a style", "fashion is the style"],
9]
10stream = []
11for _ in range(400): # paragraphs of 3-5 sentences on one topic
12 T = topics[rng.integers(len(topics))]
13 for _ in range(rng.integers(3, 6)):
14 stream += T[rng.integers(len(T))].split()
15
16vocab = sorted(set(stream))
17idx = {w: i for i, w in enumerate(vocab)}
18V = len(vocab)
19
20# 1. co-occurrence counts: symmetric window of 10, a pair at distance d adds 1/d
21X = np.zeros((V, V))
22ids = [idx[w] for w in stream]
23for i, wi in enumerate(ids):
24 for dist in range(1, 11):
25 if i + dist < len(ids):
26 X[wi, ids[i + dist]] += 1 / dist
27 X[ids[i + dist], wi] += 1 / dist
28
29# 2. weighted least squares on log X over the non-zero entries, AdaGrad as in glove.c
30x_max, alpha, eta, d, iters = 100.0, 0.75, 0.05, 10, 50
31I, J = np.nonzero(X)
32logx = np.log(X[I, J])
33f = np.minimum(1.0, (X[I, J] / x_max) ** alpha)
34W, Wt = (rng.random((V, d)) - 0.5) / d, (rng.random((V, d)) - 0.5) / d
35b, bt = (rng.random(V) - 0.5) / d, (rng.random(V) - 0.5) / d
36gW, gWt, gb, gbt = np.ones((V, d)), np.ones((V, d)), np.ones(V), np.ones(V)
37for it in range(iters):
38 cost = 0.0
39 for n in rng.permutation(len(I)):
40 i, j = I[n], J[n]
41 diff = W[i] @ Wt[j] + b[i] + bt[j] - logx[n]
42 cost += 0.5 * f[n] * diff ** 2
43 g = eta * f[n] * diff # step size folded in, as in glove.c
44 g1, g2 = g * Wt[j], g * W[i]
45 W[i] -= g1 / np.sqrt(gW[i]); Wt[j] -= g2 / np.sqrt(gWt[j])
46 gW[i] += g1 ** 2; gWt[j] += g2 ** 2
47 b[i] -= g / np.sqrt(gb[i]); bt[j] -= g / np.sqrt(gbt[j])
48 gb[i] += g ** 2; gbt[j] += g ** 2
49 if it in (0, iters - 1):
50 print(f"iter {it + 1}: J/2 = {cost:.2f}") # iter 1: J/2 = 970.13 ... iter 50: J/2 = 3.27
51
52# 3. the ratio argument: P(k|ice) / P(k|steam) in the data and in the fitted model
53P = X / X.sum(1, keepdims=True)
54Xhat = np.exp(W @ Wt.T + b[:, None] + bt[None, :])
55Phat = Xhat / X.sum(1, keepdims=True)
56for k in ("solid", "gas", "water", "fashion"):
57 i, j, q = idx["ice"], idx["steam"], idx[k]
58 print(f"{k:8s} data {P[i, q] / P[j, q]:7.3f} model {Phat[i, q] / Phat[j, q]:7.3f}")
59# solid data 24.655 model 33.792
60# gas data 0.115 model 0.091
61# water data 1.008 model 1.060
62# fashion data 1.314 model 1.567
63E = W + Wt # the paper's final vectors
GloVe from scratch in numpy: 1/d-weighted window counts, f(x), AdaGrad as in glove.c, and the probe-ratio check (run; outputs in comments)

Use when

  • +Uses global statistics directly; training cost scales with non-zero counts, not corpus length
  • +Vector differences are derived to encode co-occurrence ratios, giving strong analogy structure
  • +Embarrassingly parallel over a fixed list of pairs; few, stable hyperparameters
  • +Widely available pretrained vectors (6B, 42B, 840B, Twitter)

Avoid when

  • −Must build and store the co-occurrence matrix: memory grows with vocabulary and window
  • −Static, one vector per word type; no out-of-vocabulary handling
  • −x_max is an absolute count and must be rescaled for small corpora
  • −With tuned hyperparameters, SGNS and PPMI + SVD often match it (Levy et al., 2015)

Check yourself

Why does GloVe model ratios P(k|i) / P(k|j) rather than single probabilities P(k|i)?

Check yourself

With x_max = 100 and α = 3/4, how much weight does a pair with X_ij = 10 get, compared with one at X_ij = 100?

Remember

  • ✓Objective: J=∑ijf(Xij) (wi⊤w~j+bi+b~j−log⁡Xij)2J=\sum_{ij}f(X_{ij})\,(w_i^\top\tilde w_j+b_i+\tilde b_j-\log X_{ij})^2 over the non-zero entries of the co-occurrence matrix.
  • ✓Weighting: f(x)=min⁡(1,(x/xmax⁡)α)f(x)=\min(1,(x/x_{\max})^{\alpha}) with xmax⁡=100x_{\max}=100 and α=3/4\alpha=3/4; f(0)=0f(0)=0 removes zeros, and the cap stops frequent pairs dominating.
  • ✓Derivation: making (wi−wj)⊤w~k(w_i-w_j)^\top\tilde w_k encode log⁡(Pik/Pjk)\log(P_{ik}/P_{jk}) forces F=exp⁡F=\exp and wi⊤w~k+bi+b~k=log⁡Xikw_i^\top\tilde w_k+b_i+\tilde b_k=\log X_{ik}.
  • ✓Training: window ±10\pm10 with 1/d1/d weights, AdaGrad at 0.05, 50 or 100 iterations, and final vectors W+W~W+\tilde W.
  • ✓GloVe and skip-gram fit the same co-occurrence statistics with different losses; with tuned settings, neither consistently wins.
← back to Deep Learning roadmap