# GloVe

> GloVe (Pennington, Socher & Manning, EMNLP 2014) fits w_iᵀw̃_j + b_i + b̃_j to log X_ij over the non-zero entries of a global co-occurrence matrix, weighting each pair by f(X_ij) = min(1, (X_ij/100)^¾), so vector differences encode ratios of co-occurrence probabilities.

Source: https://www.mlfoundry.in/learn/glove  
Section: Deep Learning › Large Language Models (LLMs)  
Level: core · 14 min read · updated 2026-10-13

**GloVe:** (Global Vectors) is the word-embedding model of Pennington, Socher and Manning, "GloVe: Global Vectors for Word Representation" (EMNLP 2014). It first counts how often each word $j$ occurs near each word $i$ in the whole corpus, giving a matrix $X$. It then learns a word vector $w_i$, a context vector $\tilde w_j$ and two biases, so that $w_i^\top\tilde w_j+b_i+\tilde b_j\approx\log X_{ij}$ for every non-zero entry. A **weighted least-squares** loss with weight $f(X_{ij})$ discounts rare, noisy pairs and caps very frequent ones. The final embedding of word $i$ is $w_i+\tilde w_i$.

In 2014 there were two families of word vectors. **Global matrix factorisation** methods, such as LSA and PPMI + SVD, use corpus-wide statistics in one pass. They do well on word similarity but poorly on analogies, and frequent words dominate the factorisation. **Local context window** methods, such as skip-gram in **Word2Vec**, scan windows one at a time. They produce strong analogy structure but use the global statistics only implicitly, and they spend most updates on repeated pairs. Pennington et al. set out to design a model whose vector arithmetic is *derived* from co-occurrence statistics. It is trained on the counts directly, so each distinct pair is visited once per pass, not once per occurrence.

## The ratio argument

Let $X_{ik}$ be the count of word $k$ in the context of word $i$, $X_i=\sum_kX_{ik}$, and $P_{ik}=X_{ik}/X_i$. A single probability mixes meaning with frequency. The *ratio* $P_{ik}/P_{jk}$ for a probe word $k$ isolates what distinguishes $i$ from $j$. The paper's example uses $i=$ ice and $j=$ steam on a 6-billion-token corpus:

Co-occurrence probabilities from Pennington et al. (2014), Table 1

| probe k | P(k \| ice) | P(k \| steam) | P(k \| ice) / P(k \| steam) |
| --- | --- | --- | --- |
| solid | 1.9 × 10⁻⁴ | 2.2 × 10⁻⁵ | 8.9 |
| gas | 6.6 × 10⁻⁵ | 7.8 × 10⁻⁴ | 8.5 × 10⁻² |
| water | 3.0 × 10⁻³ | 2.2 × 10⁻³ | 1.36 |
| fashion | 1.7 × 10⁻⁵ | 1.8 × 10⁻⁵ | 0.96 |

"solid" relates to ice but not steam, so the ratio is large. "gas" relates to steam, so it is small. "water" relates to both and "fashion" to neither, so their ratios are near 1. The ratio cancels the noise from words like "water" and "fashion" and keeps the relevant contrast. GloVe makes vector differences carry exactly this quantity.

$$
F\big((w_i-w_j)^\top\tilde w_k\big)=\frac{P_{ik}}{P_{jk}}\ \Rightarrow\ F=\exp,\ \ w_i^\top\tilde w_k=\log P_{ik}=\log X_{ik}-\log X_i\ \Rightarrow\ w_i^\top\tilde w_k+b_i+\tilde b_k=\log X_{ik}
$$

- $w_i-w_j$: the information distinguishing $i$ from $j$ should live in their vector difference
- $(\cdot)^\top\tilde w_k$: a dot product keeps $F$ a function of a scalar and does not mix vector dimensions
- $F=\exp$: $F(a-b)=F(a)/F(b)$ (a homomorphism from $(\mathbb R,+)$ to $(\mathbb R_{>0},\times)$) forces the exponential
- $b_i,\ \tilde b_k$: $\log X_i$ does not depend on $k$, so it is absorbed into $b_i$; $\tilde b_k$ restores the symmetry between words and contexts

> Figure: The GloVe pipeline. One pass over the corpus builds X with a symmetric 10-word window, where a pair d words apart adds 1/d. The zeros are dropped, and every non-zero entry becomes one regression target log X_ij. The weighted least-squares objective is minimised with AdaGrad (initial rate 0.05, 50 iterations below 300 dimensions and 100 otherwise), and the released vectors are W + W̃. Bottom left: the weighting function with x_max = 100 and α = 3/4, which gives f(50) = 0.59 and f = 1 from 100 up. Bottom right: the paper's ice/steam probe ratios that motivate the model.

## Weighted least squares

$$
J=\sum_{i,j=1}^{V}f(X_{ij})\,\big(w_i^\top\tilde w_j+b_i+\tilde b_j-\log X_{ij}\big)^2,\qquad f(x)=\begin{cases}(x/x_{\max})^{\alpha}&x<x_{\max}\\1&\text{otherwise}\end{cases},\quad x_{\max}=100,\ \alpha=\tfrac34
$$

- $f(0)=0$: zero counts drop out, so $\log0$ never appears; only the non-zero entries of $X$ are visited
- $(x/x_{\max})^{3/4}$: rare pairs are noisy and get small weight: $f(1)=0.032$, $f(10)=0.178$
- $\text{cap at }1$: very frequent pairs ("of the") are not allowed to dominate
- $\alpha=3/4$: gave "a modest improvement" over $\alpha=1$; the authors note it is the same power Word2Vec found for its noise distribution

The paper connects $J$ to skip-gram. Sum skip-gram's softmax cross-entropy over the corpus and group identical pairs. The global objective becomes $-\sum_iX_i\sum_jP_{ij}\log Q_{ij}$, a cross-entropy between the true context distribution $P_i$ and the model's $Q_i$, weighted by $X_i$. Cross-entropy needs the normalised $Q$, which is expensive, and it gives large weight to unlikely events. Replacing it with a squared error on the unnormalised logs, $(w_i^\top\tilde w_j-\log X_{ij})^2$, and replacing the weight $X_i$ with the general $f(X_{ij})$ gives GloVe. In this sense GloVe and skip-gram fit the same statistics with different losses and weights. **Word2Vec**'s SGNS fits $\mathrm{PMI}-\log k$, and GloVe fits $\log X_{ij}$ minus two biases, which is PMI up to per-word constants.

Cost scales with the number of non-zero entries $|X|$, not with corpus length. Under a power-law model of co-occurrence counts, the paper estimates $|X|=O(|C|^{0.8})$ for a corpus of $|C|$ tokens. Window methods scale as $O(|C|)$. Counting is one pass, after which training iterates over a fixed list of pairs.

**Setup in the paper**

- **Corpora:** Wikipedia 2010 (1B tokens), Wikipedia 2014 (1.6B), Gigaword 5 (4.3B), Wikipedia 2014 + Gigaword 5 (6B), Common Crawl (42B); tokenised and lowercased
- **Vocabulary:** the 400,000 most frequent words; about 2M for the 42B corpus
- **Counting:** 10 words to the left and 10 to the right; a pair $d$ words apart contributes $1/d$
- **Weighting:** $x_{\max}=100$, $\alpha=3/4$ in all experiments
- **Optimiser:** AdaGrad on randomly ordered non-zero entries, initial learning rate 0.05; 50 iterations for $d<300$, 100 otherwise
- **Output:** $W+\tilde W$, which typically gave a small boost over $W$ alone
- **Result:** 75% on the Word2Vec analogy task (abstract); the released 6B vectors come in $d=50,100,200,300$

> Figure: GloVe trained live with AdaGrad (step 0.05, accumulators started at 1, as in glove.c) on a toy stream of 5,106 tokens. The paragraphs are about ice, steam, water or fashion, and the window is 10 with 1/d weighting. That gives 256 non-zero counts, from 1.78 to 727. At the defaults (d = 5, x_max = 100, α = 0.75, 100 iterations), 102 pairs are at the cap f = 1, and J falls from 3815 at initialisation to 82.63 after 10 iterations, 13.15 after 50 and 10.68 after 100. The fit panel shows the weighting at work. The log-space RMSE is 0.725 on the 33 rare pairs (X < 10), 0.327 on the 121 middle pairs and 0.203 on the 102 frequent ones. The probe table reproduces the paper's pattern: data ratios 13.503 (solid), 0.099 (gas), 0.965 (water) and 0.971 (fashion), against model ratios 11.220, 0.104, 0.774 and 0.955. ice's nearest words in W + W̃ are cold 0.973, solid 0.948 and melts 0.829; steam's are hot 0.931, gas 0.926 and boils 0.697. Switch to uniform weights and the trade-off reverses: rare-pair RMSE 0.171, frequent 0.313. Setting α = 0 gives the same result, since f is then 1 for every observed pair. With d = 10, J reaches 2.42, the frequent-pair RMSE 0.063, and the model ratios 15.798, 0.095, 1.073 and 1.006. With d = 2, J is 24.53, yet the ratios still come out at 11.521, 0.095, 0.883 and 0.969.

## Worked example: weights and one update

f(x) with x_max = 100, α = 3/4 (computed in numpy)

| X_ij | 1 | 5 | 10 | 20 | 50 | 100 | 400 |
| --- | --- | --- | --- | --- | --- | --- | --- |
| f(X_ij) | 0.0316 | 0.1057 | 0.1778 | 0.2991 | 0.5946 | 1 | 1 |

**One pair, X_ij = 20, d = 2**

1. Parameters $w_i=(0.3,-0.2)$, $\tilde w_j=(0.5,0.4)$, $b_i=0.1$, $\tilde b_j=0.2$. Prediction $0.15-0.08+0.1+0.2=0.37$, target $\log20=2.9957$, residual $-2.6257$.
2. Weight $f(20)=0.2\,^{0.75}=0.2991$. Loss term $0.2991\cdot2.6257^2=2.0619$. A pair seen once would get weight $0.0316$, about a tenth as much.
3. Gradients: $\partial J/\partial(\text{pred})=2\cdot0.2991\cdot(-2.6257)=-1.5706$, so $\partial J/\partial w_i=-1.5706\,\tilde w_j=(-0.7853,-0.6282)$, $\partial J/\partial\tilde w_j=-1.5706\,w_i=(-0.4712,0.3141)$, and $-1.5706$ for each bias.
4. First AdaGrad step as in glove.c, which drops the factor 2 and starts the accumulators at 1: $g=0.05\cdot0.2991\cdot(-2.6257)=-0.0393$. Then $w_i\to(0.3196,-0.1843)$, $\tilde w_j\to(0.5118,0.3921)$, $b_i\to0.1393$ and $\tilde b_j\to0.2393$.
5. New prediction $0.4698$, residual $-2.5259$. The accumulators grow by $g^2$ terms, so later steps on this pair get smaller.

```python
import numpy as np

rng = np.random.default_rng(0)
topics = [
    ["ice is a cold solid", "the ice is solid", "ice melts into water"],
    ["steam is a hot gas", "the steam is gas", "water boils into steam"],
    ["water is a liquid", "the water is cold", "the water is hot"],
    ["the fashion is a style", "fashion is the style"],
]
stream = []
for _ in range(400):                                  # paragraphs of 3-5 sentences on one topic
    T = topics[rng.integers(len(topics))]
    for _ in range(rng.integers(3, 6)):
        stream += T[rng.integers(len(T))].split()

vocab = sorted(set(stream))
idx = {w: i for i, w in enumerate(vocab)}
V = len(vocab)

# 1. co-occurrence counts: symmetric window of 10, a pair at distance d adds 1/d
X = np.zeros((V, V))
ids = [idx[w] for w in stream]
for i, wi in enumerate(ids):
    for dist in range(1, 11):
        if i + dist < len(ids):
            X[wi, ids[i + dist]] += 1 / dist
            X[ids[i + dist], wi] += 1 / dist

# 2. weighted least squares on log X over the non-zero entries, AdaGrad as in glove.c
x_max, alpha, eta, d, iters = 100.0, 0.75, 0.05, 10, 50
I, J = np.nonzero(X)
logx = np.log(X[I, J])
f = np.minimum(1.0, (X[I, J] / x_max) ** alpha)
W, Wt = (rng.random((V, d)) - 0.5) / d, (rng.random((V, d)) - 0.5) / d
b, bt = (rng.random(V) - 0.5) / d, (rng.random(V) - 0.5) / d
gW, gWt, gb, gbt = np.ones((V, d)), np.ones((V, d)), np.ones(V), np.ones(V)
for it in range(iters):
    cost = 0.0
    for n in rng.permutation(len(I)):
        i, j = I[n], J[n]
        diff = W[i] @ Wt[j] + b[i] + bt[j] - logx[n]
        cost += 0.5 * f[n] * diff ** 2
        g = eta * f[n] * diff                         # step size folded in, as in glove.c
        g1, g2 = g * Wt[j], g * W[i]
        W[i] -= g1 / np.sqrt(gW[i]); Wt[j] -= g2 / np.sqrt(gWt[j])
        gW[i] += g1 ** 2;            gWt[j] += g2 ** 2
        b[i] -= g / np.sqrt(gb[i]);  bt[j] -= g / np.sqrt(gbt[j])
        gb[i] += g ** 2;             gbt[j] += g ** 2
    if it in (0, iters - 1):
        print(f"iter {it + 1}: J/2 = {cost:.2f}")      # iter 1: J/2 = 970.13 ... iter 50: J/2 = 3.27

# 3. the ratio argument: P(k|ice) / P(k|steam) in the data and in the fitted model
P = X / X.sum(1, keepdims=True)
Xhat = np.exp(W @ Wt.T + b[:, None] + bt[None, :])
Phat = Xhat / X.sum(1, keepdims=True)
for k in ("solid", "gas", "water", "fashion"):
    i, j, q = idx["ice"], idx["steam"], idx[k]
    print(f"{k:8s} data {P[i, q] / P[j, q]:7.3f}   model {Phat[i, q] / Phat[j, q]:7.3f}")
# solid    data  24.655   model  33.792
# gas      data   0.115   model   0.091
# water    data   1.008   model   1.060
# fashion  data   1.314   model   1.567
E = W + Wt                                            # the paper's final vectors
```

GloVe from scratch in numpy: 1/d-weighted window counts, f(x), AdaGrad as in glove.c, and the probe-ratio check (run; outputs in comments)

> **Do not smooth the zeros:** It is tempting to add 1 to every count so that $\log X_{ij}$ is defined everywhere. That turns a sparse problem with $|X|$ entries into a dense one with $V^2$ entries, and it trains the model to predict $\log1=0$ for millions of unrelated pairs. GloVe's $f(0)=0$ removes zeros by construction. Keep $X$ sparse and iterate only over its non-zero entries.

> **Weighting chooses where to be accurate:** With fewer dimensions than words, the fit must be wrong somewhere. $f$ decides where. In the figure the weighted fit gets frequent pairs within RMSE 0.203 in log space and leaves rare ones at 0.725. Uniform weights invert this (0.313 and 0.171). Frequent counts are reliable, so fitting them tightly gives better vectors. Note that $x_{\max}=100$ is relative to corpus size: on a small corpus most pairs fall below it.

> **Why two vector tables:** $X$ is symmetric for a symmetric window, so $W$ and $\tilde W$ play the same role and differ only through random initialisation. Summing them averages out some of that noise. The authors report that $W+\tilde W$ typically gives a small boost. Levy, Goldberg and Dagan (2015) found that the same trick, $w+c$, can help SGNS and SVD vectors too.

**Strengths**

- Uses global statistics directly; training cost scales with non-zero counts, not corpus length
- Vector differences are derived to encode co-occurrence ratios, giving strong analogy structure
- Embarrassingly parallel over a fixed list of pairs; few, stable hyperparameters
- Widely available pretrained vectors (6B, 42B, 840B, Twitter)

**Limitations**

- Must build and store the co-occurrence matrix: memory grows with vocabulary and window
- Static, one vector per word type; no out-of-vocabulary handling
- x_max is an absolute count and must be rescaled for small corpora
- With tuned hyperparameters, SGNS and PPMI + SVD often match it (Levy et al., 2015)

**Question:** Why does GloVe model ratios P(k|i) / P(k|j) rather than single probabilities P(k|i)?

- A. Ratios are always between 0 and 1
- B. A ratio cancels context words that relate to both or neither word (ratio ≈ 1) and isolates the ones that distinguish i from j
- C. Single probabilities cannot be estimated from counts
- D. Ratios make the matrix X symmetric

**Answer:** B. A ratio cancels context words that relate to both or neither word (ratio ≈ 1) and isolates the ones that distinguish i from j — $P(\text{water}\mid\text{ice})$ is large, but so is $P(\text{water}\mid\text{steam})$, so it says nothing about how ice differs from steam. The ratio is near 1 for such words (1.36 for water, 0.96 for fashion) and far from 1 only for discriminative probes (8.9 for solid, 0.085 for gas). The model makes $(w_i-w_j)^\top\tilde w_k$ equal the log of that ratio.

**Question:** With x_max = 100 and α = 3/4, how much weight does a pair with X_ij = 10 get, compared with one at X_ij = 100?

- A. 0.100
- B. 0.178
- C. 0.316
- D. 1, because both are non-zero

**Answer:** B. 0.178 — $f(10)=(10/100)^{0.75}=0.1^{0.75}=0.178$, and $f(100)=1$. With $\alpha=1$ it would be $0.1$. The $3/4$ power discounts rare pairs less harshly than a linear weight.

**Key points**

- Objective: $J=\sum_{ij}f(X_{ij})\,(w_i^\top\tilde w_j+b_i+\tilde b_j-\log X_{ij})^2$ over the non-zero entries of the co-occurrence matrix.
- Weighting: $f(x)=\min(1,(x/x_{\max})^{\alpha})$ with $x_{\max}=100$ and $\alpha=3/4$; $f(0)=0$ removes zeros, and the cap stops frequent pairs dominating.
- Derivation: making $(w_i-w_j)^\top\tilde w_k$ encode $\log(P_{ik}/P_{jk})$ forces $F=\exp$ and $w_i^\top\tilde w_k+b_i+\tilde b_k=\log X_{ik}$.
- Training: window $\pm10$ with $1/d$ weights, AdaGrad at 0.05, 50 or 100 iterations, and final vectors $W+\tilde W$.
- GloVe and skip-gram fit the same co-occurrence statistics with different losses; with tuned settings, neither consistently wins.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
