# Word2Vec

> Word2Vec (Mikolov et al., 2013) learns word vectors with a log-linear model: skip-gram predicts context words from the centre word, CBOW the reverse. Negative sampling from U(w)^¾ and subsampling of frequent words make it fast, and SGNS implicitly factorises PMI − log k.

Source: https://www.mlfoundry.in/learn/word2vec  
Section: Deep Learning › Large Language Models (LLMs)  
Level: core · 15 min read · updated 2026-10-13

**Word2Vec:** is the family of shallow models from Mikolov, Chen, Corrado and Dean, "Efficient Estimation of Word Representations in Vector Space" (ICLR workshop, 2013), and Mikolov, Sutskever, Chen, Corrado and Dean, "Distributed Representations of Words and Phrases and their Compositionality" (NIPS 2013). It learns two vectors per word, an input vector $v_w$ and an output vector $v'_w$. Training sets them so that $v'_c\cdot v_w$ is large when $c$ appears near $w$. **Skip-gram** predicts each context word from the centre word. **CBOW** (continuous bag of words) predicts the centre word from the average of its context. The second paper added **negative sampling**, **subsampling of frequent words** and phrase vectors. The input vectors are the **word embeddings**.

Neural language models already learned word vectors as a by-product: the feed-forward model of Bengio et al. (2003) and Mikolov's recurrent LM (2010). Their cost was in the hidden layers and in the softmax over the full vocabulary. Mikolov et al. asked how much of the representation quality survives if the model is made almost linear. They removed the non-linear hidden layer entirely. Each prediction is then one dot product per candidate word, and the remaining cost is the softmax. That made it possible to train on billions of words. The first paper reports high-quality vectors from a 1.6-billion-word dataset in under a day, and evaluates on a 6B-token Google News corpus with the vocabulary restricted to the 1M most frequent words.

> Figure: Both architectures on the sentence "the king wore his crown" with centre word wore and a ±2 window. Skip-gram looks up the centre word's row of W and scores each of the four context words separately with σ(u_c · v_w): four training pairs. CBOW averages the four context rows into h and scores the centre word once, so word order inside the window is lost. Hover a panel to highlight it. Below, negative sampling replaces the V-way softmax for one pair (wore, king) with one positive logistic example and k = 5 negatives drawn from the unigram distribution raised to 3/4.

## Skip-gram and CBOW

$$
\frac1T\sum_{t=1}^{T}\ \sum_{-c\le j\le c,\ j\ne0}\log p(w_{t+j}\mid w_t),\qquad p(w_O\mid w_I)=\frac{\exp\!\big(v'^{\top}_{w_O}v_{w_I}\big)}{\sum_{w=1}^{V}\exp\!\big(v'^{\top}_{w}v_{w_I}\big)}
$$

- $T,\ c$: corpus length and window radius; the reference code samples the actual radius uniformly from $1\dots c$ for each word, which weights nearby words more
- $v_w,\ v'_w$: input and output vectors, two separate $V\times d$ tables; $v_w$ is kept as the embedding
- $p(w_O\mid w_I)$: a full softmax over $V$ words: exact, but $O(V)$ per pair, which is too slow for $V=10^6$

$$
\text{CBOW:}\quad h_t=\frac{1}{2c}\sum_{-c\le j\le c,\ j\ne0}v_{w_{t+j}},\qquad \log p(w_t\mid h_t)=v'^{\top}_{w_t}h_t-\log\sum_{w}\exp\!\big(v'^{\top}_{w}h_t\big)
$$

- $h_t$: the mean of the context's input vectors: a bag of words, so order is lost
- $\text{cost}$: one prediction per position instead of $2c$; CBOW is faster, skip-gram is better for rare words

The first paper avoided the full softmax with **hierarchical softmax**. The vocabulary sits at the leaves of a binary Huffman tree, and $p(w\mid w_I)$ is a product of about $\log_2V$ sigmoid decisions along the path to $w$. Frequent words have short codes, so the average cost is even lower. The second paper introduced a simpler alternative that became the default.

## Negative sampling

$$
\log\sigma\!\big(v'^{\top}_{w_O}v_{w_I}\big)+\sum_{i=1}^{k}\mathbb E_{w_i\sim P_n(w)}\Big[\log\sigma\!\big(-v'^{\top}_{w_i}v_{w_I}\big)\Big],\qquad P_n(w)=\frac{U(w)^{3/4}}{Z}
$$

- $\sigma$: logistic sigmoid; the task is now binary: real pair or noise pair?
- $k$: negatives per positive pair: 5–20 for small datasets, 2–5 suffice for large ones (Mikolov et al., 2013b)
- $U(w)^{3/4}/Z$: unigram distribution to the power 3/4, which raises rare words relative to frequent ones; it "outperformed significantly" both the unigram and the uniform distribution
- $\text{vs. NCE}$: a simplification of noise-contrastive estimation (Gutmann and Hyvärinen, 2010): it needs only samples from $P_n$, not their probabilities, and it does not approximate the softmax likelihood

Each pair now costs $k+1$ dot products. With $v=v_{w_I}$, $u_+=v'_{w_O}$ and negatives $u_i$, the gradients are short. The positive output vector moves toward $v$, each negative moves away from it, and $v$ moves toward $u_+$ and away from the $u_i$, each weighted by how wrong the current sigmoid is:

$$
\frac{\partial\mathcal L}{\partial v}=\big(\sigma(u_+^\top v)-1\big)u_+ +\sum_{i=1}^{k}\sigma(u_i^\top v)\,u_i,\qquad \frac{\partial\mathcal L}{\partial u_+}=\big(\sigma(u_+^\top v)-1\big)v,\qquad \frac{\partial\mathcal L}{\partial u_i}=\sigma(u_i^\top v)\,v
$$

- $\mathcal L$: the negated objective for one pair, $-\log\sigma(u_+^\top v)-\sum_i\log\sigma(-u_i^\top v)$
- $\sigma(\cdot)-1$: near 0 once the positive pair is confidently recognised, so easy pairs stop contributing

## Subsampling frequent words

$$
P_{\text{discard}}(w_i)=1-\sqrt{\frac{t}{f(w_i)}},\qquad t\approx10^{-5}
$$

- $f(w_i)$: relative frequency of the word in the corpus
- $t$: threshold; words with $f\le t$ are never dropped. The reference code uses a slightly different keep probability, $(\sqrt{f/t}+1)\,t/f$, with default $t=10^{-3}$

Each occurrence is dropped independently before the windows are formed. Very frequent words such as "the", "in" and "a" carry little information about their neighbours, and they supply most of the training pairs. Removing most of their occurrences gave a speed-up of about 2–10× and improved the vectors of rarer words. Dropping a word also brings more distant words inside the window, so subsampling widens the effective context.

## What SGNS actually optimises

Levy and Goldberg ("Neural Word Embedding as Implicit Matrix Factorization", NeurIPS 2014) summed the SGNS objective over the corpus. The pair $(w,c)$ appears $\#(w,c)$ times as a positive. As a negative it appears in expectation $k\,\#(w)\,P_n(c)$ times. Write $x=w\cdot c$ and treat each $x$ as a free variable, which holds when $d$ is large enough:

$$
\ell(x)=\#(w,c)\log\sigma(x)+k\,\#(w)\,P_n(c)\log\sigma(-x)\ \ \Rightarrow\ \ \ell'(x)=0\iff x=\log\frac{\#(w,c)}{\#(w)\,P_n(c)}-\log k=\mathrm{PMI}(w,c)-\log k
$$

- $\ell'(x)$: $\#(w,c)\,\sigma(-x)-k\,\#(w)P_n(c)\,\sigma(x)=0$, and $\sigma(x)/\sigma(-x)=e^x$
- $P_n(c)$: with $P_n\propto\#(c)$ this is the ordinary PMI; with the $3/4$ power, PMI with a smoothed context distribution
- $-\log k$: more negatives shift every target down; they proposed the count-based **SPPMI** $=\max(\mathrm{PMI}-\log k,0)$

So skip-gram with negative sampling is a weighted, implicit factorisation $WC^\top\approx\mathrm{PMI}-\log k$ of the matrix used in **Word embeddings**. It is never formed, and frequent pairs get more weight. At low $d$ the fit is approximate, and the model chooses which cells to get right. This explains why the $3/4$ power helped. It is context distribution smoothing, and it carries over to PPMI + SVD.

**Settings from the papers and the reference code**

- **Papers:** ICLR 2013 workshop (skip-gram, CBOW, hierarchical softmax, analogy test set) and NIPS 2013 (negative sampling, subsampling, phrases)
- **Analogy set:** 19,544 questions: 8,869 semantic ("Athens : Greece :: Oslo : Norway") and 10,675 syntactic ("big : biggest :: small : smallest")
- **Paper hyperparameters:** $k$ = 5–20 for small corpora, 2–5 for large; $t\approx10^{-5}$; noise $U(w)^{3/4}/Z$
- **word2vec.c defaults:** `-size 100 -window 5 -negative 5 -hs 0 -sample 1e-3 -min-count 5 -iter 5`; learning rate 0.025 for skip-gram, 0.05 for CBOW, decayed linearly to near zero
- **Initialisation:** input vectors uniform in $[-0.5/d,\,0.5/d]$; output vectors zero; noise sampled from a precomputed unigram$^{0.75}$ table
- **GoogleNews vectors:** 300-d, 3M words and phrases, trained on about 100B words
- **gensim:** `Word2Vec(sentences, vector_size=300, window=5, min_count=5, sg=1, negative=5, ns_exponent=0.75, sample=1e-3)`; `sg=0` is CBOW

> Figure: Skip-gram with negative sampling trained live, epoch by epoch, on the toy corpus of the Word embeddings figure (1,860 tokens, 21 words). The defaults are d = 10, window ±2, k = 5 negatives from U^0.75, learning rate 0.05 decaying linearly, and 40 epochs of 5,040 (w, c) pairs. Press play to watch the vectors spread from their tiny random start, with each word's trail shown. The map is a fixed PCA of the final vectors (41% of the variance in 2-D). The mean loss falls from 2.658 after epoch 1 to 1.699 after epoch 40. The right panel tests Levy and Goldberg's result directly. Each of the 126 observed pairs is plotted as its learned w·c against log(#(w,c) / (#(w)·P_n(c))) − log k. At epoch 0 every context vector is zero and r is undefined; then r = 0.350 at epoch 1, 0.773 at 5, 0.839 at 10 and 0.975 at 40. king − man + woman gives queen 0.991. king's neighbours are man 0.653, queen 0.564 and land 0.537. The noise distribution gives the 0.163 instead of its unigram share 0.231, and fish 0.023 instead of 0.017. With d = 2 the matrix cannot fit (r = 0.234), and king's three nearest words all have cosine ≥ 0.999. At d = 5, r = 0.867 and king's nearest word is queen (0.991). CBOW is not a per-pair PMI fit (r = 0.453). Window ±1 gives 2,920 pairs per epoch, r = 0.835, and king → queen 0.994. Uniform noise (U^0) gives r = 0.992. Subsampling with t = 0.05 (scaled up for a tiny corpus) drops 53.5% of the occurrences of the, 18.5% of ate and 13.4% of his. That leaves 3,820 updates in epoch 40, and r is 0.958.

## Worked example: one SGNS update

Take $d=3$, centre vector $v=(0.2,-0.1,0.4)$, the true context's output vector $u_+=(0.3,0.2,0.1)$, $k=2$ negatives $u_1=(-0.2,0.4,0.1)$ and $u_2=(0.1,-0.3,0.2)$, and learning rate $0.5$.

Scores and loss terms before the update (computed in numpy)

| vector | u·v | σ(u·v) | loss term |
| --- | --- | --- | --- |
| positive u₊ | 0.080 | 0.5200 | −log σ(0.08) = 0.6539 |
| negative u₁ | −0.040 | 0.4900 | −log σ(0.04) = 0.6733 |
| negative u₂ | 0.130 | 0.5325 | −log σ(−0.13) = 0.7603 |
| total |  |  | 2.0876 |

**Gradients and the step**

1. $\partial\mathcal L/\partial v=(0.5200-1)\,u_++0.4900\,u_1+0.5325\,u_2=(-0.144,-0.096,-0.048)+(-0.098,0.196,0.049)+(0.053,-0.160,0.106)=(-0.1888,-0.0597,0.1075)$.
2. $\partial\mathcal L/\partial u_+=-0.48\,v=(-0.096,0.048,-0.192)$; $\partial\mathcal L/\partial u_1=0.49\,v=(0.098,-0.049,0.196)$; $\partial\mathcal L/\partial u_2=0.5325\,v=(0.1065,-0.0532,0.2130)$.
3. Step all four vectors with rate 0.5, each using the old values: $v\to(0.2944,-0.0701,0.3463)$ and $u_+\to(0.348,0.176,0.196)$.
4. New scores: positive $0.158$ (up from 0.080), negatives $-0.102$ and $0.065$ (both down). The loss falls from $2.088$ to $1.987$.
5. Noise distribution: three words with counts 900, 90 and 10 have unigram shares $0.900,0.090,0.010$. To the power $3/4$ they become $0.825,0.147,0.028$, almost three times as many negatives for the rarest word.
6. Subsampling with $t=10^{-5}$: a word with $f=0.05$ is dropped with probability $1-\sqrt{2\cdot10^{-4}}=0.986$. With $f=10^{-3}$ the probability is $0.900$, with $f=10^{-4}$ it is $0.684$, and with $f\le10^{-5}$ it is 0.

```python
import numpy as np

rng = np.random.default_rng(0)
people = [("king", "his", 1), ("queen", "her", 1), ("man", "his", 0), ("woman", "her", 0)]
def sentence():
    p, poss, royal = people[rng.integers(4)]
    options = [("wore", "crown"), ("ruled", "land"), ("walked", "dog")] if royal else [("worked", "farm"), ("walked", "dog")]
    verb, obj = options[rng.integers(len(options))]
    return ["the", p, verb, poss, obj]
corpus = [sentence() for _ in range(2000)]

vocab = sorted({w for s in corpus for w in s})
idx = {w: i for i, w in enumerate(vocab)}
counts = np.zeros(len(vocab))
for s in corpus:
    for w in s:
        counts[idx[w]] += 1
freq = counts / counts.sum()

d, window, k, t, epochs, lr0 = 16, 2, 5, 1e-2, 5, 0.025
noise = counts ** 0.75 / (counts ** 0.75).sum()     # P_n(w) ∝ U(w)^(3/4)
p_drop = np.maximum(0, 1 - np.sqrt(t / freq))     # subsampling; t = 1e-5 on real corpora
W = (rng.random((len(vocab), d)) - 0.5) / d       # input vectors (the embeddings)
C = np.zeros((len(vocab), d))                     # output / context vectors
sigmoid = lambda x: 1 / (1 + np.exp(-x))

steps, total = 0, epochs * sum(len(s) for s in corpus)
for epoch in range(epochs):
    for s in corpus:
        ids = [idx[w] for w in s if rng.random() >= p_drop[idx[w]]]
        for i, w in enumerate(ids):
            lr = lr0 * max(1e-4, 1 - steps / total); steps += 1
            b = rng.integers(1, window + 1)        # word2vec's random window shrink
            for j in range(max(0, i - b), min(len(ids), i + b + 1)):
                if j == i:
                    continue
                targets = np.concatenate(([ids[j]], rng.choice(len(vocab), k, p=noise)))
                labels = np.array([1.0] + [0.0] * k)
                g = lr * (labels - sigmoid(C[targets] @ W[w]))   # gradient of log σ(±u·v)
                dW = g @ C[targets]
                np.add.at(C, targets, np.outer(g, W[w]))
                W[w] += dW

E = W / np.linalg.norm(W, axis=1, keepdims=True)
def nearest(w):
    s = E @ E[idx[w]]
    return [(vocab[j], round(float(s[j]), 2)) for j in np.argsort(-s)[1:4]]
q = E[idx["king"]] - E[idx["man"]] + E[idx["woman"]]
best = [vocab[j] for j in np.argsort(-(E @ q)) if vocab[j] not in ("king", "man", "woman")][0]
print("drop prob:", {w: round(float(p_drop[idx[w]]), 2) for w in ("the", "king", "crown")})
# drop prob: {'the': 0.78, 'king': 0.54, 'crown': 0.43}
print("king ->", nearest("king"))           # [('queen', 0.72), ('man', 0.52), ('crown', 0.51)]
print("king - man + woman =", best)         # queen
```

Skip-gram with negative sampling in numpy: subsampling, unigram^0.75 noise, random window shrink and linear learning-rate decay (run; outputs in comments)

> **Thresholds are corpus-size dependent:** $t=10^{-5}$ was tuned for billions of tokens. On a small corpus almost no word has $f<10^{-5}$, so nearly every token, including content words, gets dropped. The code above uses $t=10^{-2}$ and the figure uses $t=0.05$ for that reason. The same applies to `min_count`: dropping words seen fewer than 5 times is right for news text and wrong for a 2,000-sentence corpus. Rescale both to your data.

> **A classifier that secretly counts:** SGNS never builds a co-occurrence matrix, yet its optimum is $w\cdot c=\mathrm{PMI}(w,c)-\log k$. In the figure the correlation with that target reaches $0.975$ at $d=10$. At $d=2$ it is $0.234$, because a rank-2 product cannot fit a $21\times21$ matrix. Low dimension forces the model to compress, and the compression produces the generalisation.

> **Which vectors to keep:** Word2Vec keeps the input vectors $v_w$ and usually discards $v'_w$. Adding the two, as **GloVe** does, sometimes helps on similarity tasks. The two tables are needed during training. With one shared table, $v_w\cdot v_w=\|v_w\|^2$ would be large for every word, yet a word rarely appears in its own context.

**Strengths**

- Very cheap per update: k + 1 dot products of size d, no hidden layer
- Streams the corpus; memory is two V × d tables, no V × V matrix
- Scales to billions of tokens on CPUs; robust, well-studied defaults
- Vector offsets capture many relations (gender, tense, capital-of)

**Limitations**

- One static vector per word type: no polysemy, no context
- No vectors for unseen words (FastText adds subword n-grams for this)
- Sensitive to window, k, t and α; many gains attributed to the model come from these
- Many passes over local windows: global count statistics are used only implicitly

**Question:** Why does negative sampling draw noise words from U(w)^{3/4}/Z rather than the plain unigram distribution?

- A. It makes the sampling faster
- B. It flattens the distribution, so rare words appear more often as negatives; empirically it gave clearly better vectors
- C. It guarantees the positive word is never sampled
- D. It makes the objective equal to the full softmax

**Answer:** B. It flattens the distribution, so rare words appear more often as negatives; empirically it gave clearly better vectors — Raising frequencies to $0.75$ moves probability mass from frequent to rare words: shares $0.9,0.09,0.01$ become $0.825,0.147,0.028$. Mikolov et al. found it better than both unigram and uniform. Levy and Goldberg later showed it amounts to context distribution smoothing of the implied PMI matrix.

**Question:** In skip-gram with k = 5, a pair (w, c) has PMI(w, c) = 1.0. If d is large enough, what dot product w·c does training converge to?

- A. 1.0
- B. 1.0 − log 5 ≈ −0.61
- C. 1.0 + log 5 ≈ 2.61
- D. σ(1.0) ≈ 0.73

**Answer:** B. 1.0 − log 5 ≈ −0.61 — The optimum is $w\cdot c=\mathrm{PMI}(w,c)-\log k=1-\ln5=1-1.609=-0.609$. The shift comes from the expected $k$ negative occurrences of each pair. A positive PMI can therefore still give a negative dot product.

**Key points**

- Skip-gram maximises $\sum_t\sum_{j}\log p(w_{t+j}\mid w_t)$; CBOW predicts $w_t$ from the mean of its context vectors. Both are log-linear, with no hidden non-linearity.
- Negative sampling: $\log\sigma(v'^\top_{w_O}v_{w_I})+\sum_{i=1}^k\log\sigma(-v'^\top_{w_i}v_{w_I})$ with $w_i\sim U(w)^{3/4}/Z$, $k$ = 5–20 (small data) or 2–5 (large).
- Subsampling drops each token with probability $1-\sqrt{t/f(w)}$, $t\approx10^{-5}$. It runs about 2–10× faster and improves rare words.
- Levy and Goldberg (2014): SGNS implicitly factorises $\mathrm{PMI}(w,c)-\log k$, so Word2Vec is a weighted PMI factorisation, and its $3/4$ power is context smoothing.
- Static embeddings: one vector per word, input vectors kept, output vectors discarded; thresholds must be rescaled for small corpora.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
