Word2Vec

Word2Vec (Mikolov et al., 2013) learns word vectors with a log-linear model: skip-gram predicts context words from the centre word, CBOW the reverse. Negative sampling from U(w)^¾ and subsampling of frequent words make it fast, and SGNS implicitly factorises PMI − log k.

15 min read922 wordscore0 viewsupdated 2026-10-13

Definition

Word2Vec is the family of shallow models from Mikolov, Chen, Corrado and Dean, "Efficient Estimation of Word Representations in Vector Space" (ICLR workshop, 2013), and Mikolov, Sutskever, Chen, Corrado and Dean, "Distributed Representations of Words and Phrases and their Compositionality" (NIPS 2013). It learns two vectors per word, an input vector vwv_w and an output vector vw′v'_w. Training sets them so that vc′⋅vwv'_c\cdot v_w is large when cc appears near ww. Skip-gram predicts each context word from the centre word. CBOW (continuous bag of words) predicts the centre word from the average of its context. The second paper added negative sampling, subsampling of frequent words and phrase vectors. The input vectors are the word embeddings.

Neural language models already learned word vectors as a by-product: the feed-forward model of Bengio et al. (2003) and Mikolov's recurrent LM (2010). Their cost was in the hidden layers and in the softmax over the full vocabulary. Mikolov et al. asked how much of the representation quality survives if the model is made almost linear. They removed the non-linear hidden layer entirely. Each prediction is then one dot product per candidate word, and the remaining cost is the softmax. That made it possible to train on billions of words. The first paper reports high-quality vectors from a 1.6-billion-word dataset in under a day, and evaluates on a 6B-token Google News corpus with the vocabulary restricted to the 1M most frequent words.

Sentence "the king wore his crown", centre w_t = wore, window ±2skip-gram: centre → each context wordworeone-hotWV × dv_wthekinghiscrownrow lookup, no non-linearity4 pairs, each scored by σ(u_c · v_w)better for rare words, slowerCBOW: mean of context → centrethekinghiscrownmeanof W rowshworeσ(u·h)word order inside thewindow is discarded1 prediction per position, fasterNegative sampling for the pair (wore, king), k = 5kinglabel 1doglabel 0thelabel 0herlabel 0fishlabel 0atelabel 0negatives drawnfrom P_n ∝ U(w)^¾loss = −log σ(u_king·v_wore) − Σᵢ log σ(−u_nᵢ·v_wore)k + 1 dot products per pair instead of a softmax over all V wordssubsampling: drop each token of w with probability 1 − √(t / f(w)), t ≈ 10⁻⁵two tables are learned: input vectors v_w (kept as the embeddings) and output vectors u_c
fig.Both architectures on the sentence "the king wore his crown" with centre word wore and a ±2 window. Skip-gram looks up the centre word's row of W and scores each of the four context words separately with σ(u_c · v_w): four training pairs. CBOW averages the four context rows into h and scores the centre word once, so word order inside the window is lost. Hover a panel to highlight it. Below, negative sampling replaces the V-way softmax for one pair (wore, king) with one positive logistic example and k = 5 negatives drawn from the unigram distribution raised to 3/4.

Skip-gram and CBOW

1T∑t=1T ∑−c≤j≤c, j≠0log⁡p(wt+j∣wt),p(wO∣wI)=exp⁡ ⁣(vwO′⊤vwI)∑w=1Vexp⁡ ⁣(vw′⊤vwI)\frac1T\sum_{t=1}^{T}\ \sum_{-c\le j\le c,\ j\ne0}\log p(w_{t+j}\mid w_t),\qquad p(w_O\mid w_I)=\frac{\exp\!\big(v'^{\top}_{w_O}v_{w_I}\big)}{\sum_{w=1}^{V}\exp\!\big(v'^{\top}_{w}v_{w_I}\big)}
T, cT,\ c
corpus length and window radius; the reference code samples the actual radius uniformly from 1…c1\dots c for each word, which weights nearby words more
vw, vw′v_w,\ v'_w
input and output vectors, two separate V×dV\times d tables; vwv_w is kept as the embedding
p(wO∣wI)p(w_O\mid w_I)
a full softmax over VV words: exact, but O(V)O(V) per pair, which is too slow for V=106V=10^6
CBOW:ht=12c∑−c≤j≤c, j≠0vwt+j,log⁡p(wt∣ht)=vwt′⊤ht−log⁡∑wexp⁡ ⁣(vw′⊤ht)\text{CBOW:}\quad h_t=\frac{1}{2c}\sum_{-c\le j\le c,\ j\ne0}v_{w_{t+j}},\qquad \log p(w_t\mid h_t)=v'^{\top}_{w_t}h_t-\log\sum_{w}\exp\!\big(v'^{\top}_{w}h_t\big)
hth_t
the mean of the context's input vectors: a bag of words, so order is lost
cost\text{cost}
one prediction per position instead of 2c2c; CBOW is faster, skip-gram is better for rare words

The first paper avoided the full softmax with hierarchical softmax. The vocabulary sits at the leaves of a binary Huffman tree, and p(w∣wI)p(w\mid w_I) is a product of about log⁡2V\log_2V sigmoid decisions along the path to ww. Frequent words have short codes, so the average cost is even lower. The second paper introduced a simpler alternative that became the default.

Negative sampling

log⁡σ ⁣(vwO′⊤vwI)+∑i=1kEwi∼Pn(w)[log⁡σ ⁣(−vwi′⊤vwI)],Pn(w)=U(w)3/4Z\log\sigma\!\big(v'^{\top}_{w_O}v_{w_I}\big)+\sum_{i=1}^{k}\mathbb E_{w_i\sim P_n(w)}\Big[\log\sigma\!\big(-v'^{\top}_{w_i}v_{w_I}\big)\Big],\qquad P_n(w)=\frac{U(w)^{3/4}}{Z}
σ\sigma
logistic sigmoid; the task is now binary: real pair or noise pair?
kk
negatives per positive pair: 5–20 for small datasets, 2–5 suffice for large ones (Mikolov et al., 2013b)
U(w)3/4/ZU(w)^{3/4}/Z
unigram distribution to the power 3/4, which raises rare words relative to frequent ones; it "outperformed significantly" both the unigram and the uniform distribution
vs. NCE\text{vs. NCE}
a simplification of noise-contrastive estimation (Gutmann and Hyvärinen, 2010): it needs only samples from PnP_n, not their probabilities, and it does not approximate the softmax likelihood

Each pair now costs k+1k+1 dot products. With v=vwIv=v_{w_I}, u+=vwO′u_+=v'_{w_O} and negatives uiu_i, the gradients are short. The positive output vector moves toward vv, each negative moves away from it, and vv moves toward u+u_+ and away from the uiu_i, each weighted by how wrong the current sigmoid is:

∂L∂v=(σ(u+⊤v)−1)u++∑i=1kσ(ui⊤v) ui,∂L∂u+=(σ(u+⊤v)−1)v,∂L∂ui=σ(ui⊤v) v\frac{\partial\mathcal L}{\partial v}=\big(\sigma(u_+^\top v)-1\big)u_+ +\sum_{i=1}^{k}\sigma(u_i^\top v)\,u_i,\qquad \frac{\partial\mathcal L}{\partial u_+}=\big(\sigma(u_+^\top v)-1\big)v,\qquad \frac{\partial\mathcal L}{\partial u_i}=\sigma(u_i^\top v)\,v
L\mathcal L
the negated objective for one pair, −log⁡σ(u+⊤v)−∑ilog⁡σ(−ui⊤v)-\log\sigma(u_+^\top v)-\sum_i\log\sigma(-u_i^\top v)
σ(⋅)−1\sigma(\cdot)-1
near 0 once the positive pair is confidently recognised, so easy pairs stop contributing

Subsampling frequent words

Pdiscard(wi)=1−tf(wi),t≈10−5P_{\text{discard}}(w_i)=1-\sqrt{\frac{t}{f(w_i)}},\qquad t\approx10^{-5}
f(wi)f(w_i)
relative frequency of the word in the corpus
tt
threshold; words with f≤tf\le t are never dropped. The reference code uses a slightly different keep probability, (f/t+1) t/f(\sqrt{f/t}+1)\,t/f, with default t=10−3t=10^{-3}

Each occurrence is dropped independently before the windows are formed. Very frequent words such as "the", "in" and "a" carry little information about their neighbours, and they supply most of the training pairs. Removing most of their occurrences gave a speed-up of about 2–10× and improved the vectors of rarer words. Dropping a word also brings more distant words inside the window, so subsampling widens the effective context.

What SGNS actually optimises

Levy and Goldberg ("Neural Word Embedding as Implicit Matrix Factorization", NeurIPS 2014) summed the SGNS objective over the corpus. The pair (w,c)(w,c) appears #(w,c)\#(w,c) times as a positive. As a negative it appears in expectation k #(w) Pn(c)k\,\#(w)\,P_n(c) times. Write x=w⋅cx=w\cdot c and treat each xx as a free variable, which holds when dd is large enough:

ℓ(x)=#(w,c)log⁡σ(x)+k #(w) Pn(c)log⁡σ(−x)  ⇒  ℓ′(x)=0  ⟺  x=log⁡#(w,c)#(w) Pn(c)−log⁡k=PMI(w,c)−log⁡k\ell(x)=\#(w,c)\log\sigma(x)+k\,\#(w)\,P_n(c)\log\sigma(-x)\ \ \Rightarrow\ \ \ell'(x)=0\iff x=\log\frac{\#(w,c)}{\#(w)\,P_n(c)}-\log k=\mathrm{PMI}(w,c)-\log k
ℓ′(x)\ell'(x)
#(w,c) σ(−x)−k #(w)Pn(c) σ(x)=0\#(w,c)\,\sigma(-x)-k\,\#(w)P_n(c)\,\sigma(x)=0, and σ(x)/σ(−x)=ex\sigma(x)/\sigma(-x)=e^x
Pn(c)P_n(c)
with Pn∝#(c)P_n\propto\#(c) this is the ordinary PMI; with the 3/43/4 power, PMI with a smoothed context distribution
−log⁡k-\log k
more negatives shift every target down; they proposed the count-based SPPMI =max⁡(PMI−log⁡k,0)=\max(\mathrm{PMI}-\log k,0)

So skip-gram with negative sampling is a weighted, implicit factorisation WC⊤≈PMI−log⁡kWC^\top\approx\mathrm{PMI}-\log k of the matrix used in Word embeddings. It is never formed, and frequent pairs get more weight. At low dd the fit is approximate, and the model chooses which cells to get right. This explains why the 3/43/4 power helped. It is context distribution smoothing, and it carries over to PPMI + SVD.

Settings from the papers and the reference code

Papers
ICLR 2013 workshop (skip-gram, CBOW, hierarchical softmax, analogy test set) and NIPS 2013 (negative sampling, subsampling, phrases)
Analogy set
19,544 questions: 8,869 semantic ("Athens : Greece :: Oslo : Norway") and 10,675 syntactic ("big : biggest :: small : smallest")
Paper hyperparameters
kk = 5–20 for small corpora, 2–5 for large; t≈10−5t\approx10^{-5}; noise U(w)3/4/ZU(w)^{3/4}/Z
word2vec.c defaults
-size 100 -window 5 -negative 5 -hs 0 -sample 1e-3 -min-count 5 -iter 5; learning rate 0.025 for skip-gram, 0.05 for CBOW, decayed linearly to near zero
Initialisation
input vectors uniform in [−0.5/d, 0.5/d][-0.5/d,\,0.5/d]; output vectors zero; noise sampled from a precomputed unigram0.75^{0.75} table
GoogleNews vectors
300-d, 3M words and phrases, trained on about 100B words
gensim
Word2Vec(sentences, vector_size=300, window=5, min_count=5, sg=1, negative=5, ns_exponent=0.75, sample=1e-3); sg=0 is CBOW
skip-gram · d = 10 · window ±2 · k = 5 negatives from U^0.75 · epoch 40/40input vectors w on the final top-2 PCs (41% of variance)thekingqueenmanwomanworeruledworkedwalkedatechasedhishercrownlandfarmdogcatbreadmeatfishmean loss per (w, c) pair by epoch2.661.69epoch 40w·c (y) vs PMI − log k (x), 126 pairsr = 0.975axes −4…4, dashed y = xnearest to king (cosine in d = 10): man 0.653 · queen 0.564 · land 0.537king − man + woman → queen 0.991 (next chased 0.566)noise P_n: the 0.163 · king 0.043 · fish 0.023 (unigram: 0.231 · 0.039 · 0.017)subsampling off; 5040 (w, c) pairs per epochLevy & Goldberg: at the optimum w·c = PMI(w, c) − log k for every observed pairlr 0.05 decaying linearly · trails show each word's path since epoch 0 · click a dot to select● people ● animals ● objects ● his/her ● verbs, the · same corpus as the Word embeddings figure
noise U^
fig.Skip-gram with negative sampling trained live, epoch by epoch, on the toy corpus of the Word embeddings figure (1,860 tokens, 21 words). The defaults are d = 10, window ±2, k = 5 negatives from U^0.75, learning rate 0.05 decaying linearly, and 40 epochs of 5,040 (w, c) pairs. Press play to watch the vectors spread from their tiny random start, with each word's trail shown. The map is a fixed PCA of the final vectors (41% of the variance in 2-D). The mean loss falls from 2.658 after epoch 1 to 1.699 after epoch 40. The right panel tests Levy and Goldberg's result directly. Each of the 126 observed pairs is plotted as its learned w·c against log(#(w,c) / (#(w)·P_n(c))) − log k. At epoch 0 every context vector is zero and r is undefined; then r = 0.350 at epoch 1, 0.773 at 5, 0.839 at 10 and 0.975 at 40. king − man + woman gives queen 0.991. king's neighbours are man 0.653, queen 0.564 and land 0.537. The noise distribution gives the 0.163 instead of its unigram share 0.231, and fish 0.023 instead of 0.017. With d = 2 the matrix cannot fit (r = 0.234), and king's three nearest words all have cosine ≥ 0.999. At d = 5, r = 0.867 and king's nearest word is queen (0.991). CBOW is not a per-pair PMI fit (r = 0.453). Window ±1 gives 2,920 pairs per epoch, r = 0.835, and king → queen 0.994. Uniform noise (U^0) gives r = 0.992. Subsampling with t = 0.05 (scaled up for a tiny corpus) drops 53.5% of the occurrences of the, 18.5% of ate and 13.4% of his. That leaves 3,820 updates in epoch 40, and r is 0.958.

Worked example: one SGNS update

Take d=3d=3, centre vector v=(0.2,−0.1,0.4)v=(0.2,-0.1,0.4), the true context's output vector u+=(0.3,0.2,0.1)u_+=(0.3,0.2,0.1), k=2k=2 negatives u1=(−0.2,0.4,0.1)u_1=(-0.2,0.4,0.1) and u2=(0.1,−0.3,0.2)u_2=(0.1,-0.3,0.2), and learning rate 0.50.5.

vectoru·vσ(u·v)loss term
positive u₊0.0800.5200−log σ(0.08) = 0.6539
negative u₁−0.0400.4900−log σ(0.04) = 0.6733
negative u₂0.1300.5325−log σ(−0.13) = 0.7603
total2.0876
Scores and loss terms before the update (computed in numpy)

Gradients and the step

  1. 1∂L/∂v=(0.5200−1) u++0.4900 u1+0.5325 u2=(−0.144,−0.096,−0.048)+(−0.098,0.196,0.049)+(0.053,−0.160,0.106)=(−0.1888,−0.0597,0.1075)\partial\mathcal L/\partial v=(0.5200-1)\,u_++0.4900\,u_1+0.5325\,u_2=(-0.144,-0.096,-0.048)+(-0.098,0.196,0.049)+(0.053,-0.160,0.106)=(-0.1888,-0.0597,0.1075).
  2. 2∂L/∂u+=−0.48 v=(−0.096,0.048,−0.192)\partial\mathcal L/\partial u_+=-0.48\,v=(-0.096,0.048,-0.192); ∂L/∂u1=0.49 v=(0.098,−0.049,0.196)\partial\mathcal L/\partial u_1=0.49\,v=(0.098,-0.049,0.196); ∂L/∂u2=0.5325 v=(0.1065,−0.0532,0.2130)\partial\mathcal L/\partial u_2=0.5325\,v=(0.1065,-0.0532,0.2130).
  3. 3Step all four vectors with rate 0.5, each using the old values: v→(0.2944,−0.0701,0.3463)v\to(0.2944,-0.0701,0.3463) and u+→(0.348,0.176,0.196)u_+\to(0.348,0.176,0.196).
  4. 4New scores: positive 0.1580.158 (up from 0.080), negatives −0.102-0.102 and 0.0650.065 (both down). The loss falls from 2.0882.088 to 1.9871.987.
  5. 5Noise distribution: three words with counts 900, 90 and 10 have unigram shares 0.900,0.090,0.0100.900,0.090,0.010. To the power 3/43/4 they become 0.825,0.147,0.0280.825,0.147,0.028, almost three times as many negatives for the rarest word.
  6. 6Subsampling with t=10−5t=10^{-5}: a word with f=0.05f=0.05 is dropped with probability 1−2⋅10−4=0.9861-\sqrt{2\cdot10^{-4}}=0.986. With f=10−3f=10^{-3} the probability is 0.9000.900, with f=10−4f=10^{-4} it is 0.6840.684, and with f≤10−5f\le10^{-5} it is 0.
python
1import numpy as np
2
3rng = np.random.default_rng(0)
4people = [("king", "his", 1), ("queen", "her", 1), ("man", "his", 0), ("woman", "her", 0)]
5def sentence():
6 p, poss, royal = people[rng.integers(4)]
7 options = [("wore", "crown"), ("ruled", "land"), ("walked", "dog")] if royal else [("worked", "farm"), ("walked", "dog")]
8 verb, obj = options[rng.integers(len(options))]
9 return ["the", p, verb, poss, obj]
10corpus = [sentence() for _ in range(2000)]
11
12vocab = sorted({w for s in corpus for w in s})
13idx = {w: i for i, w in enumerate(vocab)}
14counts = np.zeros(len(vocab))
15for s in corpus:
16 for w in s:
17 counts[idx[w]] += 1
18freq = counts / counts.sum()
19
20d, window, k, t, epochs, lr0 = 16, 2, 5, 1e-2, 5, 0.025
21noise = counts ** 0.75 / (counts ** 0.75).sum() # P_n(w) ∝ U(w)^(3/4)
22p_drop = np.maximum(0, 1 - np.sqrt(t / freq)) # subsampling; t = 1e-5 on real corpora
23W = (rng.random((len(vocab), d)) - 0.5) / d # input vectors (the embeddings)
24C = np.zeros((len(vocab), d)) # output / context vectors
25sigmoid = lambda x: 1 / (1 + np.exp(-x))
26
27steps, total = 0, epochs * sum(len(s) for s in corpus)
28for epoch in range(epochs):
29 for s in corpus:
30 ids = [idx[w] for w in s if rng.random() >= p_drop[idx[w]]]
31 for i, w in enumerate(ids):
32 lr = lr0 * max(1e-4, 1 - steps / total); steps += 1
33 b = rng.integers(1, window + 1) # word2vec's random window shrink
34 for j in range(max(0, i - b), min(len(ids), i + b + 1)):
35 if j == i:
36 continue
37 targets = np.concatenate(([ids[j]], rng.choice(len(vocab), k, p=noise)))
38 labels = np.array([1.0] + [0.0] * k)
39 g = lr * (labels - sigmoid(C[targets] @ W[w])) # gradient of log σ(±u·v)
40 dW = g @ C[targets]
41 np.add.at(C, targets, np.outer(g, W[w]))
42 W[w] += dW
43
44E = W / np.linalg.norm(W, axis=1, keepdims=True)
45def nearest(w):
46 s = E @ E[idx[w]]
47 return [(vocab[j], round(float(s[j]), 2)) for j in np.argsort(-s)[1:4]]
48q = E[idx["king"]] - E[idx["man"]] + E[idx["woman"]]
49best = [vocab[j] for j in np.argsort(-(E @ q)) if vocab[j] not in ("king", "man", "woman")][0]
50print("drop prob:", {w: round(float(p_drop[idx[w]]), 2) for w in ("the", "king", "crown")})
51# drop prob: {'the': 0.78, 'king': 0.54, 'crown': 0.43}
52print("king ->", nearest("king")) # [('queen', 0.72), ('man', 0.52), ('crown', 0.51)]
53print("king - man + woman =", best) # queen
Skip-gram with negative sampling in numpy: subsampling, unigram^0.75 noise, random window shrink and linear learning-rate decay (run; outputs in comments)

Use when

  • +Very cheap per update: k + 1 dot products of size d, no hidden layer
  • +Streams the corpus; memory is two V × d tables, no V × V matrix
  • +Scales to billions of tokens on CPUs; robust, well-studied defaults
  • +Vector offsets capture many relations (gender, tense, capital-of)

Avoid when

  • −One static vector per word type: no polysemy, no context
  • −No vectors for unseen words (FastText adds subword n-grams for this)
  • −Sensitive to window, k, t and α; many gains attributed to the model come from these
  • −Many passes over local windows: global count statistics are used only implicitly

Check yourself

Why does negative sampling draw noise words from U(w)^{3/4}/Z rather than the plain unigram distribution?

Check yourself

In skip-gram with k = 5, a pair (w, c) has PMI(w, c) = 1.0. If d is large enough, what dot product w·c does training converge to?

Remember

  • ✓Skip-gram maximises ∑t∑jlog⁡p(wt+j∣wt)\sum_t\sum_{j}\log p(w_{t+j}\mid w_t); CBOW predicts wtw_t from the mean of its context vectors. Both are log-linear, with no hidden non-linearity.
  • ✓Negative sampling: log⁡σ(vwO′⊤vwI)+∑i=1klog⁡σ(−vwi′⊤vwI)\log\sigma(v'^\top_{w_O}v_{w_I})+\sum_{i=1}^k\log\sigma(-v'^\top_{w_i}v_{w_I}) with wi∼U(w)3/4/Zw_i\sim U(w)^{3/4}/Z, kk = 5–20 (small data) or 2–5 (large).
  • ✓Subsampling drops each token with probability 1−t/f(w)1-\sqrt{t/f(w)}, t≈10−5t\approx10^{-5}. It runs about 2–10× faster and improves rare words.
  • ✓Levy and Goldberg (2014): SGNS implicitly factorises PMI(w,c)−log⁡k\mathrm{PMI}(w,c)-\log k, so Word2Vec is a weighted PMI factorisation, and its 3/43/4 power is context smoothing.
  • ✓Static embeddings: one vector per word, input vectors kept, output vectors discarded; thresholds must be rescaled for small corpora.
← back to Deep Learning roadmap