Definition
Word2Vec is the family of shallow models from Mikolov, Chen, Corrado and Dean, "Efficient Estimation of Word Representations in Vector Space" (ICLR workshop, 2013), and Mikolov, Sutskever, Chen, Corrado and Dean, "Distributed Representations of Words and Phrases and their Compositionality" (NIPS 2013). It learns two vectors per word, an input vector and an output vector . Training sets them so that is large when appears near . Skip-gram predicts each context word from the centre word. CBOW (continuous bag of words) predicts the centre word from the average of its context. The second paper added negative sampling, subsampling of frequent words and phrase vectors. The input vectors are the word embeddings.
Neural language models already learned word vectors as a by-product: the feed-forward model of Bengio et al. (2003) and Mikolov's recurrent LM (2010). Their cost was in the hidden layers and in the softmax over the full vocabulary. Mikolov et al. asked how much of the representation quality survives if the model is made almost linear. They removed the non-linear hidden layer entirely. Each prediction is then one dot product per candidate word, and the remaining cost is the softmax. That made it possible to train on billions of words. The first paper reports high-quality vectors from a 1.6-billion-word dataset in under a day, and evaluates on a 6B-token Google News corpus with the vocabulary restricted to the 1M most frequent words.
Skip-gram and CBOW
- corpus length and window radius; the reference code samples the actual radius uniformly from for each word, which weights nearby words more
- input and output vectors, two separate tables; is kept as the embedding
- a full softmax over words: exact, but per pair, which is too slow for
- the mean of the context's input vectors: a bag of words, so order is lost
- one prediction per position instead of ; CBOW is faster, skip-gram is better for rare words
The first paper avoided the full softmax with hierarchical softmax. The vocabulary sits at the leaves of a binary Huffman tree, and is a product of about sigmoid decisions along the path to . Frequent words have short codes, so the average cost is even lower. The second paper introduced a simpler alternative that became the default.
Negative sampling
- logistic sigmoid; the task is now binary: real pair or noise pair?
- negatives per positive pair: 5–20 for small datasets, 2–5 suffice for large ones (Mikolov et al., 2013b)
- unigram distribution to the power 3/4, which raises rare words relative to frequent ones; it "outperformed significantly" both the unigram and the uniform distribution
- a simplification of noise-contrastive estimation (Gutmann and Hyvärinen, 2010): it needs only samples from , not their probabilities, and it does not approximate the softmax likelihood
Each pair now costs dot products. With , and negatives , the gradients are short. The positive output vector moves toward , each negative moves away from it, and moves toward and away from the , each weighted by how wrong the current sigmoid is:
- the negated objective for one pair,
- near 0 once the positive pair is confidently recognised, so easy pairs stop contributing
Subsampling frequent words
- relative frequency of the word in the corpus
- threshold; words with are never dropped. The reference code uses a slightly different keep probability, , with default
Each occurrence is dropped independently before the windows are formed. Very frequent words such as "the", "in" and "a" carry little information about their neighbours, and they supply most of the training pairs. Removing most of their occurrences gave a speed-up of about 2–10× and improved the vectors of rarer words. Dropping a word also brings more distant words inside the window, so subsampling widens the effective context.
What SGNS actually optimises
Levy and Goldberg ("Neural Word Embedding as Implicit Matrix Factorization", NeurIPS 2014) summed the SGNS objective over the corpus. The pair appears times as a positive. As a negative it appears in expectation times. Write and treat each as a free variable, which holds when is large enough:
- , and
- with this is the ordinary PMI; with the power, PMI with a smoothed context distribution
- more negatives shift every target down; they proposed the count-based SPPMI
So skip-gram with negative sampling is a weighted, implicit factorisation of the matrix used in Word embeddings. It is never formed, and frequent pairs get more weight. At low the fit is approximate, and the model chooses which cells to get right. This explains why the power helped. It is context distribution smoothing, and it carries over to PPMI + SVD.
Settings from the papers and the reference code
- Papers
- ICLR 2013 workshop (skip-gram, CBOW, hierarchical softmax, analogy test set) and NIPS 2013 (negative sampling, subsampling, phrases)
- Analogy set
- 19,544 questions: 8,869 semantic ("Athens : Greece :: Oslo : Norway") and 10,675 syntactic ("big : biggest :: small : smallest")
- Paper hyperparameters
- = 5–20 for small corpora, 2–5 for large; ; noise
- word2vec.c defaults
-size 100 -window 5 -negative 5 -hs 0 -sample 1e-3 -min-count 5 -iter 5; learning rate 0.025 for skip-gram, 0.05 for CBOW, decayed linearly to near zero- Initialisation
- input vectors uniform in ; output vectors zero; noise sampled from a precomputed unigram table
- GoogleNews vectors
- 300-d, 3M words and phrases, trained on about 100B words
- gensim
Word2Vec(sentences, vector_size=300, window=5, min_count=5, sg=1, negative=5, ns_exponent=0.75, sample=1e-3);sg=0is CBOW
Worked example: one SGNS update
Take , centre vector , the true context's output vector , negatives and , and learning rate .
| vector | u·v | σ(u·v) | loss term |
|---|---|---|---|
| positive u₊ | 0.080 | 0.5200 | −log σ(0.08) = 0.6539 |
| negative u₁ | −0.040 | 0.4900 | −log σ(0.04) = 0.6733 |
| negative u₂ | 0.130 | 0.5325 | −log σ(−0.13) = 0.7603 |
| total | 2.0876 |
Gradients and the step
- 1.
- 2; ; .
- 3Step all four vectors with rate 0.5, each using the old values: and .
- 4New scores: positive (up from 0.080), negatives and (both down). The loss falls from to .
- 5Noise distribution: three words with counts 900, 90 and 10 have unigram shares . To the power they become , almost three times as many negatives for the rarest word.
- 6Subsampling with : a word with is dropped with probability . With the probability is , with it is , and with it is 0.
1import numpy as np23rng = np.random.default_rng(0)4people = [("king", "his", 1), ("queen", "her", 1), ("man", "his", 0), ("woman", "her", 0)]5def sentence():6 p, poss, royal = people[rng.integers(4)]7 options = [("wore", "crown"), ("ruled", "land"), ("walked", "dog")] if royal else [("worked", "farm"), ("walked", "dog")]8 verb, obj = options[rng.integers(len(options))]9 return ["the", p, verb, poss, obj]10corpus = [sentence() for _ in range(2000)]1112vocab = sorted({w for s in corpus for w in s})13idx = {w: i for i, w in enumerate(vocab)}14counts = np.zeros(len(vocab))15for s in corpus:16 for w in s:17 counts[idx[w]] += 118freq = counts / counts.sum()1920d, window, k, t, epochs, lr0 = 16, 2, 5, 1e-2, 5, 0.02521noise = counts ** 0.75 / (counts ** 0.75).sum() # P_n(w) ∝ U(w)^(3/4)22p_drop = np.maximum(0, 1 - np.sqrt(t / freq)) # subsampling; t = 1e-5 on real corpora23W = (rng.random((len(vocab), d)) - 0.5) / d # input vectors (the embeddings)24C = np.zeros((len(vocab), d)) # output / context vectors25sigmoid = lambda x: 1 / (1 + np.exp(-x))2627steps, total = 0, epochs * sum(len(s) for s in corpus)28for epoch in range(epochs):29 for s in corpus:30 ids = [idx[w] for w in s if rng.random() >= p_drop[idx[w]]]31 for i, w in enumerate(ids):32 lr = lr0 * max(1e-4, 1 - steps / total); steps += 133 b = rng.integers(1, window + 1) # word2vec's random window shrink34 for j in range(max(0, i - b), min(len(ids), i + b + 1)):35 if j == i:36 continue37 targets = np.concatenate(([ids[j]], rng.choice(len(vocab), k, p=noise)))38 labels = np.array([1.0] + [0.0] * k)39 g = lr * (labels - sigmoid(C[targets] @ W[w])) # gradient of log σ(±u·v)40 dW = g @ C[targets]41 np.add.at(C, targets, np.outer(g, W[w]))42 W[w] += dW4344E = W / np.linalg.norm(W, axis=1, keepdims=True)45def nearest(w):46 s = E @ E[idx[w]]47 return [(vocab[j], round(float(s[j]), 2)) for j in np.argsort(-s)[1:4]]48q = E[idx["king"]] - E[idx["man"]] + E[idx["woman"]]49best = [vocab[j] for j in np.argsort(-(E @ q)) if vocab[j] not in ("king", "man", "woman")][0]50print("drop prob:", {w: round(float(p_drop[idx[w]]), 2) for w in ("the", "king", "crown")})51# drop prob: {'the': 0.78, 'king': 0.54, 'crown': 0.43}52print("king ->", nearest("king")) # [('queen', 0.72), ('man', 0.52), ('crown', 0.51)]53print("king - man + woman =", best) # queenUse when
- +Very cheap per update: k + 1 dot products of size d, no hidden layer
- +Streams the corpus; memory is two V × d tables, no V × V matrix
- +Scales to billions of tokens on CPUs; robust, well-studied defaults
- +Vector offsets capture many relations (gender, tense, capital-of)
Avoid when
- −One static vector per word type: no polysemy, no context
- −No vectors for unseen words (FastText adds subword n-grams for this)
- −Sensitive to window, k, t and α; many gains attributed to the model come from these
- −Many passes over local windows: global count statistics are used only implicitly
Check yourself
Why does negative sampling draw noise words from U(w)^{3/4}/Z rather than the plain unigram distribution?
Check yourself
In skip-gram with k = 5, a pair (w, c) has PMI(w, c) = 1.0. If d is large enough, what dot product w·c does training converge to?
Remember
- ✓Skip-gram maximises ; CBOW predicts from the mean of its context vectors. Both are log-linear, with no hidden non-linearity.
- ✓Negative sampling: with , = 5–20 (small data) or 2–5 (large).
- ✓Subsampling drops each token with probability , . It runs about 2–10× faster and improves rare words.
- ✓Levy and Goldberg (2014): SGNS implicitly factorises , so Word2Vec is a weighted PMI factorisation, and its power is context smoothing.
- ✓Static embeddings: one vector per word, input vectors kept, output vectors discarded; thresholds must be rescaled for small corpora.