FastText

FastText (Bojanowski et al. 2017) builds a word vector from the word plus its character n-grams (n = 3–6, with < > markers) hashed into 2M buckets and trained by skip-gram, so unseen words get vectors; the fastText classifier (Joulin et al. 2017) averages embeddings into a linear softmax.

14 min read1177 wordscore0 viewsupdated 2026-10-13

Definition

FastText is two related methods from Facebook AI Research. The first is the subword word-embedding model of Bojanowski, Grave, Joulin and Mikolov, "Enriching Word Vectors with Subword Information" (TACL 2017, arXiv 2016). It extends skip-gram so that a word's vector is the sum of the vectors of its character n-grams plus a vector for the word itself. The second is the fastText text classifier of Joulin, Grave, Bojanowski and Mikolov, "Bag of Tricks for Efficient Text Classification" (EACL 2017). It averages word and word-n-gram embeddings into one text vector and feeds a linear softmax. Both ship in the open-source fastText library.

Word2Vec and GloVe give every word type its own vector and treat the type as an atom. Nothing ties "run", "runs", "running" and "runner" together except the contexts they happen to appear in. That has two costs. Rare inflections get poor vectors because they have few contexts. Words absent from the training vocabulary get no vector at all. English hides the problem fairly well. It is severe in morphologically rich languages such as Finnish, Turkish, Czech or German, where one lemma can have dozens of surface forms and compounds are formed freely.

Bojanowski et al. kept the skip-gram objective and changed only how a word is represented. The vector of a word is built from pieces shared with other words. "unhappiness" shares "happ", "ness" and "<un" with words seen in training, so it gets a sensible vector even if it never occurred. Earlier work had composed word vectors from morphemes, which needs a morphological analyser. Character n-grams need nothing beyond the string.

Character n-grams with boundary markers

Each word ww is padded with < at the start and > at the end. The set Gw\mathcal G_w contains every character n-gram of the padded string with 3≤n≤63\le n\le 6, plus the padded word itself as a special sequence. For "where" that gives <wh, whe, her, ere, re> at n=3n=3, then the 4-, 5- and 6-grams, and finally <where>. The markers matter. The trigram her inside "where" is a different string from the word "her", which is <her>. Prefixes such as <un and suffixes such as ess> become distinct units from the same letters in the middle of a word.

∣Gw∣=1+∑n=36(∣w∣+3−n)=1+4∣w∣−6(∣w∣≥4)|\mathcal G_w|=1+\sum_{n=3}^{6}\big(|w|+3-n\big)=1+4|w|-6\qquad(|w|\ge 4)
∣w∣|w|
word length in characters; the padded string has ∣w∣+2|w|+2
∣w∣+3−n|w|+3-n
number of n-grams of length nn in the padded string
11
the whole word <w>, which keeps its own vector when it is in the vocabulary
subword skip-gram (Bojanowski et al. 2017)defaults: minn 3 · maxn 6 · K = 2M · d = 300where<where>add < and >n=3<wh whe her ere re>n=4<whe wher here ere>n=5<wher where here>n=6<where where>word<where> (own row)14 n-grams + 1 word = G_w"her" ≠ "<her>": markers keep affixes apartFNV-1a hashmod K = 2,000,000→ row nwords + hinput matrix Znwords word rows+ 2M bucket rows, d = 300u_w = Σ z_gg ∈ G_w(code: mean)s(w,c) = Σ z_gᵀ v_clogistic loss, 5 negativescontext c: window 1–5OOV word: the same path without the word row, so any string gets a vector from its n-gramsfastText classifier (Joulin et al. 2017)tokens + bigramsbigrams hashedinto bucketslookup, averageone vector per texth = 10 in the paperlinear → softmaxhierarchical softmaxfor many labelslabel
fig.Top: the subword model. "where" is padded, split into its 14 n-grams of length 3 to 6, and each n-gram is hashed with FNV-1a into one of 2,000,000 bucket rows that sit after the word rows of the input matrix. The word vector is the sum of those rows (the code averages them), and it is scored against a context vector exactly as in skip-gram. Bottom: the classifier, which averages word and hashed word-bigram embeddings and feeds a linear softmax.

Scoring and the training objective

s(w,c)=∑g∈Gwzg⊤vc,L=∑t∑c∈Ct[log⁡(1+e−s(wt,c))+∑n∈Nt,clog⁡(1+es(wt,n))]s(w,c)=\sum_{g\in\mathcal G_w} \mathbf z_g^{\top}\mathbf v_c,\qquad \mathcal L=\sum_{t}\sum_{c\in\mathcal C_t}\Big[\log\big(1+e^{-s(w_t,c)}\big)+\sum_{n\in\mathcal N_{t,c}}\log\big(1+e^{s(w_t,n)}\big)\Big]
zg\mathbf z_g
input vector of n-gram (or word) gg; a row of the input matrix
vc\mathbf v_c
output vector of context word cc; contexts stay whole words
Ct\mathcal C_t
context positions around tt; the window size is sampled uniformly from 1 to 5
Nt,c\mathcal N_{t,c}
5 negative words sampled by frequency; the fastText code draws them with probability proportional to the square root of the count
log⁡(1+e−s)\log(1+e^{-s})
binary logistic loss: push true pairs up, sampled pairs down

The only change from skip-gram with negative sampling is the first sum. A word's input vector uw=∑g∈Gwzg\mathbf u_w=\sum_{g\in\mathcal G_w}\mathbf z_g is no longer a free parameter. Every gradient step on "running" also updates the rows for run, unn and ing>, which "runner", "sunning" and "playing" share. Rare words borrow statistical strength from frequent words with the same pieces.

Hashing n-grams into buckets

The number of distinct n-grams in a large corpus runs into the hundreds of millions, so they are not given a vocabulary. Instead each n-gram is hashed with the 32-bit FNV-1a function and reduced modulo K=2,000,000K=2{,}000{,}000. The input matrix has ∣V∣|V| word rows followed by KK bucket rows, and n-gram gg uses row ∣V∣+FNV(g) mod K|V|+\mathrm{FNV}(g)\bmod K. Two different n-grams can land in the same bucket and then share a vector. With KK in the millions such collisions are rare and act as mild noise. The memory cost is fixed and known in advance: at d=300d=300 in float32, the bucket rows alone take 2⋅106⋅300⋅42\cdot10^6\cdot300\cdot4 bytes =2.4=2.4 GB.

h0=2166136261,hi=(hi−1⊕bi)⋅16777619 mod 232,row(g)=∣V∣+h∣g∣ mod Kh_0=2166136261,\qquad h_{i}=\big(h_{i-1}\oplus b_i\big)\cdot 16777619\bmod 2^{32},\qquad \mathrm{row}(g)=|V|+h_{|g|}\bmod K
bib_i
the ii-th UTF-8 byte of gg; the code sign-extends it from int8 first, which only matters for non-ASCII bytes
⊕\oplus
bitwise XOR
KK
the -bucket option; 2,000,000 by default

Vectors for out-of-vocabulary words

Because the n-gram rows exist independently of the vocabulary, any string has a vector. For an out-of-vocabulary word fastText sums (in the code, averages) the rows of its n-grams and leaves out the word row it does not have. The paper calls this model sisg. Its ablation sisg- gives OOV words a null vector instead, and sisg is better on rare-word similarity. A misspelling such as "happpiness" or a new compound lands near the words whose pieces it shares.

The vector says nothing about meaning beyond what the pieces carry. "Unhappiness" works because happ and ness> carry meaning. A name or an opaque loanword gets a vector dominated by accidental letter overlap. Used on its own, a subword vector is a good guess for inflected and derived forms and a weak one for everything else.

Settings in the papers and the library

Subword model
skip-gram with negative sampling, 5 negatives, window sampled from 1 to 5, subsampling threshold 10−410^{-4}, words with fewer than 5 occurrences dropped, d=300d=300, n-grams 3≤n≤63\le n\le 6, K=2⋅106K=2\cdot10^6 buckets
Training data
Wikipedia dumps in nine languages (Arabic, Czech, German, English, Spanish, French, Italian, Romanian, Russian) in the 2017 paper
Library defaults
-minn 3 -maxn 6 -bucket 2000000 -neg 5 -ws 5 -minCount 5 -t 1e-4; dim defaults to 100
Released vectors
Wikipedia vectors for 294 languages; English crawl-300d-2M-subword (Mikolov et al., LREC 2018, 600B Common Crawl tokens, 2M words); Common Crawl + Wikipedia vectors for 157 languages (Grave et al., LREC 2018: CBOW with position weights, d=300d=300, character 5-grams, window 5, 10 negatives)
Classifier
text vector = mean of word (and hashed word n-gram) embeddings, then a linear softmax or hierarchical softmax; h=10h=10 hidden units, 5 epochs, learning rate tuned on validation in the paper
Classifier results
AG News accuracy 91.5% with unigrams and 92.5% with bigrams, comparable to or better than the char-CNN and VDCNN baselines; the paper trains on more than a billion words in under ten minutes on a multicore CPU
v(w) = mean of rows: word row (if in vocab) + row NW + FNV-1a(g) mod K for each n-gram g<unhappiness>out of vocabulary: 38 n-gram rows, no word rown=3n=4n=5n=6<un<unh<unha<unhapunhunhaunhapunhappnhanhapnhappnhappihaphapphappihappinappappiappinappineppippinppineppinespinpinepinespinessineinesinessiness>nesnessness>essess>ss>hover: "<un" → FNV-1a 0x3c7fd1ec = 1,015,009,772 → mod 2M = 1,009,772 → row 1,009,782■ same n-gram in "happiness" · ■ different n-gram, same bucket (hash collision)vocabulary wordcosine (bar) · overlap prediction (|)happiness0.76726 sharedunhappy0.51714 sharedkindness0.2106 sharedsadness0.1556 sharedhappy0.1123 sharedrunner0.0790 sharedtable0.0640 sharednowhere0.0550 sharedwhere0.0060 sharedrunning-0.0000 sharedrows averaged for the query38rows shared with "happiness"26cosine (d = 300, random rows)0.767overlap prediction Σaᵢbᵢ/‖a‖‖b‖0.758distinct n-grams → buckets157 → 157collisions on this page0input matrix (10 + 2M) × 3002.40 GB
word
vocabulary: happiness, unhappy, happy, sadness, kindness, where, nowhere, running, runner, table. Click a row to compare; hover a chip to see its hash.
fig.The query word is padded and split into n-grams (the chips, one row per n), and each n-gram is hashed with fastText's own FNV-1a into a bucket. Hover a chip to see the hash. There is no trained model here: every row of the input matrix is a fixed random 300-dimensional vector, so similarity can only come from shared rows, and the orange tick shows the exact prediction Σaᵢbᵢ/‖a‖‖b‖ for orthonormal rows. At the defaults ("unhappiness", n from 3 to 6, 2M buckets) the word is out of vocabulary and its vector averages 38 n-gram rows. It shares 26 rows with "happiness" (cosine 0.767, prediction 0.758), 14 with "unhappy" (0.517 vs 0.474) and 6 with "kindness" (0.210 vs 0.187). Words with no shared rows stay near 0, within the ±0.06 crosstalk expected of random 300-d vectors ("runner" reads 0.079). The 157 distinct n-grams on the page fill 157 buckets, so nothing collides. Cut the buckets to 64 and they fall into 60 buckets: "running" now shares 9 rows with "unhappiness" through collisions alone and jumps to 0.425. Raise minn to 5 and the query has only 17 rows; "happiness" drops to 0.677. "somewhere" lands near "where" (0.460) and "nowhere" (0.384) through shared n-grams such as wher, here and ere>.

Worked example: from "where" to a row index, and an OOV vector

nn-gramscount
3<wh, whe, her, ere, re>5
4<whe, wher, here, ere>4
5<wher, where, here>3
6<where, where>2
word<where>1
Character n-grams of <where> with minn = 3, maxn = 6 (enumerated in Python)

Rows, hashes and one similarity

  1. 1"where" has ∣w∣=5|w|=5, so the padded string has 7 characters and there are 5+4+3+2=145+4+3+2=14 n-grams, matching 4∣w∣−6=144|w|-6=14. With the word itself, ∣Gw∣=15|\mathcal G_w|=15.
  2. 2Hash <wh: FNV-1a gives 1,048,167,6521{,}048{,}167{,}652, and 1,048,167,652 mod 2,000,000=167,6521{,}048{,}167{,}652\bmod 2{,}000{,}000=167{,}652. The n-gram uses row ∣V∣+167,652|V|+167{,}652.
  3. 3Markers keep units apart: the trigram her hashes to bucket 1,473,4201{,}473{,}420, the word <her> to bucket 972,692972{,}692.
  4. 4"unhappiness" (∣w∣=11|w|=11) is out of vocabulary: 4⋅11−6=384\cdot11-6=38 n-gram rows and no word row. "happiness" (∣w∣=9|w|=9) is in vocabulary: 3030 n-gram rows plus its word row, 31 rows. They share 26 n-grams, such as hap, happi, iness and ess>.
  5. 5If all rows were orthonormal, the cosine of the two averaged vectors would be 26/38⋅31=0.75826/\sqrt{38\cdot31}=0.758. With random 300-d rows (the figure) it is 0.7670.767; the difference is crosstalk of size about 1/300=0.0581/\sqrt{300}=0.058.
python
1import numpy as np
2
3def fnv1a(s: str) -> int:
4 """fastText's hash: 32-bit FNV-1a over UTF-8 bytes, each byte sign-extended from int8."""
5 h = 2166136261
6 for b in s.encode("utf-8"):
7 h ^= b | 0xFFFFFF00 if b >= 128 else b
8 h = (h * 16777619) & 0xFFFFFFFF
9 return h
10
11def subwords(word: str, minn: int = 3, maxn: int = 6) -> list[str]:
12 w = f"<{word}>"
13 return [w[i:i + n] for i in range(len(w)) for n in range(minn, maxn + 1) if i + n <= len(w)]
14
15class SubwordTable:
16 def __init__(self, vocab, dim=300, bucket=2_000_000, minn=3, maxn=6, seed=0):
17 self.word2id = {w: i for i, w in enumerate(vocab)}
18 self.bucket, self.minn, self.maxn, self.dim = bucket, minn, maxn, dim
19 self.rng, self.rows = np.random.default_rng(seed), {} # allocate only the rows we touch
20
21 def row_ids(self, word):
22 ids = [self.word2id[word]] if word in self.word2id else [] # OOV: no word row
23 nw = len(self.word2id)
24 return ids + [nw + fnv1a(g) % self.bucket for g in subwords(word, self.minn, self.maxn)]
25
26 def row(self, i): # fastText initialises rows uniformly in ±1/dim
27 if i not in self.rows:
28 self.rows[i] = self.rng.uniform(-1 / self.dim, 1 / self.dim, self.dim)
29 return self.rows[i]
30
31 def vector(self, word): # the library averages; the paper writes the sum
32 return np.mean([self.row(i) for i in self.row_ids(word)], axis=0)
33
34print(subwords("where"))
35print(len(subwords("where")), len(subwords("unhappiness"))) # 14 38
36print([fnv1a(g) % 2_000_000 for g in ("<wh", "her", "<her>")]) # [167652, 1473420, 972692]
37
38table = SubwordTable(["happiness", "unhappy", "table"])
39cos = lambda a, b: a @ b / np.linalg.norm(a) / np.linalg.norm(b)
40q = table.vector("unhappiness") # out of vocabulary
41for w in ("happiness", "unhappy", "table"):
42 shared = set(table.row_ids("unhappiness")) & set(table.row_ids(w))
43 print(w, len(shared), round(float(cos(q, table.vector(w))), 3))
44# happiness 26 0.778
45# unhappy 14 0.499
46# table 0 -0.118 (random rows: unrelated words only cancel out on average)
47
48# The official library (pip install fasttext), not run here:
49# import fasttext
50# m = fasttext.train_unsupervised("corpus.txt", model="skipgram", dim=300, minn=3, maxn=6)
51# m.get_subwords("where") # the n-grams and their row ids
52# m.get_word_vector("unhappiness") # works for any string, in vocabulary or not
53# clf = fasttext.train_supervised("train.txt", wordNgrams=2) # lines look like "__label__pos great film"
54# clf.predict("a surprisingly good movie")
fastText's n-gram extraction and hashing in numpy, with an OOV vector composed from shared rows (run with numpy; the rows are random, not trained). The commented lines show the official library.

The fastText classifier

−1N∑n=1Nyn⊤log⁡(softmax(B A xn))-\frac1N\sum_{n=1}^{N} y_n^{\top}\log\big(\mathrm{softmax}(B\,A\,x_n)\big)
xnx_n
normalised bag of features of document nn: its words and, optionally, hashed word bigrams
AA
embedding matrix; AxnA x_n is the average of the feature embeddings, a low-rank text vector
BB
linear classifier from the text vector to the labels
yny_n
one-hot label; with many classes a hierarchical softmax over a Huffman tree cuts the cost from O(kh) to O(h log₂ k)

Joulin et al. showed that this shallow model matches much deeper convolutional classifiers on standard sentiment and topic benchmarks while training orders of magnitude faster. Word order enters only through hashed bigrams, which the paper adds with the same hashing trick as the subword model. The model is a strong baseline that runs on a CPU. It also underlies practical tools such as fastText's language-identification model and many data-filtering classifiers used to build LLM pre-training corpora (see Pretraining and continued pretraining). A follow-up, "FastText.zip" (Joulin et al. 2016), compresses such models with product quantisation so they fit on small devices.

Use when

  • +Vectors for any string, including misspellings, inflections and new compounds
  • +Better vectors for rare words and morphologically rich languages
  • +Fixed, predictable memory through hashing; no n-gram vocabulary to build
  • +The classifier trains on a billion words in minutes on a CPU and is a strong baseline

Avoid when

  • −Still one static vector per word type: no handling of polysemy (see Contextual embeddings)
  • −Large models: 2M bucket rows at d = 300 add 2.4 GB on top of the word rows
  • −Hash collisions make unrelated n-grams share vectors when buckets are few
  • −Surface similarity can mislead: names and opaque words get vectors from accidental letter overlap

Where it's used

Embeddings for many languages

Pre-trained vectors for 157 languages are a common starting point where no large contextual model exists.

Noisy text

Social media and search queries, where misspellings and creative spellings are everywhere.

Fast classification and filtering

Language identification, spam and topic labelling, and quality filters for web-scale pre-training data.

Check yourself

Why does fastText pad words with < and > before extracting n-grams?

Check yourself

"unhappiness" never appeared in training. How does fastText produce its vector?

Remember

  • ✓A word's vector is ∑g∈Gwzg\sum_{g\in\mathcal G_w}\mathbf z_g: the padded word <w> plus all its character n-grams with 3≤n≤63\le n\le6, scored against context vectors with skip-gram negative sampling.
  • ✓N-grams are hashed with FNV-1a into K=2⋅106K=2\cdot10^6 buckets, the rows after the ∣V∣|V| word rows, so memory is fixed and collisions are rare.
  • ✓Out-of-vocabulary words get vectors from their n-grams alone; this helps most for rare words and morphologically rich languages.
  • ✓The fastText classifier averages word and hashed bigram embeddings into a linear (hierarchical) softmax and trains in minutes on a CPU.
  • ✓Load the .bin model, not the .vec file, if you need vectors for unseen words.
← back to Deep Learning roadmap