Definition
FastText is two related methods from Facebook AI Research. The first is the subword word-embedding model of Bojanowski, Grave, Joulin and Mikolov, "Enriching Word Vectors with Subword Information" (TACL 2017, arXiv 2016). It extends skip-gram so that a word's vector is the sum of the vectors of its character n-grams plus a vector for the word itself. The second is the fastText text classifier of Joulin, Grave, Bojanowski and Mikolov, "Bag of Tricks for Efficient Text Classification" (EACL 2017). It averages word and word-n-gram embeddings into one text vector and feeds a linear softmax. Both ship in the open-source fastText library.
Word2Vec and GloVe give every word type its own vector and treat the type as an atom. Nothing ties "run", "runs", "running" and "runner" together except the contexts they happen to appear in. That has two costs. Rare inflections get poor vectors because they have few contexts. Words absent from the training vocabulary get no vector at all. English hides the problem fairly well. It is severe in morphologically rich languages such as Finnish, Turkish, Czech or German, where one lemma can have dozens of surface forms and compounds are formed freely.
Bojanowski et al. kept the skip-gram objective and changed only how a word is represented. The vector of a word is built from pieces shared with other words. "unhappiness" shares "happ", "ness" and "<un" with words seen in training, so it gets a sensible vector even if it never occurred. Earlier work had composed word vectors from morphemes, which needs a morphological analyser. Character n-grams need nothing beyond the string.
Character n-grams with boundary markers
Each word is padded with < at the start and > at the end. The set contains every character n-gram of the padded string with , plus the padded word itself as a special sequence. For "where" that gives <wh, whe, her, ere, re> at , then the 4-, 5- and 6-grams, and finally <where>. The markers matter. The trigram her inside "where" is a different string from the word "her", which is <her>. Prefixes such as <un and suffixes such as ess> become distinct units from the same letters in the middle of a word.
- word length in characters; the padded string has
- number of n-grams of length in the padded string
- the whole word
<w>, which keeps its own vector when it is in the vocabulary
Scoring and the training objective
- input vector of n-gram (or word) ; a row of the input matrix
- output vector of context word ; contexts stay whole words
- context positions around ; the window size is sampled uniformly from 1 to 5
- 5 negative words sampled by frequency; the fastText code draws them with probability proportional to the square root of the count
- binary logistic loss: push true pairs up, sampled pairs down
The only change from skip-gram with negative sampling is the first sum. A word's input vector is no longer a free parameter. Every gradient step on "running" also updates the rows for run, unn and ing>, which "runner", "sunning" and "playing" share. Rare words borrow statistical strength from frequent words with the same pieces.
Hashing n-grams into buckets
The number of distinct n-grams in a large corpus runs into the hundreds of millions, so they are not given a vocabulary. Instead each n-gram is hashed with the 32-bit FNV-1a function and reduced modulo . The input matrix has word rows followed by bucket rows, and n-gram uses row . Two different n-grams can land in the same bucket and then share a vector. With in the millions such collisions are rare and act as mild noise. The memory cost is fixed and known in advance: at in float32, the bucket rows alone take bytes GB.
- the -th UTF-8 byte of ; the code sign-extends it from int8 first, which only matters for non-ASCII bytes
- bitwise XOR
- the
-bucketoption; 2,000,000 by default
Vectors for out-of-vocabulary words
Because the n-gram rows exist independently of the vocabulary, any string has a vector. For an out-of-vocabulary word fastText sums (in the code, averages) the rows of its n-grams and leaves out the word row it does not have. The paper calls this model sisg. Its ablation sisg- gives OOV words a null vector instead, and sisg is better on rare-word similarity. A misspelling such as "happpiness" or a new compound lands near the words whose pieces it shares.
The vector says nothing about meaning beyond what the pieces carry. "Unhappiness" works because happ and ness> carry meaning. A name or an opaque loanword gets a vector dominated by accidental letter overlap. Used on its own, a subword vector is a good guess for inflected and derived forms and a weak one for everything else.
Settings in the papers and the library
- Subword model
- skip-gram with negative sampling, 5 negatives, window sampled from 1 to 5, subsampling threshold , words with fewer than 5 occurrences dropped, , n-grams , buckets
- Training data
- Wikipedia dumps in nine languages (Arabic, Czech, German, English, Spanish, French, Italian, Romanian, Russian) in the 2017 paper
- Library defaults
-minn 3 -maxn 6 -bucket 2000000 -neg 5 -ws 5 -minCount 5 -t 1e-4;dimdefaults to 100- Released vectors
- Wikipedia vectors for 294 languages; English
crawl-300d-2M-subword(Mikolov et al., LREC 2018, 600B Common Crawl tokens, 2M words); Common Crawl + Wikipedia vectors for 157 languages (Grave et al., LREC 2018: CBOW with position weights, , character 5-grams, window 5, 10 negatives) - Classifier
- text vector = mean of word (and hashed word n-gram) embeddings, then a linear softmax or hierarchical softmax; hidden units, 5 epochs, learning rate tuned on validation in the paper
- Classifier results
- AG News accuracy 91.5% with unigrams and 92.5% with bigrams, comparable to or better than the char-CNN and VDCNN baselines; the paper trains on more than a billion words in under ten minutes on a multicore CPU
Worked example: from "where" to a row index, and an OOV vector
| n | n-grams | count |
|---|---|---|
| 3 | <wh, whe, her, ere, re> | 5 |
| 4 | <whe, wher, here, ere> | 4 |
| 5 | <wher, where, here> | 3 |
| 6 | <where, where> | 2 |
| word | <where> | 1 |
Rows, hashes and one similarity
- 1"where" has , so the padded string has 7 characters and there are n-grams, matching . With the word itself, .
- 2Hash
<wh: FNV-1a gives , and . The n-gram uses row . - 3Markers keep units apart: the trigram
herhashes to bucket , the word<her>to bucket . - 4"unhappiness" () is out of vocabulary: n-gram rows and no word row. "happiness" () is in vocabulary: n-gram rows plus its word row, 31 rows. They share 26 n-grams, such as
hap,happi,inessandess>. - 5If all rows were orthonormal, the cosine of the two averaged vectors would be . With random 300-d rows (the figure) it is ; the difference is crosstalk of size about .
1import numpy as np23def fnv1a(s: str) -> int:4 """fastText's hash: 32-bit FNV-1a over UTF-8 bytes, each byte sign-extended from int8."""5 h = 21661362616 for b in s.encode("utf-8"):7 h ^= b | 0xFFFFFF00 if b >= 128 else b8 h = (h * 16777619) & 0xFFFFFFFF9 return h1011def subwords(word: str, minn: int = 3, maxn: int = 6) -> list[str]:12 w = f"<{word}>"13 return [w[i:i + n] for i in range(len(w)) for n in range(minn, maxn + 1) if i + n <= len(w)]1415class SubwordTable:16 def __init__(self, vocab, dim=300, bucket=2_000_000, minn=3, maxn=6, seed=0):17 self.word2id = {w: i for i, w in enumerate(vocab)}18 self.bucket, self.minn, self.maxn, self.dim = bucket, minn, maxn, dim19 self.rng, self.rows = np.random.default_rng(seed), {} # allocate only the rows we touch2021 def row_ids(self, word):22 ids = [self.word2id[word]] if word in self.word2id else [] # OOV: no word row23 nw = len(self.word2id)24 return ids + [nw + fnv1a(g) % self.bucket for g in subwords(word, self.minn, self.maxn)]2526 def row(self, i): # fastText initialises rows uniformly in ±1/dim27 if i not in self.rows:28 self.rows[i] = self.rng.uniform(-1 / self.dim, 1 / self.dim, self.dim)29 return self.rows[i]3031 def vector(self, word): # the library averages; the paper writes the sum32 return np.mean([self.row(i) for i in self.row_ids(word)], axis=0)3334print(subwords("where"))35print(len(subwords("where")), len(subwords("unhappiness"))) # 14 3836print([fnv1a(g) % 2_000_000 for g in ("<wh", "her", "<her>")]) # [167652, 1473420, 972692]3738table = SubwordTable(["happiness", "unhappy", "table"])39cos = lambda a, b: a @ b / np.linalg.norm(a) / np.linalg.norm(b)40q = table.vector("unhappiness") # out of vocabulary41for w in ("happiness", "unhappy", "table"):42 shared = set(table.row_ids("unhappiness")) & set(table.row_ids(w))43 print(w, len(shared), round(float(cos(q, table.vector(w))), 3))44# happiness 26 0.77845# unhappy 14 0.49946# table 0 -0.118 (random rows: unrelated words only cancel out on average)4748# The official library (pip install fasttext), not run here:49# import fasttext50# m = fasttext.train_unsupervised("corpus.txt", model="skipgram", dim=300, minn=3, maxn=6)51# m.get_subwords("where") # the n-grams and their row ids52# m.get_word_vector("unhappiness") # works for any string, in vocabulary or not53# clf = fasttext.train_supervised("train.txt", wordNgrams=2) # lines look like "__label__pos great film"54# clf.predict("a surprisingly good movie")The fastText classifier
- normalised bag of features of document : its words and, optionally, hashed word bigrams
- embedding matrix; is the average of the feature embeddings, a low-rank text vector
- linear classifier from the text vector to the labels
- one-hot label; with many classes a hierarchical softmax over a Huffman tree cuts the cost from O(kh) to O(h log₂ k)
Joulin et al. showed that this shallow model matches much deeper convolutional classifiers on standard sentiment and topic benchmarks while training orders of magnitude faster. Word order enters only through hashed bigrams, which the paper adds with the same hashing trick as the subword model. The model is a strong baseline that runs on a CPU. It also underlies practical tools such as fastText's language-identification model and many data-filtering classifiers used to build LLM pre-training corpora (see Pretraining and continued pretraining). A follow-up, "FastText.zip" (Joulin et al. 2016), compresses such models with product quantisation so they fit on small devices.
Use when
- +Vectors for any string, including misspellings, inflections and new compounds
- +Better vectors for rare words and morphologically rich languages
- +Fixed, predictable memory through hashing; no n-gram vocabulary to build
- +The classifier trains on a billion words in minutes on a CPU and is a strong baseline
Avoid when
- −Still one static vector per word type: no handling of polysemy (see Contextual embeddings)
- −Large models: 2M bucket rows at d = 300 add 2.4 GB on top of the word rows
- −Hash collisions make unrelated n-grams share vectors when buckets are few
- −Surface similarity can mislead: names and opaque words get vectors from accidental letter overlap
Where it's used
Embeddings for many languages
Pre-trained vectors for 157 languages are a common starting point where no large contextual model exists.
Noisy text
Social media and search queries, where misspellings and creative spellings are everywhere.
Fast classification and filtering
Language identification, spam and topic labelling, and quality filters for web-scale pre-training data.
Check yourself
Why does fastText pad words with < and > before extracting n-grams?
Check yourself
"unhappiness" never appeared in training. How does fastText produce its vector?
Remember
- ✓A word's vector is : the padded word
<w>plus all its character n-grams with , scored against context vectors with skip-gram negative sampling. - ✓N-grams are hashed with FNV-1a into buckets, the rows after the word rows, so memory is fixed and collisions are rare.
- ✓Out-of-vocabulary words get vectors from their n-grams alone; this helps most for rare words and morphologically rich languages.
- ✓The fastText classifier averages word and hashed bigram embeddings into a linear (hierarchical) softmax and trains in minutes on a CPU.
- ✓Load the .bin model, not the .vec file, if you need vectors for unseen words.