# FastText

> FastText (Bojanowski et al. 2017) builds a word vector from the word plus its character n-grams (n = 3–6, with < > markers) hashed into 2M buckets and trained by skip-gram, so unseen words get vectors; the fastText classifier (Joulin et al. 2017) averages embeddings into a linear softmax.

Source: https://www.mlfoundry.in/learn/fasttext  
Section: Deep Learning › Large Language Models (LLMs)  
Level: core · 14 min read · updated 2026-10-13

**FastText:** is two related methods from Facebook AI Research. The first is the **subword word-embedding model** of Bojanowski, Grave, Joulin and Mikolov, "Enriching Word Vectors with Subword Information" (TACL 2017, arXiv 2016). It extends skip-gram so that a word's vector is the **sum of the vectors of its character n-grams** plus a vector for the word itself. The second is the **fastText text classifier** of Joulin, Grave, Bojanowski and Mikolov, "Bag of Tricks for Efficient Text Classification" (EACL 2017). It averages word and word-n-gram embeddings into one text vector and feeds a linear softmax. Both ship in the open-source `fastText` library.

**Word2Vec** and **GloVe** give every word type its own vector and treat the type as an atom. Nothing ties "run", "runs", "running" and "runner" together except the contexts they happen to appear in. That has two costs. Rare inflections get poor vectors because they have few contexts. Words absent from the training vocabulary get no vector at all. English hides the problem fairly well. It is severe in morphologically rich languages such as Finnish, Turkish, Czech or German, where one lemma can have dozens of surface forms and compounds are formed freely.

Bojanowski et al. kept the skip-gram objective and changed only how a word is represented. The vector of a word is built from pieces shared with other words. "unhappiness" shares "happ", "ness" and "<un" with words seen in training, so it gets a sensible vector even if it never occurred. Earlier work had composed word vectors from morphemes, which needs a morphological analyser. Character n-grams need nothing beyond the string.

## Character n-grams with boundary markers

Each word $w$ is padded with `<` at the start and `>` at the end. The set $\mathcal G_w$ contains every character n-gram of the padded string with $3\le n\le 6$, plus the padded word itself as a special sequence. For "where" that gives `<wh`, `whe`, `her`, `ere`, `re>` at $n=3$, then the 4-, 5- and 6-grams, and finally `<where>`. The markers matter. The trigram `her` inside "where" is a different string from the word "her", which is `<her>`. Prefixes such as `<un` and suffixes such as `ess>` become distinct units from the same letters in the middle of a word.

$$
|\mathcal G_w|=1+\sum_{n=3}^{6}\big(|w|+3-n\big)=1+4|w|-6\qquad(|w|\ge 4)
$$

- $|w|$: word length in characters; the padded string has $|w|+2$
- $|w|+3-n$: number of n-grams of length $n$ in the padded string
- $1$: the whole word `<w>`, which keeps its own vector when it is in the vocabulary

> Figure: Top: the subword model. "where" is padded, split into its 14 n-grams of length 3 to 6, and each n-gram is hashed with FNV-1a into one of 2,000,000 bucket rows that sit after the word rows of the input matrix. The word vector is the sum of those rows (the code averages them), and it is scored against a context vector exactly as in skip-gram. Bottom: the classifier, which averages word and hashed word-bigram embeddings and feeds a linear softmax.

## Scoring and the training objective

$$
s(w,c)=\sum_{g\in\mathcal G_w} \mathbf z_g^{\top}\mathbf v_c,\qquad \mathcal L=\sum_{t}\sum_{c\in\mathcal C_t}\Big[\log\big(1+e^{-s(w_t,c)}\big)+\sum_{n\in\mathcal N_{t,c}}\log\big(1+e^{s(w_t,n)}\big)\Big]
$$

- $\mathbf z_g$: input vector of n-gram (or word) $g$; a row of the input matrix
- $\mathbf v_c$: output vector of context word $c$; contexts stay whole words
- $\mathcal C_t$: context positions around $t$; the window size is sampled uniformly from 1 to 5
- $\mathcal N_{t,c}$: 5 negative words sampled by frequency; the fastText code draws them with probability proportional to the square root of the count
- $\log(1+e^{-s})$: binary logistic loss: push true pairs up, sampled pairs down

The only change from skip-gram with negative sampling is the first sum. A word's input vector $\mathbf u_w=\sum_{g\in\mathcal G_w}\mathbf z_g$ is no longer a free parameter. Every gradient step on "running" also updates the rows for `run`, `unn` and `ing>`, which "runner", "sunning" and "playing" share. Rare words borrow statistical strength from frequent words with the same pieces.

## Hashing n-grams into buckets

The number of distinct n-grams in a large corpus runs into the hundreds of millions, so they are not given a vocabulary. Instead each n-gram is hashed with the 32-bit **FNV-1a** function and reduced modulo $K=2{,}000{,}000$. The input matrix has $|V|$ word rows followed by $K$ bucket rows, and n-gram $g$ uses row $|V|+\mathrm{FNV}(g)\bmod K$. Two different n-grams can land in the same bucket and then share a vector. With $K$ in the millions such collisions are rare and act as mild noise. The memory cost is fixed and known in advance: at $d=300$ in float32, the bucket rows alone take $2\cdot10^6\cdot300\cdot4$ bytes $=2.4$ GB.

$$
h_0=2166136261,\qquad h_{i}=\big(h_{i-1}\oplus b_i\big)\cdot 16777619\bmod 2^{32},\qquad \mathrm{row}(g)=|V|+h_{|g|}\bmod K
$$

- $b_i$: the $i$-th UTF-8 byte of $g$; the code sign-extends it from int8 first, which only matters for non-ASCII bytes
- $\oplus$: bitwise XOR
- $K$: the `-bucket` option; 2,000,000 by default

## Vectors for out-of-vocabulary words

Because the n-gram rows exist independently of the vocabulary, any string has a vector. For an out-of-vocabulary word fastText sums (in the code, averages) the rows of its n-grams and leaves out the word row it does not have. The paper calls this model **sisg**. Its ablation **sisg-** gives OOV words a null vector instead, and sisg is better on rare-word similarity. A misspelling such as "happpiness" or a new compound lands near the words whose pieces it shares.

The vector says nothing about meaning beyond what the pieces carry. "Unhappiness" works because `happ` and `ness>` carry meaning. A name or an opaque loanword gets a vector dominated by accidental letter overlap. Used on its own, a subword vector is a good guess for inflected and derived forms and a weak one for everything else.

**Settings in the papers and the library**

- **Subword model:** skip-gram with negative sampling, 5 negatives, window sampled from 1 to 5, subsampling threshold $10^{-4}$, words with fewer than 5 occurrences dropped, $d=300$, n-grams $3\le n\le 6$, $K=2\cdot10^6$ buckets
- **Training data:** Wikipedia dumps in nine languages (Arabic, Czech, German, English, Spanish, French, Italian, Romanian, Russian) in the 2017 paper
- **Library defaults:** `-minn 3 -maxn 6 -bucket 2000000 -neg 5 -ws 5 -minCount 5 -t 1e-4`; `dim` defaults to 100
- **Released vectors:** Wikipedia vectors for 294 languages; English `crawl-300d-2M-subword` (Mikolov et al., LREC 2018, 600B Common Crawl tokens, 2M words); Common Crawl + Wikipedia vectors for 157 languages (Grave et al., LREC 2018: CBOW with position weights, $d=300$, character 5-grams, window 5, 10 negatives)
- **Classifier:** text vector = mean of word (and hashed word n-gram) embeddings, then a linear softmax or hierarchical softmax; $h=10$ hidden units, 5 epochs, learning rate tuned on validation in the paper
- **Classifier results:** AG News accuracy 91.5% with unigrams and 92.5% with bigrams, comparable to or better than the char-CNN and VDCNN baselines; the paper trains on more than a billion words in under ten minutes on a multicore CPU

> Figure: The query word is padded and split into n-grams (the chips, one row per n), and each n-gram is hashed with fastText's own FNV-1a into a bucket. Hover a chip to see the hash. There is no trained model here: every row of the input matrix is a fixed random 300-dimensional vector, so similarity can only come from shared rows, and the orange tick shows the exact prediction Σaᵢbᵢ/‖a‖‖b‖ for orthonormal rows. At the defaults ("unhappiness", n from 3 to 6, 2M buckets) the word is out of vocabulary and its vector averages 38 n-gram rows. It shares 26 rows with "happiness" (cosine 0.767, prediction 0.758), 14 with "unhappy" (0.517 vs 0.474) and 6 with "kindness" (0.210 vs 0.187). Words with no shared rows stay near 0, within the ±0.06 crosstalk expected of random 300-d vectors ("runner" reads 0.079). The 157 distinct n-grams on the page fill 157 buckets, so nothing collides. Cut the buckets to 64 and they fall into 60 buckets: "running" now shares 9 rows with "unhappiness" through collisions alone and jumps to 0.425. Raise minn to 5 and the query has only 17 rows; "happiness" drops to 0.677. "somewhere" lands near "where" (0.460) and "nowhere" (0.384) through shared n-grams such as wher, here and ere>.

## Worked example: from "where" to a row index, and an OOV vector

Character n-grams of <where> with minn = 3, maxn = 6 (enumerated in Python)

| n | n-grams | count |
| --- | --- | --- |
| 3 | <wh, whe, her, ere, re> | 5 |
| 4 | <whe, wher, here, ere> | 4 |
| 5 | <wher, where, here> | 3 |
| 6 | <where, where> | 2 |
| word | <where> | 1 |

**Rows, hashes and one similarity**

1. "where" has $|w|=5$, so the padded string has 7 characters and there are $5+4+3+2=14$ n-grams, matching $4|w|-6=14$. With the word itself, $|\mathcal G_w|=15$.
2. Hash `<wh`: FNV-1a gives $1{,}048{,}167{,}652$, and $1{,}048{,}167{,}652\bmod 2{,}000{,}000=167{,}652$. The n-gram uses row $|V|+167{,}652$.
3. Markers keep units apart: the trigram `her` hashes to bucket $1{,}473{,}420$, the word `<her>` to bucket $972{,}692$.
4. "unhappiness" ($|w|=11$) is out of vocabulary: $4\cdot11-6=38$ n-gram rows and no word row. "happiness" ($|w|=9$) is in vocabulary: $30$ n-gram rows plus its word row, 31 rows. They share 26 n-grams, such as `hap`, `happi`, `iness` and `ess>`.
5. If all rows were orthonormal, the cosine of the two averaged vectors would be $26/\sqrt{38\cdot31}=0.758$. With random 300-d rows (the figure) it is $0.767$; the difference is crosstalk of size about $1/\sqrt{300}=0.058$.

```python
import numpy as np

def fnv1a(s: str) -> int:
    """fastText's hash: 32-bit FNV-1a over UTF-8 bytes, each byte sign-extended from int8."""
    h = 2166136261
    for b in s.encode("utf-8"):
        h ^= b | 0xFFFFFF00 if b >= 128 else b
        h = (h * 16777619) & 0xFFFFFFFF
    return h

def subwords(word: str, minn: int = 3, maxn: int = 6) -> list[str]:
    w = f"<{word}>"
    return [w[i:i + n] for i in range(len(w)) for n in range(minn, maxn + 1) if i + n <= len(w)]

class SubwordTable:
    def __init__(self, vocab, dim=300, bucket=2_000_000, minn=3, maxn=6, seed=0):
        self.word2id = {w: i for i, w in enumerate(vocab)}
        self.bucket, self.minn, self.maxn, self.dim = bucket, minn, maxn, dim
        self.rng, self.rows = np.random.default_rng(seed), {}   # allocate only the rows we touch

    def row_ids(self, word):
        ids = [self.word2id[word]] if word in self.word2id else []      # OOV: no word row
        nw = len(self.word2id)
        return ids + [nw + fnv1a(g) % self.bucket for g in subwords(word, self.minn, self.maxn)]

    def row(self, i):                    # fastText initialises rows uniformly in ±1/dim
        if i not in self.rows:
            self.rows[i] = self.rng.uniform(-1 / self.dim, 1 / self.dim, self.dim)
        return self.rows[i]

    def vector(self, word):              # the library averages; the paper writes the sum
        return np.mean([self.row(i) for i in self.row_ids(word)], axis=0)

print(subwords("where"))
print(len(subwords("where")), len(subwords("unhappiness")))          # 14 38
print([fnv1a(g) % 2_000_000 for g in ("<wh", "her", "<her>")])         # [167652, 1473420, 972692]

table = SubwordTable(["happiness", "unhappy", "table"])
cos = lambda a, b: a @ b / np.linalg.norm(a) / np.linalg.norm(b)
q = table.vector("unhappiness")                                        # out of vocabulary
for w in ("happiness", "unhappy", "table"):
    shared = set(table.row_ids("unhappiness")) & set(table.row_ids(w))
    print(w, len(shared), round(float(cos(q, table.vector(w))), 3))
# happiness 26 0.778
# unhappy 14 0.499
# table 0 -0.118      (random rows: unrelated words only cancel out on average)

# The official library (pip install fasttext), not run here:
# import fasttext
# m = fasttext.train_unsupervised("corpus.txt", model="skipgram", dim=300, minn=3, maxn=6)
# m.get_subwords("where")                 # the n-grams and their row ids
# m.get_word_vector("unhappiness")        # works for any string, in vocabulary or not
# clf = fasttext.train_supervised("train.txt", wordNgrams=2)   # lines look like "__label__pos great film"
# clf.predict("a surprisingly good movie")
```

fastText's n-gram extraction and hashing in numpy, with an OOV vector composed from shared rows (run with numpy; the rows are random, not trained). The commented lines show the official library.

## The fastText classifier

$$
-\frac1N\sum_{n=1}^{N} y_n^{\top}\log\big(\mathrm{softmax}(B\,A\,x_n)\big)
$$

- $x_n$: normalised bag of features of document $n$: its words and, optionally, hashed word bigrams
- $A$: embedding matrix; $A x_n$ is the average of the feature embeddings, a low-rank text vector
- $B$: linear classifier from the text vector to the labels
- $y_n$: one-hot label; with many classes a hierarchical softmax over a Huffman tree cuts the cost from O(kh) to O(h log₂ k)

Joulin et al. showed that this shallow model matches much deeper convolutional classifiers on standard sentiment and topic benchmarks while training orders of magnitude faster. Word order enters only through hashed bigrams, which the paper adds with the same hashing trick as the subword model. The model is a strong baseline that runs on a CPU. It also underlies practical tools such as fastText's language-identification model and many data-filtering classifiers used to build LLM pre-training corpora (see **Pretraining and continued pretraining**). A follow-up, "FastText.zip" (Joulin et al. 2016), compresses such models with product quantisation so they fit on small devices.

> **The .vec file has no subwords:** Pre-trained fastText vectors ship as `.vec` (text, whole words only) and `.bin` (the full model with n-gram buckets). Loading the `.vec` file into a word-vector library gives you an ordinary static embedding with no OOV support. You need the `.bin` model, via `fasttext.load_model` or gensim's `load_facebook_model`, to compose vectors for unseen words.

> **Parameter sharing, not a bigger vocabulary:** Subwords do not add information that skip-gram lacks. They change which parameters are tied. "running", "runner" and "rerunning" update a common set of rows, so evidence about one becomes evidence about the others. The gains in the paper are largest exactly where per-word evidence is scarce: rare words, small corpora and languages with many surface forms. They are smallest for frequent English words, which have enough contexts on their own.

> **From n-grams to subword tokenizers:** FastText keeps whole words as the unit of prediction and only builds *input* vectors from pieces. Modern language models go one step further and make the pieces the tokens themselves, with a learned vocabulary of a few tens of thousands of units (see **Tokenization algorithms**, **Byte-pair encoding (BPE)**, **WordPiece**). The motivation is the same: no word is ever out of vocabulary.

**Strengths**

- Vectors for any string, including misspellings, inflections and new compounds
- Better vectors for rare words and morphologically rich languages
- Fixed, predictable memory through hashing; no n-gram vocabulary to build
- The classifier trains on a billion words in minutes on a CPU and is a strong baseline

**Limitations**

- Still one static vector per word type: no handling of polysemy (see **Contextual embeddings**)
- Large models: 2M bucket rows at d = 300 add 2.4 GB on top of the word rows
- Hash collisions make unrelated n-grams share vectors when buckets are few
- Surface similarity can mislead: names and opaque words get vectors from accidental letter overlap

- **Embeddings for many languages:** Pre-trained vectors for 157 languages are a common starting point where no large contextual model exists.
- **Noisy text:** Social media and search queries, where misspellings and creative spellings are everywhere.
- **Fast classification and filtering:** Language identification, spam and topic labelling, and quality filters for web-scale pre-training data.

**Question:** Why does fastText pad words with < and > before extracting n-grams?

- A. To make every word at least six characters long
- B. So that prefixes, suffixes and short whole words are distinct from the same letters inside a word, e.g. <her> versus her in where
- C. Because FNV-1a cannot hash strings shorter than three bytes
- D. To mark which n-grams are context words

**Answer:** B. So that prefixes, suffixes and short whole words are distinct from the same letters inside a word, e.g. <her> versus her in where — Without markers the word "her" and the trigram inside "where" would be the same unit. With them, `<her>` and `her` hash to different buckets (972,692 and 1,473,420), and affix n-grams such as `<un` or `ess>` carry position information.

**Question:** "unhappiness" never appeared in training. How does fastText produce its vector?

- A. It returns the vector of the nearest vocabulary word by edit distance
- B. It returns a zero vector
- C. It averages (or sums) the rows of the word's character n-grams, found by hashing each n-gram into the bucket table
- D. It runs a forward pass of the classifier

**Answer:** C. It averages (or sums) the rows of the word's character n-grams, found by hashing each n-gram into the bucket table — The n-gram rows are trained independently of the vocabulary. An OOV word uses all of its 38 n-gram rows (for n from 3 to 6) and no word row. The paper's "sisg-" ablation, which gives OOV words a null vector, performs worse on rare words.

**Key points**

- A word's vector is $\sum_{g\in\mathcal G_w}\mathbf z_g$: the padded word `<w>` plus all its character n-grams with $3\le n\le6$, scored against context vectors with skip-gram negative sampling.
- N-grams are hashed with FNV-1a into $K=2\cdot10^6$ buckets, the rows after the $|V|$ word rows, so memory is fixed and collisions are rare.
- Out-of-vocabulary words get vectors from their n-grams alone; this helps most for rare words and morphologically rich languages.
- The fastText classifier averages word and hashed bigram embeddings into a linear (hierarchical) softmax and trains in minutes on a CPU.
- Load the .bin model, not the .vec file, if you need vectors for unseen words.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
