Tokenization algorithms

A tokenizer maps text to ids from a fixed vocabulary: normalize, pre-tokenize, segment, add special tokens. Subword algorithms (BPE, WordPiece, Unigram) sit between characters (long sequences) and words (unknown words). Byte-level bases remove <unk>, and fertility measures tokens per word.

14 min read1317 wordscore0 viewsupdated 2026-10-13

Definition

Tokenization is the fixed, deterministic map from a string to a sequence of integer ids drawn from a vocabulary of size VV, together with its inverse. A language model only ever sees those ids (see Language modeling). Its embedding table has VV rows, its softmax has VV outputs, and its context window is counted in tokens. Modern LLMs use subword vocabularies learned from data. They include Byte-pair encoding (BPE) (Sennrich, Haddow and Birch, ACL 2016), WordPiece (Schuster and Nakajima, ICASSP 2012) and the Unigram language-model tokenizer (Kudo, ACL 2018). They are usually trained with tools like SentencePiece (Kudo and Richardson, EMNLP 2018) or Hugging Face's tokenizers library.

Every tokenizer trades vocabulary size against sequence length. Word-level vocabularies, as in early neural translation and word2vec, give short sequences. But any word missing from the list becomes one <unk> token, and real text has an endless tail of names, numbers, typos, inflections and compounds. A 50,000-word vocabulary still misses words, and every new word costs a full embedding row. Character-level models never meet an unknown word, but sequences get 4–5 times longer. Attention cost grows quadratically with length, and the model must spend capacity learning to spell.

Subword tokenization was the compromise that won. Frequent words get one token (Ġthe, Ġmodel), and rare words are split into reusable pieces (Ġtoken izer s), so nothing is unknown when the base symbols cover every character or byte. The idea came from speech recognition and translation: WordPiece for Japanese and Korean voice search (2012) and Google's neural translation system (Wu et al., 2016), and BPE for translating rare words (2016). GPT-2 (2019) made BPE byte-level. Since then every major LLM has used a subword vocabulary of 32k–256k tokens.

The pipeline

A tokenizer is more than its vocabulary. The Hugging Face tokenizers library makes the stages explicit, and most tokenizers fit the same mould:

From string to ids

  1. 1Normalization. Clean the string with Unicode normalization (NFC or NFKC, which folds "fi" to "fi" and full-width letters to ASCII), optional lowercasing and accent stripping (BERT-uncased does both), and whitespace cleanup. Whatever is normalized away cannot be recovered by decoding.
  2. 2Pre-tokenization. Split the text into chunks that merges will never cross. Options are whitespace and punctuation (BERT), GPT-2's regex of letter runs, digit runs, symbol runs and contractions, splitting every digit (Llama 1 and 2), or none at all. SentencePiece treats the input as a raw stream and turns spaces into a visible ▁ (U+2581).
  3. 3Model. Segment each chunk with the learned vocabulary: replay BPE merges, take the greedy longest WordPiece match, or find the most probable Unigram segmentation with Viterbi.
  4. 4Post-processing. Add special tokens such as [CLS]/[SEP] (BERT), <s>/</s> (Llama, T5) or chat-template markers, then map pieces to ids.
  5. 5Decoding. Invert the steps: ids to pieces, join, then undo the space marker (Ġ, ▁ or ##). Byte-level and SentencePiece tokenizers are lossless. With BERT-uncased the case and accents are gone for good.
pipeline (Hugging Face tokenizers): each stage is a separate, swappable componentraw text"Hello WORLD!"any Unicodenormalizer"hello world!"NFKC, lowercasepre-tokenizerhello|Ġworld|!spaces, punct.modelhell|o|Ġworld|!learned piecespost-processor<s> … </s>specials → idsonly the model stage is learned from data; the other stages are fixed rulesBPElearn: merge the most frequentadjacent pair, bottom-upencode: replay merges by rankGPT-2/3/4, RoBERTa, LlamaWordPiecelearn: merge the pair with thehighest c(ab) / (c(a)·c(b))encode: greedy longest-matchBERT, DistilBERT, ELECTRAUnigram LMlearn: start big, prune piecesthat least reduce likelihoodencode: Viterbi best splitT5, ALBERT, XLNetgranularity: vocabulary size against sequence lengthbytesV = 256charactersV ≈ 100s to 150ksubwordsV ≈ 32k to 256kwordsV ≥ 100k + [UNK]← longer sequences, tiny embedding table, nothing is out of vocabularyshorter sequences, huge embedding table, rare words become [UNK] →English with a 50k–100k subword vocabulary: about 4 characters per token, roughly 1.3 tokens per wordunder-represented languages and scripts: several times more tokens for the same content
fig.The tokenization pipeline and the three subword families. Only the model stage is learned. BPE merges the most frequent pair, bottom-up. WordPiece merges the pair with the highest likelihood gain, c(ab)/(c(a)·c(b)), and encodes by greedy longest match with ## continuation pieces. Unigram starts from a large candidate set, prunes the pieces whose removal costs the least likelihood, and encodes with Viterbi. The axis at the bottom is the basic trade-off: bytes and characters give long sequences and never an unknown token, words give short sequences and many unknown words, and subword vocabularies of 32k–256k sit in between.

The three subword algorithms

BPE builds up. It starts from characters or bytes and repeatedly merges the most frequent adjacent pair. The vocabulary is the base alphabet plus the merges, and encoding replays the merges in order. GPT-2/3/4, RoBERTa, Llama and Mistral use it. See Byte-pair encoding (BPE) for a step-by-step run.

WordPiece also builds up, but it scores a pair by how much merging it raises the likelihood of the training data under a unigram model. That reduces to the pair count normalised by the counts of its parts. It prefers pieces whose parts rarely occur apart, not just frequent ones. Encoding takes the longest vocabulary match from the left, marks word-internal pieces with ##, and emits [UNK] for the whole word if a prefix cannot be matched. BERT's 30,522-entry vocabulary (bert-base-uncased) is WordPiece. See WordPiece.

BPE: arg⁡max⁡(a,b)c(a,b)WordPiece: arg⁡max⁡(a,b)c(a,b)c(a) c(b)Unigram: max⁡V∑slog⁡∑x∈S(s)∏ip(xi)\text{BPE: }\arg\max_{(a,b)} c(a,b)\qquad\text{WordPiece: }\arg\max_{(a,b)}\frac{c(a,b)}{c(a)\,c(b)}\qquad\text{Unigram: }\max_{\mathcal V}\sum_{s}\log\sum_{\mathbf x\in S(s)}\prod_{i}p(x_i)
c(a,b)c(a,b)
count of the adjacent pair in the training words, weighted by word frequency
c(a) c(b)c(a)\,c(b)
WordPiece's normaliser: a pair whose parts are common anyway scores lower
S(s)S(s)
all segmentations of sentence ss into vocabulary pieces; Unigram treats the segmentation as a latent variable and fits p(x)p(x) with EM
V\mathcal V
the vocabulary; Unigram starts large (e.g. all frequent substrings) and removes a fixed fraction per round (Kudo keeps the best 80%), always keeping single characters

Unigram works top-down. It assigns each piece a probability, models a word as a sequence of independent pieces, and fits the probabilities with EM over all segmentations. It then prunes the pieces whose removal lowers the corpus likelihood least, and repeats until it reaches the target size. Encoding picks the most probable segmentation with Viterbi. Because it is a real probabilistic model, it can also sample segmentations during training (subword regularization), which makes models more robust. T5, ALBERT, XLNet and mBART use Unigram through SentencePiece. See Unigram tokenization and SentencePiece.

Vocabularies of real models

BERT (2018)
WordPiece, 30,522 entries (bert-base-uncased), lowercasing and accent stripping, ## continuation prefix
GPT-2 / GPT-3
byte-level BPE, 50,257 tokens; regex pre-tokenization; a leading space is part of the token (Ġthe)
T5 (2020)
SentencePiece Unigram, 32,000 pieces, shared across English, German, French and Romanian
Llama 1 / 2
SentencePiece BPE, 32,000 tokens, every digit a separate token, byte fallback for unknown characters
Llama 3 (2024)
tiktoken-based BPE, 128K tokens (128,256 with specials); Meta reported up to 15% fewer tokens than Llama 2's tokenizer
GPT-4 / GPT-4o
tiktoken cl100k_base (about 100k) and o200k_base (about 200k); cl100k_base splits numbers into chunks of up to three digits
Gemma (2024)
SentencePiece with a 256k vocabulary, split digits and byte fallback

Vocabulary size, fertility and fairness

fertility=#tokens#words,compression=#bytes#tokens,embedding params=V⋅d  (×2 if the LM head is untied)\text{fertility}=\frac{\#\text{tokens}}{\#\text{words}},\qquad \text{compression}=\frac{\#\text{bytes}}{\#\text{tokens}},\qquad \text{embedding params}=V\cdot d\ \ (\times2\ \text{if the LM head is untied})
fertility\text{fertility}
average subwords per word (Rust et al., "How Good is Your Tokenizer?", ACL 2021). It is 1.0 for a word tokenizer. Higher values mean longer sequences and more fragmented words
compression\text{compression}
bytes (or characters) per token: about 4 for English with 50k–100k BPE vocabularies, and much lower for languages the tokenizer saw little of
V⋅dV\cdot d
Llama 3 8B: 128,256⋅4096=525,336,576128{,}256\cdot4096=525{,}336{,}576 parameters per matrix, and it has two (untied input and output). With V=32,000V=32{,}000 it would be 131,072,000131{,}072{,}000

A larger vocabulary shortens sequences, so the same context window holds more text and generation needs fewer steps. The costs are a bigger embedding table and softmax, and rarer tokens that each get fewer training updates. A vocabulary trained mostly on English also splits other languages into many more pieces, and more tokens means more compute, more API cost and less effective context for the same content. Rust et al. found that monolingual tokenizers with lower fertility helped downstream performance compared with multilingual ones. That is one reason recent multilingual models moved to 128k–256k vocabularies.

"the model reads the tokens" · 5 words · 26 bytescharacter · 26 tokens · vocab 29 · fertility 5.20the␣model␣reads␣the␣tokensbyte (UTF-8) · 26 tokens · vocab 256 · fertility 5.20the␣model␣reads␣the␣tokensword · 5 tokens · vocab 87 · fertility 1.00themodelreadsthetokensbyte-level BPE, 50 merges · 8 tokens · vocab 306 · fertility 1.60ĠtheĠmodelĠreadsĠtheĠtokensBPE tokens per word vs merges learned (vocab = 256 + merges)024680306090120the model reads the toke…1.60 tokens/wordtokenizers split unseen …3.75 tokens/wordle café est très bon3.60 tokens/wordTHE MODEL READS THE TOKE…5.40 tokens/wordcorpus: 1.86 bytes/tokentraining stops at 120 merges(no pair occurs twice)BPE prepends a space (Ġ); char and word vocabularies come from the same corpus, +1 for [UNK]
sentence
normalizer
fig.One sentence, four tokenizers, all built from the same 131-word English corpus. A byte-level BPE is trained live on that corpus and stops at 120 merges, when no pair occurs twice. At the default 50 merges (vocabulary 306), 'the model reads the tokens' is 26 characters, 26 bytes, 5 words, and 8 BPE tokens: Ġthe|Ġmodel|Ġr|ea|ds|Ġthe|Ġtoken|s, fertility 1.60. With all 120 merges it drops to 6 tokens (1.20), and the corpus compresses from 1.0 to 2.90 bytes per token (1.86 at 50 merges). 'tokenizers split unseen wordz' shows the open-vocabulary win. The word tokenizer gives two [UNK], while BPE spells everything from pieces: 15 tokens at 50 merges and 10 at 120 (Ġtokenizer|s|Ġspl|it|Ġun|s|e|en|Ġword|z). The French sentence has two characters (é, è) missing from the character vocabulary, all 5 of its words are unknown to the word tokenizer, and BPE plateaus at 18 tokens (3.60 per word) from 50 merges on, because English merges barely apply. é and è cost two byte tokens each, but nothing is lost. The upper-case sentence stays at 27 BPE tokens (5.40 per word) at any merge count, with 22 unknown characters. Turn on the lowercase normalizer and it becomes identical to the first sentence.

Worked example: one sentence, four tokenizers

Take the sentence "tokenizers split unseen wordz": 4 words, 29 bytes, all ASCII. Build every tokenizer from the 131-word corpus in the figure. The character vocabulary has 28 symbols, the word vocabulary 86 words (each +1 for [UNK]), and the BPE vocabulary has 256+m256+m entries.

tokenizervocabtokensfertilityunknownsegmentation
character29297.250t o k e n i z e r s ␣ s p l i t …
byte256297.250same 29 bytes
word8741.002tokenizers split [UNK] [UNK]
BPE, 0 merges256307.500Ġ t o k e n … (bytes + prefix space)
BPE, 25 merges281194.750Ġtoken i z er s Ġs p l i t Ġ u n s e en Ġwor d z
BPE, 50 merges306153.750Ġtokenizer s Ġsp l i t Ġ u n s e en Ġwor d z
BPE, 120 merges376102.500Ġtokenizer s Ġspl it Ġun s e en Ġword z
Token counts and fertility (computed with the figure's code and checked in Python)

Reading the table

  1. 1The word tokenizer is shortest (4 tokens) but loses two of four words to [UNK]. The model cannot tell unseen from wordz, or from any other unknown word.
  2. 2Characters and bytes are lossless but 7.25 tokens per word. For a 4,096-token context that is about 565 words, against 4,096 words for an idealised word tokenizer.
  3. 3BPE with 120 merges keeps every byte of information in 10 tokens. tokenizers becomes Ġtokenizer + s, and the typo wordz becomes the known Ġword + z, so the model sees the word it resembles.
  4. 4The prepended space costs one token at 0 merges (30 vs 29). Merges then absorb it into word-initial tokens like Ġtoken, which is why GPT-style tokens carry their leading space.
  5. 5Scale check: at d=4096d=4096, going from 376 to 128,256 entries adds (128,256−376)⋅4096≈524(128{,}256-376)\cdot4096\approx524M parameters per embedding matrix. That is the price of real-world fertility near 1.3.
python
1from tokenizers import Tokenizer, decoders, models, normalizers, pre_tokenizers, trainers
2
3def build(kind, vocab_size=8000, files=("corpus.txt",)):
4 if kind == "bpe": # GPT-2 style: byte-level, lossless, no unknown token
5 tok = Tokenizer(models.BPE())
6 tok.pre_tokenizer = pre_tokenizers.ByteLevel(add_prefix_space=True)
7 tok.decoder = decoders.ByteLevel()
8 trainer = trainers.BpeTrainer(vocab_size=vocab_size, special_tokens=["<|endoftext|>"],
9 initial_alphabet=pre_tokenizers.ByteLevel.alphabet())
10 elif kind == "wordpiece": # BERT style: lowercase, strip accents, ## continuations
11 tok = Tokenizer(models.WordPiece(unk_token="[UNK]"))
12 tok.normalizer = normalizers.BertNormalizer(lowercase=True)
13 tok.pre_tokenizer = pre_tokenizers.BertPreTokenizer()
14 tok.decoder = decoders.WordPiece()
15 trainer = trainers.WordPieceTrainer(vocab_size=vocab_size,
16 special_tokens=["[UNK]", "[CLS]", "[SEP]", "[PAD]", "[MASK]"])
17 else: # T5 / SentencePiece style: NFKC, ▁ for spaces, Viterbi encoding
18 tok = Tokenizer(models.Unigram())
19 tok.normalizer = normalizers.NFKC()
20 tok.pre_tokenizer = pre_tokenizers.Metaspace()
21 tok.decoder = decoders.Metaspace()
22 trainer = trainers.UnigramTrainer(vocab_size=vocab_size, unk_token="<unk>", special_tokens=["<unk>"])
23 tok.train(list(files), trainer)
24 return tok
25
26text = "Tokenizers split unseen wordz, and le café est très bon."
27n_words = len(text.split())
28for kind in ("bpe", "wordpiece", "unigram"):
29 tok = build(kind)
30 enc = tok.encode(text)
31 print(f"{kind:9s} V={tok.get_vocab_size():5d} tokens={len(enc.ids):3d} "
32 f"fertility={len(enc.ids) / n_words:.2f} lossless={tok.decode(enc.ids) == text}")
33 print(" ", enc.tokens)
Train BPE, WordPiece and Unigram tokenizers on the same corpus with Hugging Face tokenizers, then compare token counts, fertility and round-trips (not run here)

Use when

  • +Subword vocabularies cover any text with a fixed V and no unknown words (with a byte or byte-fallback base)
  • +Frequent words stay whole, so sequences are short where data is rich
  • +Rare and new words share pieces (Ġword + z), so the model can generalise from them
  • +Training a tokenizer takes minutes and is independent of the model

Avoid when

  • −The vocabulary freezes the training-data mix: under-represented languages and domains get high fertility
  • −Tokens hide characters: spelling, counting letters and digit arithmetic get harder
  • −Large V costs V·d parameters in the embedding and LM head, and rare tokens are undertrained
  • −Brittle at the edges: leading spaces, Unicode normalization and special tokens must match training exactly

Where it's used

Pretraining

Choosing V and the training mix of the tokenizer sets sequence lengths, and so the compute per document, for the whole run.

Multilingual models

Fertility per language is the first thing to check; 128k–256k vocabularies and balanced tokenizer training data reduce the gap.

Code models

Pre-tokenization of whitespace and indentation runs matters: merging runs of spaces shrinks Python files considerably.

Cost and context budgeting

API prices and context limits are per token, so the same document costs different amounts with different tokenizers.

Check yourself

A word-level tokenizer gives 'tokenizers split unseen wordz' as 4 tokens and BPE gives 10. Why is BPE still preferred?

Check yourself

How does Unigram tokenization build its vocabulary, compared with BPE?

Remember

  • ✓Pipeline: normalize → pre-tokenize → segment with a learned model → add special tokens → ids; only the segmentation model is learned.
  • ✓Characters and bytes never meet unknown symbols but give 4–7 tokens per word; words give 1 but leave [UNK]s; subwords (32k–256k) sit in between.
  • ✓BPE merges by pair count, WordPiece by c(ab)/(c(a)c(b))c(ab)/(c(a)c(b)) with greedy longest match, Unigram prunes a large vocabulary by likelihood and encodes with Viterbi.
  • ✓Fertility (tokens per word) and bytes per token measure tokenizer efficiency; the embedding table costs V⋅dV\cdot d (Llama 3 8B: 525M per matrix).
  • ✓A tokenizer trained mostly on one language splits others into many more tokens, so they pay more in compute, money and context.
← back to Deep Learning roadmap