Definition
Tokenization is the fixed, deterministic map from a string to a sequence of integer ids drawn from a vocabulary of size , together with its inverse. A language model only ever sees those ids (see Language modeling). Its embedding table has rows, its softmax has outputs, and its context window is counted in tokens. Modern LLMs use subword vocabularies learned from data. They include Byte-pair encoding (BPE) (Sennrich, Haddow and Birch, ACL 2016), WordPiece (Schuster and Nakajima, ICASSP 2012) and the Unigram language-model tokenizer (Kudo, ACL 2018). They are usually trained with tools like SentencePiece (Kudo and Richardson, EMNLP 2018) or Hugging Face's tokenizers library.
Every tokenizer trades vocabulary size against sequence length. Word-level vocabularies, as in early neural translation and word2vec, give short sequences. But any word missing from the list becomes one <unk> token, and real text has an endless tail of names, numbers, typos, inflections and compounds. A 50,000-word vocabulary still misses words, and every new word costs a full embedding row. Character-level models never meet an unknown word, but sequences get 4–5 times longer. Attention cost grows quadratically with length, and the model must spend capacity learning to spell.
Subword tokenization was the compromise that won. Frequent words get one token (Ġthe, Ġmodel), and rare words are split into reusable pieces (Ġtoken izer s), so nothing is unknown when the base symbols cover every character or byte. The idea came from speech recognition and translation: WordPiece for Japanese and Korean voice search (2012) and Google's neural translation system (Wu et al., 2016), and BPE for translating rare words (2016). GPT-2 (2019) made BPE byte-level. Since then every major LLM has used a subword vocabulary of 32k–256k tokens.
The pipeline
A tokenizer is more than its vocabulary. The Hugging Face tokenizers library makes the stages explicit, and most tokenizers fit the same mould:
From string to ids
- 1Normalization. Clean the string with Unicode normalization (NFC or NFKC, which folds "fi" to "fi" and full-width letters to ASCII), optional lowercasing and accent stripping (BERT-uncased does both), and whitespace cleanup. Whatever is normalized away cannot be recovered by decoding.
- 2Pre-tokenization. Split the text into chunks that merges will never cross. Options are whitespace and punctuation (BERT), GPT-2's regex of letter runs, digit runs, symbol runs and contractions, splitting every digit (Llama 1 and 2), or none at all. SentencePiece treats the input as a raw stream and turns spaces into a visible
▁(U+2581). - 3Model. Segment each chunk with the learned vocabulary: replay BPE merges, take the greedy longest WordPiece match, or find the most probable Unigram segmentation with Viterbi.
- 4Post-processing. Add special tokens such as
[CLS]/[SEP](BERT),<s>/</s>(Llama, T5) or chat-template markers, then map pieces to ids. - 5Decoding. Invert the steps: ids to pieces, join, then undo the space marker (
Ġ,▁or##). Byte-level and SentencePiece tokenizers are lossless. With BERT-uncased the case and accents are gone for good.
The three subword algorithms
BPE builds up. It starts from characters or bytes and repeatedly merges the most frequent adjacent pair. The vocabulary is the base alphabet plus the merges, and encoding replays the merges in order. GPT-2/3/4, RoBERTa, Llama and Mistral use it. See Byte-pair encoding (BPE) for a step-by-step run.
WordPiece also builds up, but it scores a pair by how much merging it raises the likelihood of the training data under a unigram model. That reduces to the pair count normalised by the counts of its parts. It prefers pieces whose parts rarely occur apart, not just frequent ones. Encoding takes the longest vocabulary match from the left, marks word-internal pieces with ##, and emits [UNK] for the whole word if a prefix cannot be matched. BERT's 30,522-entry vocabulary (bert-base-uncased) is WordPiece. See WordPiece.
- count of the adjacent pair in the training words, weighted by word frequency
- WordPiece's normaliser: a pair whose parts are common anyway scores lower
- all segmentations of sentence into vocabulary pieces; Unigram treats the segmentation as a latent variable and fits with EM
- the vocabulary; Unigram starts large (e.g. all frequent substrings) and removes a fixed fraction per round (Kudo keeps the best 80%), always keeping single characters
Unigram works top-down. It assigns each piece a probability, models a word as a sequence of independent pieces, and fits the probabilities with EM over all segmentations. It then prunes the pieces whose removal lowers the corpus likelihood least, and repeats until it reaches the target size. Encoding picks the most probable segmentation with Viterbi. Because it is a real probabilistic model, it can also sample segmentations during training (subword regularization), which makes models more robust. T5, ALBERT, XLNet and mBART use Unigram through SentencePiece. See Unigram tokenization and SentencePiece.
Vocabularies of real models
- BERT (2018)
- WordPiece, 30,522 entries (
bert-base-uncased), lowercasing and accent stripping,##continuation prefix - GPT-2 / GPT-3
- byte-level BPE, 50,257 tokens; regex pre-tokenization; a leading space is part of the token (
Ġthe) - T5 (2020)
- SentencePiece Unigram, 32,000 pieces, shared across English, German, French and Romanian
- Llama 1 / 2
- SentencePiece BPE, 32,000 tokens, every digit a separate token, byte fallback for unknown characters
- Llama 3 (2024)
- tiktoken-based BPE, 128K tokens (128,256 with specials); Meta reported up to 15% fewer tokens than Llama 2's tokenizer
- GPT-4 / GPT-4o
- tiktoken
cl100k_base(about 100k) ando200k_base(about 200k);cl100k_basesplits numbers into chunks of up to three digits - Gemma (2024)
- SentencePiece with a 256k vocabulary, split digits and byte fallback
Vocabulary size, fertility and fairness
- average subwords per word (Rust et al., "How Good is Your Tokenizer?", ACL 2021). It is 1.0 for a word tokenizer. Higher values mean longer sequences and more fragmented words
- bytes (or characters) per token: about 4 for English with 50k–100k BPE vocabularies, and much lower for languages the tokenizer saw little of
- Llama 3 8B: parameters per matrix, and it has two (untied input and output). With it would be
A larger vocabulary shortens sequences, so the same context window holds more text and generation needs fewer steps. The costs are a bigger embedding table and softmax, and rarer tokens that each get fewer training updates. A vocabulary trained mostly on English also splits other languages into many more pieces, and more tokens means more compute, more API cost and less effective context for the same content. Rust et al. found that monolingual tokenizers with lower fertility helped downstream performance compared with multilingual ones. That is one reason recent multilingual models moved to 128k–256k vocabularies.
Worked example: one sentence, four tokenizers
Take the sentence "tokenizers split unseen wordz": 4 words, 29 bytes, all ASCII. Build every tokenizer from the 131-word corpus in the figure. The character vocabulary has 28 symbols, the word vocabulary 86 words (each +1 for [UNK]), and the BPE vocabulary has entries.
| tokenizer | vocab | tokens | fertility | unknown | segmentation |
|---|---|---|---|---|---|
| character | 29 | 29 | 7.25 | 0 | t o k e n i z e r s ␣ s p l i t … |
| byte | 256 | 29 | 7.25 | 0 | same 29 bytes |
| word | 87 | 4 | 1.00 | 2 | tokenizers split [UNK] [UNK] |
| BPE, 0 merges | 256 | 30 | 7.50 | 0 | Ġ t o k e n … (bytes + prefix space) |
| BPE, 25 merges | 281 | 19 | 4.75 | 0 | Ġtoken i z er s Ġs p l i t Ġ u n s e en Ġwor d z |
| BPE, 50 merges | 306 | 15 | 3.75 | 0 | Ġtokenizer s Ġsp l i t Ġ u n s e en Ġwor d z |
| BPE, 120 merges | 376 | 10 | 2.50 | 0 | Ġtokenizer s Ġspl it Ġun s e en Ġword z |
Reading the table
- 1The word tokenizer is shortest (4 tokens) but loses two of four words to
[UNK]. The model cannot tell unseen from wordz, or from any other unknown word. - 2Characters and bytes are lossless but 7.25 tokens per word. For a 4,096-token context that is about 565 words, against 4,096 words for an idealised word tokenizer.
- 3BPE with 120 merges keeps every byte of information in 10 tokens. tokenizers becomes
Ġtokenizer+s, and the typo wordz becomes the knownĠword+z, so the model sees the word it resembles. - 4The prepended space costs one token at 0 merges (30 vs 29). Merges then absorb it into word-initial tokens like
Ġtoken, which is why GPT-style tokens carry their leading space. - 5Scale check: at , going from 376 to 128,256 entries adds M parameters per embedding matrix. That is the price of real-world fertility near 1.3.
1from tokenizers import Tokenizer, decoders, models, normalizers, pre_tokenizers, trainers23def build(kind, vocab_size=8000, files=("corpus.txt",)):4 if kind == "bpe": # GPT-2 style: byte-level, lossless, no unknown token5 tok = Tokenizer(models.BPE())6 tok.pre_tokenizer = pre_tokenizers.ByteLevel(add_prefix_space=True)7 tok.decoder = decoders.ByteLevel()8 trainer = trainers.BpeTrainer(vocab_size=vocab_size, special_tokens=["<|endoftext|>"],9 initial_alphabet=pre_tokenizers.ByteLevel.alphabet())10 elif kind == "wordpiece": # BERT style: lowercase, strip accents, ## continuations11 tok = Tokenizer(models.WordPiece(unk_token="[UNK]"))12 tok.normalizer = normalizers.BertNormalizer(lowercase=True)13 tok.pre_tokenizer = pre_tokenizers.BertPreTokenizer()14 tok.decoder = decoders.WordPiece()15 trainer = trainers.WordPieceTrainer(vocab_size=vocab_size,16 special_tokens=["[UNK]", "[CLS]", "[SEP]", "[PAD]", "[MASK]"])17 else: # T5 / SentencePiece style: NFKC, ▁ for spaces, Viterbi encoding18 tok = Tokenizer(models.Unigram())19 tok.normalizer = normalizers.NFKC()20 tok.pre_tokenizer = pre_tokenizers.Metaspace()21 tok.decoder = decoders.Metaspace()22 trainer = trainers.UnigramTrainer(vocab_size=vocab_size, unk_token="<unk>", special_tokens=["<unk>"])23 tok.train(list(files), trainer)24 return tok2526text = "Tokenizers split unseen wordz, and le café est très bon."27n_words = len(text.split())28for kind in ("bpe", "wordpiece", "unigram"):29 tok = build(kind)30 enc = tok.encode(text)31 print(f"{kind:9s} V={tok.get_vocab_size():5d} tokens={len(enc.ids):3d} "32 f"fertility={len(enc.ids) / n_words:.2f} lossless={tok.decode(enc.ids) == text}")33 print(" ", enc.tokens)Use when
- +Subword vocabularies cover any text with a fixed V and no unknown words (with a byte or byte-fallback base)
- +Frequent words stay whole, so sequences are short where data is rich
- +Rare and new words share pieces (Ġword + z), so the model can generalise from them
- +Training a tokenizer takes minutes and is independent of the model
Avoid when
- −The vocabulary freezes the training-data mix: under-represented languages and domains get high fertility
- −Tokens hide characters: spelling, counting letters and digit arithmetic get harder
- −Large V costs V·d parameters in the embedding and LM head, and rare tokens are undertrained
- −Brittle at the edges: leading spaces, Unicode normalization and special tokens must match training exactly
Where it's used
Pretraining
Choosing V and the training mix of the tokenizer sets sequence lengths, and so the compute per document, for the whole run.
Multilingual models
Fertility per language is the first thing to check; 128k–256k vocabularies and balanced tokenizer training data reduce the gap.
Code models
Pre-tokenization of whitespace and indentation runs matters: merging runs of spaces shrinks Python files considerably.
Cost and context budgeting
API prices and context limits are per token, so the same document costs different amounts with different tokenizers.
Check yourself
A word-level tokenizer gives 'tokenizers split unseen wordz' as 4 tokens and BPE gives 10. Why is BPE still preferred?
Check yourself
How does Unigram tokenization build its vocabulary, compared with BPE?
Remember
- ✓Pipeline: normalize → pre-tokenize → segment with a learned model → add special tokens → ids; only the segmentation model is learned.
- ✓Characters and bytes never meet unknown symbols but give 4–7 tokens per word; words give 1 but leave [UNK]s; subwords (32k–256k) sit in between.
- ✓BPE merges by pair count, WordPiece by with greedy longest match, Unigram prunes a large vocabulary by likelihood and encodes with Viterbi.
- ✓Fertility (tokens per word) and bytes per token measure tokenizer efficiency; the embedding table costs (Llama 3 8B: 525M per matrix).
- ✓A tokenizer trained mostly on one language splits others into many more tokens, so they pay more in compute, money and context.