Definition
WordPiece is a subword tokenization method introduced by Mike Schuster and Kaisuke Nakajima in "Japanese and Korean Voice Search" (ICASSP 2012). It was later used in Google's neural machine translation system (Wu et al., 2016) and in BERT (Devlin et al., 2019). Training starts from single characters. It repeatedly adds the merge of two existing units that most increases the likelihood of the training data under a language model, and stops at a fixed vocabulary size. In BERT's form, a piece that continues a word carries the prefix ##. Text is encoded one word at a time by greedy longest-match-first lookup: unaffable becomes un ##aff ##able.
Schuster and Nakajima built voice search for Japanese and Korean. Both languages are written without spaces between words and have very large character inventories, so a word-level vocabulary was impractical. Their language model needed units that are frequent enough to estimate, short enough that any input can be covered, and long enough to carry meaning. Their answer was a data-driven inventory of "word pieces", learned greedily from the training text.
The idea moved to translation in 2016. Wu et al., "Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation", used a shared wordpiece vocabulary for source and target. They reported that between 8k and 32k pieces gave good BLEU and fast decoding. GNMT marked the start of each word with an underscore, as in _J et _makers _fe ud. BERT inverted the convention. It leaves word-initial pieces bare and marks continuations with ##. That is the form most people now mean by WordPiece.
WordPiece sits between Byte-pair encoding (BPE) and Unigram tokenization. Like BPE it is bottom-up: it starts from characters and merges pairs. Like the unigram model it chooses merges by likelihood under a probabilistic model, not by raw frequency. The overview article Tokenization algorithms compares all three.
Training: likelihood-based merges
The 2012 paper describes a loop. Start with the basic characters as the inventory. Build a language model on the training data using the current inventory. Generate candidate units by combining two existing units, and add the one that increases training likelihood the most. Repeat until the vocabulary reaches its target size or the gain drops below a threshold.
Refitting a language model for every candidate is expensive. The rule taught and implemented today is a closed-form proxy: score each adjacent pair by how much more often it occurs than independence would predict.
- count of immediately followed by in the current segmentation of the corpus, weighted by word frequency
- counts of each unit on its own in the same segmentation
- total number of units; it is the same for every pair in a step, so the ranking equals the ranking by , the argument of pointwise mutual information
- ranks by alone, so it prefers frequent pairs even when and are also common apart
The denominator is the whole difference from BPE. Take two pairs that each occur 9 times. If and appear nowhere else, the score is . If also occurs 8 more times on its own, the score drops to . WordPiece merges units that almost always occur together. BPE merges whatever is most common.
Google never published BERT's vocabulary-building code. The ratio above is the form given in Hugging Face's tokenizer course. Hugging Face's own fast WordPieceTrainer actually reuses the BPE trainer: it ranks pairs by frequency and then exports the result with ## prefixes. A "WordPiece" vocabulary therefore says a lot about how text is encoded, and less about how the vocabulary was learned.
The ## prefix
Before training, each word is split into its first character and continuation characters: newest becomes n ##e ##w ##e ##s ##t. Merges keep the prefix of the left unit, so ##e + ##s gives ##es and n + ##e gives ne. The vocabulary therefore holds two versions of most strings. s can begin a word. ##s can only continue one.
This is useful. It tells the model, at no cost, whether a piece starts a word, and it lets the decoder rebuild word boundaries: delete each " ##" and join. It also has a cost. A word-initial character that never began a training word has no bare entry. In the figure below, s exists only as ##s, so slow cannot be encoded at all.
Encoding: greedy longest-match-first
WordPiece does not replay merges at inference time. It keeps only the vocabulary list, and BERT's vocab.txt is one token per line. For each pre-tokenized word, the encoder does the following:
MaxMatch for one word
- 1Set . Look up the whole remaining string , with the prefix
##if . - 2If it is missing, drop the last character and look up again. Continue until a match is found.
- 3Emit the match and move to its end. Repeat until the word is consumed.
- 4If no prefix matches at some position, emit a single
[UNK]for the whole word, not just the failing piece. Words above a length cap become[UNK]too (100 characters in Hugging Face's implementation).
The naive loop does lookups for a word of characters in the worst case. Song et al., "Fast WordPiece Tokenization" (EMNLP 2021), gave a linear-time version, LinMaxMatch. It walks a trie of the vocabulary with precomputed failure links, in the style of Aho–Corasick, so no character is re-read.
Longest-match is not the same as replaying merges. A BPE encoder applies its merge list in order, while MaxMatch only needs the final strings. On the same vocabulary the two can disagree, and the interactive figure shows cases where they do. A tokenizer is defined by its vocabulary and its encoder, so never encode with one rule a vocabulary that was trained for the other.
WordPiece in practice
- Origin
- Schuster & Nakajima, "Japanese and Korean Voice Search", ICASSP 2012: inventory grown by likelihood-maximising merges for a voice-search language model
- GNMT (2016)
- shared source/target wordpiece vocabulary; 8k–32k pieces reported as a good range; word-start marker
_ - BERT vocab
- about 30,000 tokens in the paper. Files: 30,522 for
bert-base-uncased, 28,996 forbert-base-cased, a 110k shared vocabulary for multilingual BERT - Special tokens
[PAD]=0,[UNK]=100,[CLS]=101,[SEP]=102,[MASK]=103 inbert-base-uncased, plus reserved[unused…]slots for fine-tuning- Pre-tokenization
- BasicTokenizer: Unicode cleanup, optional lowercasing and accent stripping (uncased models), split on whitespace and punctuation, spaces added around CJK characters
- Encoder
- greedy longest-match-first per word,
##on non-initial pieces, whole word →[UNK]on failure - Used by
- BERT, DistilBERT, ELECTRA and most BERT derivatives; ALBERT and T5 use SentencePiece instead
Worked example: the first merge
Use the corpus from the figure. After the initial split there are units. Their counts are l 7, ##o 7, ##w 13, ##e 17, ##r 2, n 6, ##s 9, ##t 9, and 3 each for w, ##i and ##d. The table lists every adjacent pair. The last column is the exact gain in training log-likelihood under a unigram model refitted after the merge, which is the quantity the 2012 paper maximises.
| pair | n(ab) | n(a) | n(b) | ratio score | Δ log-lik (nats) |
|---|---|---|---|---|---|
| w + ##i | 3 | 3 | 3 | 0.3333 | +12.755 |
| ##i + ##d | 3 | 3 | 3 | 0.3333 | +12.755 |
| l + ##o | 7 | 7 | 7 | 0.1429 | +23.645 |
| ##s + ##t | 9 | 9 | 9 | 0.1111 | +28.017 |
| ##o + ##w | 7 | 7 | 13 | 0.0769 | +14.673 |
| ##e + ##r | 2 | 17 | 2 | 0.0588 | +3.169 |
| n + ##e | 6 | 6 | 17 | 0.0588 | +10.195 |
| ##e + ##s | 9 | 17 | 9 | 0.0588 | +16.263 |
| ##d + ##e | 3 | 3 | 17 | 0.0588 | +4.833 |
| ##w + ##e | 8 | 13 | 17 | 0.0362 | +5.485 |
| ##e + ##w | 6 | 17 | 13 | 0.0271 | +1.223 |
Reading the table
- 1Ratio score.
wand##ioccur 3 times each, always together, so the score is . It ties with##i + ##d, and the first one in corpus order wins. The merge produceswi, and the vocabulary grows from 11 to 12. - 2BPE. It takes the most frequent pair.
##e + ##sand##s + ##tboth occur 9 times, and##e + ##scomes first, so BPE makes##es. Its ratio is only , because##eoccurs 8 more times elsewhere. - 3Exact likelihood. Before any merge the unigram log-likelihood is nats. Merging
##s + ##tremoves 9 units and raises it by 28.017 nats, the best of the 11 pairs.w + ##igains only 12.755. - 4Why they disagree. The likelihood gain grows with how often the pair occurs as well as with how strongly it is associated. The ratio keeps only the association, so it favours rare pairs that never occur apart. Running the loop to 8 merges gives
wi wid lo ##st low ##er lower ne. The figure shows each step. - 5Encoding. With those 19 entries,
lowest→low ##e ##st,newer→ne ##w ##er,wider→wid ##er, andslow→[UNK].
1from collections import Counter23def train_wordpiece(word_counts, vocab_size):4 """Greedy WordPiece training with the ratio score n(ab) / (n(a) n(b))."""5 splits = {w: [w[0]] + ["##" + c for c in w[1:]] for w in word_counts}6 vocab = sorted({s for toks in splits.values() for s in toks})7 while len(vocab) < vocab_size:8 unit, pair = Counter(), Counter()9 for w, n in word_counts.items():10 toks = splits[w]11 for t in toks:12 unit[t] += n13 for a, b in zip(toks, toks[1:]):14 pair[a, b] += n15 if not pair:16 break17 (a, b), _ = max(pair.items(), key=lambda kv: kv[1] / (unit[kv[0][0]] * unit[kv[0][1]]))18 new = a + b[2:] if b.startswith("##") else a + b19 vocab.append(new)20 for w, toks in splits.items(): # apply the merge everywhere21 out, i = [], 022 while i < len(toks):23 if i + 1 < len(toks) and toks[i] == a and toks[i + 1] == b:24 out.append(new); i += 225 else:26 out.append(toks[i]); i += 127 splits[w] = out28 return vocab2930def wordpiece_encode(word, vocab, unk="[UNK]"):31 """BERT's greedy longest-match-first encoder for one pre-tokenized word."""32 tokens, start = [], 033 while start < len(word):34 end = len(word)35 while end > start:36 piece = ("##" if start > 0 else "") + word[start:end]37 if piece in vocab:38 break39 end -= 140 else: # no prefix matched: give up on the word41 return [unk]42 tokens.append(piece)43 start = end44 return tokens4546counts = {"low": 5, "lower": 2, "newest": 6, "widest": 3}47vocab = train_wordpiece(counts, vocab_size=19) # 11 symbols + 8 merges48print(vocab[11:]) # ['wi', 'wid', 'lo', '##st', 'low', '##er', 'lower', 'ne']49for w in ["lowest", "newer", "wider", "slow"]:50 print(w, wordpiece_encode(w, set(vocab)))51# lowest ['low', '##e', '##st']52# newer ['ne', '##w', '##er']53# wider ['wid', '##er']54# slow ['[UNK]']5556# The real thing (Hugging Face transformers):57# from transformers import BertTokenizer58# tok = BertTokenizer.from_pretrained("bert-base-uncased")59# tok.tokenize("unaffable") # ['un', '##aff', '##able'], as in BERT's READMEUse when
- +Merges chosen by association (likelihood-style), not only frequency
- +Encoding needs only a word list; greedy MaxMatch is simple and has a linear-time variant
- +## marks word-internal pieces, so word boundaries are explicit
- +Proven at scale: BERT and its descendants, GNMT
Avoid when
- −Whole word → [UNK] on any unmatched character; no byte fallback
- −Depends on language-specific pre-tokenization (whitespace, punctuation, CJK rules)
- −Not lossless: case, accents and spacing are lost in uncased pipelines
- −Google's training procedure was never released; open trainers differ (Hugging Face's is frequency-based)
Check yourself
Pair A occurs 20 times, and its parts occur 100 and 50 times. Pair B occurs 4 times, and its parts occur 4 and 5 times. Which do BPE and the WordPiece ratio score pick?
Check yourself
A WordPiece vocabulary contains ##s, low, ##er and lower, but no word-initial s. What does BERT's encoder output for the word slower?
Remember
- ✓WordPiece (Schuster & Nakajima, ICASSP 2012) grows a vocabulary from characters by adding the merge that most increases training likelihood. The common proxy score is .
- ✓Unlike BPE's raw pair count, the ratio score favours pairs whose parts rarely appear apart.
- ✓BERT marks continuation pieces with
##, sosand##sare different entries. bert-base-uncased has 30,522 entries, with [CLS]=101, [SEP]=102 and [UNK]=100. - ✓Encoding is greedy longest-match-first per word, not a replay of merges. If any position fails to match, the whole word becomes [UNK].
- ✓The pipeline is lossy (lowercasing, accent stripping, whitespace splitting), which is acceptable for encoders but not for generative models.