WordPiece

WordPiece (Schuster & Nakajima 2012) grows a subword vocabulary by merging the pair that most raises training likelihood, usually scored n(ab)/(n(a)·n(b)). BERT's 30,522-entry version marks word-internal pieces with ## and encodes each word greedily, longest match first, or as [UNK].

14 min read1455 wordscore0 viewsupdated 2026-10-13

Definition

WordPiece is a subword tokenization method introduced by Mike Schuster and Kaisuke Nakajima in "Japanese and Korean Voice Search" (ICASSP 2012). It was later used in Google's neural machine translation system (Wu et al., 2016) and in BERT (Devlin et al., 2019). Training starts from single characters. It repeatedly adds the merge of two existing units that most increases the likelihood of the training data under a language model, and stops at a fixed vocabulary size. In BERT's form, a piece that continues a word carries the prefix ##. Text is encoded one word at a time by greedy longest-match-first lookup: unaffable becomes un ##aff ##able.

Schuster and Nakajima built voice search for Japanese and Korean. Both languages are written without spaces between words and have very large character inventories, so a word-level vocabulary was impractical. Their language model needed units that are frequent enough to estimate, short enough that any input can be covered, and long enough to carry meaning. Their answer was a data-driven inventory of "word pieces", learned greedily from the training text.

The idea moved to translation in 2016. Wu et al., "Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation", used a shared wordpiece vocabulary for source and target. They reported that between 8k and 32k pieces gave good BLEU and fast decoding. GNMT marked the start of each word with an underscore, as in _J et _makers _fe ud. BERT inverted the convention. It leaves word-initial pieces bare and marks continuations with ##. That is the form most people now mean by WordPiece.

WordPiece sits between Byte-pair encoding (BPE) and Unigram tokenization. Like BPE it is bottom-up: it starts from characters and merges pairs. Like the unigram model it chooses merges by likelihood under a probabilistic model, not by raw frequency. The overview article Tokenization algorithms compares all three.

Training: likelihood-based merges

The 2012 paper describes a loop. Start with the basic characters as the inventory. Build a language model on the training data using the current inventory. Generate candidate units by combining two existing units, and add the one that increases training likelihood the most. Repeat until the vocabulary reaches its target size or the gain drops below a threshold.

Refitting a language model for every candidate is expensive. The rule taught and implemented today is a closed-form proxy: score each adjacent pair by how much more often it occurs than independence would predict.

score(a,b)=n(ab)n(a) n(b)=1N⋅p(ab)p(a) p(b),p(u)=n(u)N\text{score}(a,b)=\frac{n(ab)}{n(a)\,n(b)}=\frac{1}{N}\cdot\frac{p(ab)}{p(a)\,p(b)},\qquad p(u)=\frac{n(u)}{N}
n(ab)n(ab)
count of aa immediately followed by bb in the current segmentation of the corpus, weighted by word frequency
n(a), n(b)n(a),\,n(b)
counts of each unit on its own in the same segmentation
NN
total number of units; it is the same for every pair in a step, so the ranking equals the ranking by p(ab)/(p(a)p(b))p(ab)/(p(a)p(b)), the argument of pointwise mutual information
BPE\text{BPE}
ranks by n(ab)n(ab) alone, so it prefers frequent pairs even when aa and bb are also common apart

The denominator is the whole difference from BPE. Take two pairs that each occur 9 times. If aa and bb appear nowhere else, the score is 9/819/81. If aa also occurs 8 more times on its own, the score drops to 9/1539/153. WordPiece merges units that almost always occur together. BPE merges whatever is most common.

Google never published BERT's vocabulary-building code. The ratio above is the form given in Hugging Face's tokenizer course. Hugging Face's own fast WordPieceTrainer actually reuses the BPE trainer: it ranks pairs by frequency and then exports the result with ## prefixes. A "WordPiece" vocabulary therefore says a lot about how text is encoded, and less about how the vocabulary was learned.

The ## prefix

Before training, each word is split into its first character and continuation characters: newest becomes n ##e ##w ##e ##s ##t. Merges keep the prefix of the left unit, so ##e + ##s gives ##es and n + ##e gives ne. The vocabulary therefore holds two versions of most strings. s can begin a word. ##s can only continue one.

This is useful. It tells the model, at no cost, whether a piece starts a word, and it lets the decoder rebuild word boundaries: delete each " ##" and join. It also has a cost. A word-initial character that never began a training word has no bare entry. In the figure below, s exists only as ##s, so slow cannot be encoded at all.

TRAINING · once, offline (Schuster & Nakajima 2012; BERT vocabulary)word countslow ×5, lower ×2newest ×6, …split into symbolsn ##e ##w ##e ##s ##t## = continues a wordscore each pairn(ab) / (n(a)·n(b))∝ p(ab) / (p(a)·p(b))merge best pairnew symbol abvocab grows by 1vocab.txt30,522 entries(bert-base-uncased)recount, rescore, repeat until |V| is reachedENCODING · every input (BERT)raw textHe's unaffable!BasicTokenizerlowercase, strip accentssplit spaces + punctuationWordPiecegreedy longest matchfirst, word by wordadd specials[CLS] … [SEP]ids[CLS]=101[SEP]=102after basic splithe'sunaffable!after WordPiecehe'sun##aff##able!model input[CLS]he'sun##aff##able![SEP]longest match first on “unaffable” (each probe is a vocabulary lookup)@0 unaffable ✗ unaffabl ✗ … una ✗ un ✓@2 ##affable ✗ ##affabl ✗ … ##affa ✗ ##aff ✓@5 ##able ✓a non-initial piece is looked up with ##if no prefix matches at some position,the whole word becomes [UNK] (id 100)
fig.WordPiece in two phases. Training (top) runs once: split words into a first character and ##-prefixed continuations, score every adjacent pair by n(ab)/(n(a)·n(b)), merge the best one, recount and repeat until the vocabulary reaches its size (30,522 entries for bert-base-uncased). Encoding (bottom) runs on every input. BERT's BasicTokenizer lowercases, strips accents and splits on whitespace and punctuation. Each word is then cut greedily, longest vocabulary match first, and [CLS] (id 101) and [SEP] (id 102) are added. The trace shows the example from BERT's README: unaffable → un ##aff ##able.

Encoding: greedy longest-match-first

WordPiece does not replay merges at inference time. It keeps only the vocabulary list, and BERT's vocab.txt is one token per line. For each pre-tokenized word, the encoder does the following:

MaxMatch for one word

  1. 1Set s=0s=0. Look up the whole remaining string w[s:]w[s:], with the prefix ## if s>0s>0.
  2. 2If it is missing, drop the last character and look up again. Continue until a match is found.
  3. 3Emit the match and move ss to its end. Repeat until the word is consumed.
  4. 4If no prefix matches at some position, emit a single [UNK] for the whole word, not just the failing piece. Words above a length cap become [UNK] too (100 characters in Hugging Face's implementation).

The naive loop does O(n2)O(n^2) lookups for a word of nn characters in the worst case. Song et al., "Fast WordPiece Tokenization" (EMNLP 2021), gave a linear-time version, LinMaxMatch. It walks a trie of the vocabulary with precomputed failure links, in the style of Aho–Corasick, so no character is re-read.

Longest-match is not the same as replaying merges. A BPE encoder applies its merge list in order, while MaxMatch only needs the final strings. On the same vocabulary the two can disagree, and the interactive figure shows cases where they do. A tokenizer is defined by its vocabulary and its encoder, so never encode with one rule a vocabulary that was trained for the other.

WordPiece in practice

Origin
Schuster & Nakajima, "Japanese and Korean Voice Search", ICASSP 2012: inventory grown by likelihood-maximising merges for a voice-search language model
GNMT (2016)
shared source/target wordpiece vocabulary; 8k–32k pieces reported as a good range; word-start marker _
BERT vocab
about 30,000 tokens in the paper. Files: 30,522 for bert-base-uncased, 28,996 for bert-base-cased, a 110k shared vocabulary for multilingual BERT
Special tokens
[PAD]=0, [UNK]=100, [CLS]=101, [SEP]=102, [MASK]=103 in bert-base-uncased, plus reserved [unused…] slots for fine-tuning
Pre-tokenization
BasicTokenizer: Unicode cleanup, optional lowercasing and accent stripping (uncased models), split on whitespace and punctuation, spaces added around CJK characters
Encoder
greedy longest-match-first per word, ## on non-initial pieces, whole word → [UNK] on failure
Used by
BERT, DistilBERT, ELECTRA and most BERT derivatives; ALBERT and T5 use SentencePiece instead
WordPiece: merge the pair maximising n(ab) / (n(a)·n(b)) · 8 merges · vocab 19 = 11 symbols + 8corpus (count) and its current segmentationlow ×5lowlower ×2lowernewest ×6ne##w##e##stwidest ×3wid##e##stnext merge: top pairs by ratio scorepairn(ab)n(a)n(b)ratione + ##w6660.1667##w + ##e6690.1111##e + ##st9990.1111BPEwid + ##e3390.1111BPE would merge ##e + ##st (n = 9)merges learned, in order (each adds one vocabulary entry)1. wi2. wid3. lo4. ##st5. low6. ##er7. lower8. neencode “lowest”: at each position try the longest remaining string first, then shorten it@0 lowest lowes lowe low ✓@3 ##est ##es ##e ✓@4 ##st ✓longest-match-first (8 lookups)low##e##streplaying the merges in orderlow##e##stsame tokens
merge score
encode
fig.Live training on Sennrich et al.'s toy corpus: low ×5, lower ×2, newest ×6, widest ×3, which has 11 starting symbols. The table ranks the candidate pairs for the next merge with their counts and ratio score. On the same segmentation, the other rule's choice is marked in orange. At 0 merges WordPiece picks w + ##i (3/(3·3) = 0.3333), a pair that never occurs apart. BPE picks ##e + ##s (n = 9), whose ratio is only 9/(17·9) = 0.0588. The default of 8 WordPiece merges learns wi, wid, lo, ##st, low, ##er, lower, ne (vocab 19). The next candidate is ne + ##w (0.1667), and BPE would take ##e + ##st (n = 9). Encoding lowest takes 8 lookups: lowest ✗ lowes ✗ lowe ✗ low ✓, then ##est ✗ ##es ✗ ##e ✓, then ##st ✓, giving low ##e ##st. With 8 BPE merges the same word is low ##est (5 lookups). slow is always [UNK] after 4 failed lookups, because s never begins a training word. Longest-match and merge replay can disagree: with 12 BPE merges lowest is lowe ##s ##t by longest match but low ##est by replay, and with 12 WordPiece merges newer is newe ##r against new ##er.

Worked example: the first merge

Use the corpus from the figure. After the initial split there are N=79N=79 units. Their counts are l 7, ##o 7, ##w 13, ##e 17, ##r 2, n 6, ##s 9, ##t 9, and 3 each for w, ##i and ##d. The table lists every adjacent pair. The last column is the exact gain in training log-likelihood under a unigram model refitted after the merge, which is the quantity the 2012 paper maximises.

pairn(ab)n(a)n(b)ratio scoreΔ log-lik (nats)
w + ##i3330.3333+12.755
##i + ##d3330.3333+12.755
l + ##o7770.1429+23.645
##s + ##t9990.1111+28.017
##o + ##w77130.0769+14.673
##e + ##r21720.0588+3.169
n + ##e66170.0588+10.195
##e + ##s91790.0588+16.263
##d + ##e33170.0588+4.833
##w + ##e813170.0362+5.485
##e + ##w617130.0271+1.223
All 11 candidate pairs before the first merge (computed in Python)

Reading the table

  1. 1Ratio score. w and ##i occur 3 times each, always together, so the score is 3/(3⋅3)=0.33333/(3\cdot3)=0.3333. It ties with ##i + ##d, and the first one in corpus order wins. The merge produces wi, and the vocabulary grows from 11 to 12.
  2. 2BPE. It takes the most frequent pair. ##e + ##s and ##s + ##t both occur 9 times, and ##e + ##s comes first, so BPE makes ##es. Its ratio is only 9/(17⋅9)=0.05889/(17\cdot9)=0.0588, because ##e occurs 8 more times elsewhere.
  3. 3Exact likelihood. Before any merge the unigram log-likelihood is ∑un(u)log⁡n(u)N=−174.860\sum_u n(u)\log\frac{n(u)}{N}=-174.860 nats. Merging ##s + ##t removes 9 units and raises it by 28.017 nats, the best of the 11 pairs. w + ##i gains only 12.755.
  4. 4Why they disagree. The likelihood gain grows with how often the pair occurs as well as with how strongly it is associated. The ratio keeps only the association, so it favours rare pairs that never occur apart. Running the loop to 8 merges gives wi wid lo ##st low ##er lower ne. The figure shows each step.
  5. 5Encoding. With those 19 entries, lowest → low ##e ##st, newer → ne ##w ##er, wider → wid ##er, and slow → [UNK].
python
1from collections import Counter
2
3def train_wordpiece(word_counts, vocab_size):
4 """Greedy WordPiece training with the ratio score n(ab) / (n(a) n(b))."""
5 splits = {w: [w[0]] + ["##" + c for c in w[1:]] for w in word_counts}
6 vocab = sorted({s for toks in splits.values() for s in toks})
7 while len(vocab) < vocab_size:
8 unit, pair = Counter(), Counter()
9 for w, n in word_counts.items():
10 toks = splits[w]
11 for t in toks:
12 unit[t] += n
13 for a, b in zip(toks, toks[1:]):
14 pair[a, b] += n
15 if not pair:
16 break
17 (a, b), _ = max(pair.items(), key=lambda kv: kv[1] / (unit[kv[0][0]] * unit[kv[0][1]]))
18 new = a + b[2:] if b.startswith("##") else a + b
19 vocab.append(new)
20 for w, toks in splits.items(): # apply the merge everywhere
21 out, i = [], 0
22 while i < len(toks):
23 if i + 1 < len(toks) and toks[i] == a and toks[i + 1] == b:
24 out.append(new); i += 2
25 else:
26 out.append(toks[i]); i += 1
27 splits[w] = out
28 return vocab
29
30def wordpiece_encode(word, vocab, unk="[UNK]"):
31 """BERT's greedy longest-match-first encoder for one pre-tokenized word."""
32 tokens, start = [], 0
33 while start < len(word):
34 end = len(word)
35 while end > start:
36 piece = ("##" if start > 0 else "") + word[start:end]
37 if piece in vocab:
38 break
39 end -= 1
40 else: # no prefix matched: give up on the word
41 return [unk]
42 tokens.append(piece)
43 start = end
44 return tokens
45
46counts = {"low": 5, "lower": 2, "newest": 6, "widest": 3}
47vocab = train_wordpiece(counts, vocab_size=19) # 11 symbols + 8 merges
48print(vocab[11:]) # ['wi', 'wid', 'lo', '##st', 'low', '##er', 'lower', 'ne']
49for w in ["lowest", "newer", "wider", "slow"]:
50 print(w, wordpiece_encode(w, set(vocab)))
51# lowest ['low', '##e', '##st']
52# newer ['ne', '##w', '##er']
53# wider ['wid', '##er']
54# slow ['[UNK]']
55
56# The real thing (Hugging Face transformers):
57# from transformers import BertTokenizer
58# tok = BertTokenizer.from_pretrained("bert-base-uncased")
59# tok.tokenize("unaffable") # ['un', '##aff', '##able'], as in BERT's README
WordPiece training with the ratio score and BERT's greedy longest-match-first encoder, in plain Python. The training and encoding part was run and its output is shown in the comments. The Hugging Face line at the end was not run here; its output is the example in BERT's README.

Use when

  • +Merges chosen by association (likelihood-style), not only frequency
  • +Encoding needs only a word list; greedy MaxMatch is simple and has a linear-time variant
  • +## marks word-internal pieces, so word boundaries are explicit
  • +Proven at scale: BERT and its descendants, GNMT

Avoid when

  • −Whole word → [UNK] on any unmatched character; no byte fallback
  • −Depends on language-specific pre-tokenization (whitespace, punctuation, CJK rules)
  • −Not lossless: case, accents and spacing are lost in uncased pipelines
  • −Google's training procedure was never released; open trainers differ (Hugging Face's is frequency-based)

Check yourself

Pair A occurs 20 times, and its parts occur 100 and 50 times. Pair B occurs 4 times, and its parts occur 4 and 5 times. Which do BPE and the WordPiece ratio score pick?

Check yourself

A WordPiece vocabulary contains ##s, low, ##er and lower, but no word-initial s. What does BERT's encoder output for the word slower?

Remember

  • ✓WordPiece (Schuster & Nakajima, ICASSP 2012) grows a vocabulary from characters by adding the merge that most increases training likelihood. The common proxy score is n(ab)/(n(a)n(b))∝p(ab)/(p(a)p(b))n(ab)/(n(a)n(b))\propto p(ab)/(p(a)p(b)).
  • ✓Unlike BPE's raw pair count, the ratio score favours pairs whose parts rarely appear apart.
  • ✓BERT marks continuation pieces with ##, so s and ##s are different entries. bert-base-uncased has 30,522 entries, with [CLS]=101, [SEP]=102 and [UNK]=100.
  • ✓Encoding is greedy longest-match-first per word, not a replay of merges. If any position fails to match, the whole word becomes [UNK].
  • ✓The pipeline is lossy (lowercasing, accent stripping, whitespace splitting), which is acceptable for encoders but not for generative models.
← back to Deep Learning roadmap