# WordPiece

> WordPiece (Schuster & Nakajima 2012) grows a subword vocabulary by merging the pair that most raises training likelihood, usually scored n(ab)/(n(a)·n(b)). BERT's 30,522-entry version marks word-internal pieces with ## and encodes each word greedily, longest match first, or as [UNK].

Source: https://www.mlfoundry.in/learn/wordpiece  
Section: Deep Learning › Large Language Models (LLMs)  
Level: core · 14 min read · updated 2026-10-13

**WordPiece:** is a subword tokenization method introduced by Mike Schuster and Kaisuke Nakajima in "Japanese and Korean Voice Search" (ICASSP 2012). It was later used in Google's neural machine translation system (Wu et al., 2016) and in **BERT** (Devlin et al., 2019). Training starts from single characters. It repeatedly adds the merge of two existing units that most increases the likelihood of the training data under a language model, and stops at a fixed vocabulary size. In BERT's form, a piece that continues a word carries the prefix `##`. Text is encoded one word at a time by **greedy longest-match-first** lookup: `unaffable` becomes `un ##aff ##able`.

Schuster and Nakajima built voice search for Japanese and Korean. Both languages are written without spaces between words and have very large character inventories, so a word-level vocabulary was impractical. Their language model needed units that are frequent enough to estimate, short enough that any input can be covered, and long enough to carry meaning. Their answer was a data-driven inventory of "word pieces", learned greedily from the training text.

The idea moved to translation in 2016. Wu et al., "Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation", used a shared wordpiece vocabulary for source and target. They reported that between 8k and 32k pieces gave good BLEU and fast decoding. GNMT marked the *start* of each word with an underscore, as in `_J et _makers _fe ud`. **BERT** inverted the convention. It leaves word-initial pieces bare and marks *continuations* with `##`. That is the form most people now mean by WordPiece.

WordPiece sits between **Byte-pair encoding (BPE)** and **Unigram tokenization**. Like BPE it is bottom-up: it starts from characters and merges pairs. Like the unigram model it chooses merges by likelihood under a probabilistic model, not by raw frequency. The overview article **Tokenization algorithms** compares all three.

## Training: likelihood-based merges

The 2012 paper describes a loop. Start with the basic characters as the inventory. Build a language model on the training data using the current inventory. Generate candidate units by combining two existing units, and add the one that increases training likelihood the most. Repeat until the vocabulary reaches its target size or the gain drops below a threshold.

Refitting a language model for every candidate is expensive. The rule taught and implemented today is a closed-form proxy: score each adjacent pair by how much more often it occurs than independence would predict.

$$
\text{score}(a,b)=\frac{n(ab)}{n(a)\,n(b)}=\frac{1}{N}\cdot\frac{p(ab)}{p(a)\,p(b)},\qquad p(u)=\frac{n(u)}{N}
$$

- $n(ab)$: count of $a$ immediately followed by $b$ in the current segmentation of the corpus, weighted by word frequency
- $n(a),\,n(b)$: counts of each unit on its own in the same segmentation
- $N$: total number of units; it is the same for every pair in a step, so the ranking equals the ranking by $p(ab)/(p(a)p(b))$, the argument of pointwise mutual information
- $\text{BPE}$: ranks by $n(ab)$ alone, so it prefers frequent pairs even when $a$ and $b$ are also common apart

The denominator is the whole difference from BPE. Take two pairs that each occur 9 times. If $a$ and $b$ appear nowhere else, the score is $9/81$. If $a$ also occurs 8 more times on its own, the score drops to $9/153$. WordPiece merges units that almost always occur together. BPE merges whatever is most common.

Google never published BERT's vocabulary-building code. The ratio above is the form given in Hugging Face's tokenizer course. Hugging Face's own fast `WordPieceTrainer` actually reuses the BPE trainer: it ranks pairs by frequency and then exports the result with `##` prefixes. A "WordPiece" vocabulary therefore says a lot about how text is *encoded*, and less about how the vocabulary was *learned*.

## The ## prefix

Before training, each word is split into its first character and continuation characters: `newest` becomes `n ##e ##w ##e ##s ##t`. Merges keep the prefix of the left unit, so `##e` + `##s` gives `##es` and `n` + `##e` gives `ne`. The vocabulary therefore holds two versions of most strings. `s` can begin a word. `##s` can only continue one.

This is useful. It tells the model, at no cost, whether a piece starts a word, and it lets the decoder rebuild word boundaries: delete each " ##" and join. It also has a cost. A word-initial character that never began a training word has no bare entry. In the figure below, `s` exists only as `##s`, so `slow` cannot be encoded at all.

> Figure: WordPiece in two phases. Training (top) runs once: split words into a first character and ##-prefixed continuations, score every adjacent pair by n(ab)/(n(a)·n(b)), merge the best one, recount and repeat until the vocabulary reaches its size (30,522 entries for bert-base-uncased). Encoding (bottom) runs on every input. BERT's BasicTokenizer lowercases, strips accents and splits on whitespace and punctuation. Each word is then cut greedily, longest vocabulary match first, and [CLS] (id 101) and [SEP] (id 102) are added. The trace shows the example from BERT's README: unaffable → un ##aff ##able.

## Encoding: greedy longest-match-first

WordPiece does not replay merges at inference time. It keeps only the vocabulary list, and BERT's `vocab.txt` is one token per line. For each pre-tokenized word, the encoder does the following:

**MaxMatch for one word**

1. Set $s=0$. Look up the whole remaining string $w[s:]$, with the prefix `##` if $s>0$.
2. If it is missing, drop the last character and look up again. Continue until a match is found.
3. Emit the match and move $s$ to its end. Repeat until the word is consumed.
4. If no prefix matches at some position, emit a single `[UNK]` for the **whole word**, not just the failing piece. Words above a length cap become `[UNK]` too (100 characters in Hugging Face's implementation).

The naive loop does $O(n^2)$ lookups for a word of $n$ characters in the worst case. Song et al., "Fast WordPiece Tokenization" (EMNLP 2021), gave a linear-time version, LinMaxMatch. It walks a trie of the vocabulary with precomputed failure links, in the style of Aho–Corasick, so no character is re-read.

Longest-match is not the same as replaying merges. A BPE encoder applies its merge list in order, while MaxMatch only needs the final strings. On the same vocabulary the two can disagree, and the interactive figure shows cases where they do. A tokenizer is defined by its vocabulary *and* its encoder, so never encode with one rule a vocabulary that was trained for the other.

**WordPiece in practice**

- **Origin:** Schuster & Nakajima, "Japanese and Korean Voice Search", ICASSP 2012: inventory grown by likelihood-maximising merges for a voice-search language model
- **GNMT (2016):** shared source/target wordpiece vocabulary; 8k–32k pieces reported as a good range; word-*start* marker `_`
- **BERT vocab:** about 30,000 tokens in the paper. Files: 30,522 for `bert-base-uncased`, 28,996 for `bert-base-cased`, a 110k shared vocabulary for multilingual BERT
- **Special tokens:** `[PAD]`=0, `[UNK]`=100, `[CLS]`=101, `[SEP]`=102, `[MASK]`=103 in `bert-base-uncased`, plus reserved `[unused…]` slots for fine-tuning
- **Pre-tokenization:** BasicTokenizer: Unicode cleanup, optional lowercasing and accent stripping (uncased models), split on whitespace and punctuation, spaces added around CJK characters
- **Encoder:** greedy longest-match-first per word, `##` on non-initial pieces, whole word → `[UNK]` on failure
- **Used by:** **BERT**, **DistilBERT**, ELECTRA and most BERT derivatives; **ALBERT** and **T5** use **SentencePiece** instead

> Figure: Live training on Sennrich et al.'s toy corpus: low ×5, lower ×2, newest ×6, widest ×3, which has 11 starting symbols. The table ranks the candidate pairs for the next merge with their counts and ratio score. On the same segmentation, the other rule's choice is marked in orange. At 0 merges WordPiece picks w + ##i (3/(3·3) = 0.3333), a pair that never occurs apart. BPE picks ##e + ##s (n = 9), whose ratio is only 9/(17·9) = 0.0588. The default of 8 WordPiece merges learns wi, wid, lo, ##st, low, ##er, lower, ne (vocab 19). The next candidate is ne + ##w (0.1667), and BPE would take ##e + ##st (n = 9). Encoding lowest takes 8 lookups: lowest ✗ lowes ✗ lowe ✗ low ✓, then ##est ✗ ##es ✗ ##e ✓, then ##st ✓, giving low ##e ##st. With 8 BPE merges the same word is low ##est (5 lookups). slow is always [UNK] after 4 failed lookups, because s never begins a training word. Longest-match and merge replay can disagree: with 12 BPE merges lowest is lowe ##s ##t by longest match but low ##est by replay, and with 12 WordPiece merges newer is newe ##r against new ##er.

## Worked example: the first merge

Use the corpus from the figure. After the initial split there are $N=79$ units. Their counts are `l` 7, `##o` 7, `##w` 13, `##e` 17, `##r` 2, `n` 6, `##s` 9, `##t` 9, and 3 each for `w`, `##i` and `##d`. The table lists every adjacent pair. The last column is the exact gain in training log-likelihood under a unigram model refitted after the merge, which is the quantity the 2012 paper maximises.

All 11 candidate pairs before the first merge (computed in Python)

| pair | n(ab) | n(a) | n(b) | ratio score | Δ log-lik (nats) |
| --- | --- | --- | --- | --- | --- |
| w + ##i | 3 | 3 | 3 | 0.3333 | +12.755 |
| ##i + ##d | 3 | 3 | 3 | 0.3333 | +12.755 |
| l + ##o | 7 | 7 | 7 | 0.1429 | +23.645 |
| ##s + ##t | 9 | 9 | 9 | 0.1111 | +28.017 |
| ##o + ##w | 7 | 7 | 13 | 0.0769 | +14.673 |
| ##e + ##r | 2 | 17 | 2 | 0.0588 | +3.169 |
| n + ##e | 6 | 6 | 17 | 0.0588 | +10.195 |
| ##e + ##s | 9 | 17 | 9 | 0.0588 | +16.263 |
| ##d + ##e | 3 | 3 | 17 | 0.0588 | +4.833 |
| ##w + ##e | 8 | 13 | 17 | 0.0362 | +5.485 |
| ##e + ##w | 6 | 17 | 13 | 0.0271 | +1.223 |

**Reading the table**

1. **Ratio score.** `w` and `##i` occur 3 times each, always together, so the score is $3/(3\cdot3)=0.3333$. It ties with `##i + ##d`, and the first one in corpus order wins. The merge produces `wi`, and the vocabulary grows from 11 to 12.
2. **BPE.** It takes the most frequent pair. `##e + ##s` and `##s + ##t` both occur 9 times, and `##e + ##s` comes first, so BPE makes `##es`. Its ratio is only $9/(17\cdot9)=0.0588$, because `##e` occurs 8 more times elsewhere.
3. **Exact likelihood.** Before any merge the unigram log-likelihood is $\sum_u n(u)\log\frac{n(u)}{N}=-174.860$ nats. Merging `##s + ##t` removes 9 units and raises it by 28.017 nats, the best of the 11 pairs. `w + ##i` gains only 12.755.
4. **Why they disagree.** The likelihood gain grows with how often the pair occurs as well as with how strongly it is associated. The ratio keeps only the association, so it favours rare pairs that never occur apart. Running the loop to 8 merges gives `wi wid lo ##st low ##er lower ne`. The figure shows each step.
5. **Encoding.** With those 19 entries, `lowest` → `low ##e ##st`, `newer` → `ne ##w ##er`, `wider` → `wid ##er`, and `slow` → `[UNK]`.

```python
from collections import Counter

def train_wordpiece(word_counts, vocab_size):
    """Greedy WordPiece training with the ratio score n(ab) / (n(a) n(b))."""
    splits = {w: [w[0]] + ["##" + c for c in w[1:]] for w in word_counts}
    vocab = sorted({s for toks in splits.values() for s in toks})
    while len(vocab) < vocab_size:
        unit, pair = Counter(), Counter()
        for w, n in word_counts.items():
            toks = splits[w]
            for t in toks:
                unit[t] += n
            for a, b in zip(toks, toks[1:]):
                pair[a, b] += n
        if not pair:
            break
        (a, b), _ = max(pair.items(), key=lambda kv: kv[1] / (unit[kv[0][0]] * unit[kv[0][1]]))
        new = a + b[2:] if b.startswith("##") else a + b
        vocab.append(new)
        for w, toks in splits.items():          # apply the merge everywhere
            out, i = [], 0
            while i < len(toks):
                if i + 1 < len(toks) and toks[i] == a and toks[i + 1] == b:
                    out.append(new); i += 2
                else:
                    out.append(toks[i]); i += 1
            splits[w] = out
    return vocab

def wordpiece_encode(word, vocab, unk="[UNK]"):
    """BERT's greedy longest-match-first encoder for one pre-tokenized word."""
    tokens, start = [], 0
    while start < len(word):
        end = len(word)
        while end > start:
            piece = ("##" if start > 0 else "") + word[start:end]
            if piece in vocab:
                break
            end -= 1
        else:                                     # no prefix matched: give up on the word
            return [unk]
        tokens.append(piece)
        start = end
    return tokens

counts = {"low": 5, "lower": 2, "newest": 6, "widest": 3}
vocab = train_wordpiece(counts, vocab_size=19)    # 11 symbols + 8 merges
print(vocab[11:])          # ['wi', 'wid', 'lo', '##st', 'low', '##er', 'lower', 'ne']
for w in ["lowest", "newer", "wider", "slow"]:
    print(w, wordpiece_encode(w, set(vocab)))
# lowest ['low', '##e', '##st']
# newer ['ne', '##w', '##er']
# wider ['wid', '##er']
# slow ['[UNK]']

# The real thing (Hugging Face transformers):
# from transformers import BertTokenizer
# tok = BertTokenizer.from_pretrained("bert-base-uncased")
# tok.tokenize("unaffable")   # ['un', '##aff', '##able'], as in BERT's README
```

WordPiece training with the ratio score and BERT's greedy longest-match-first encoder, in plain Python. The training and encoding part was run and its output is shown in the comments. The Hugging Face line at the end was not run here; its output is the example in BERT's README.

> **One unknown character costs the whole word:** When MaxMatch fails at any position, BERT emits one `[UNK]` for the entire word, not for the bad character. One rare symbol, such as an emoji, a math sign or an unusual accent in a cased model, wipes out the word around it. This is the main robustness gap compared with byte-level BPE and with **SentencePiece**'s byte fallback, which can encode any string. Check `[UNK]` rates on your own domain before fine-tuning a WordPiece model on it.

> **The score is PMI, not frequency:** $n(ab)/(n(a)n(b))$ is $p(ab)/(p(a)p(b))$ up to a constant, the argument of pointwise mutual information. WordPiece merges units that are glued together, BPE merges units that are common. In the worked example, the ratio picks a 3-count pair while the true likelihood gain favours a 9-count pair. That is the price of the cheap proxy.

> **WordPiece is not lossless:** BERT's pipeline lowercases and strips accents (uncased models), drops whitespace and splits punctuation before WordPiece runs. Decoding by joining pieces and removing " ##" gives `he ' s un ##aff ##able` → `he ' s unaffable`, not the original `He's unaffable!` with its spacing and case. Encoder models do not need to reproduce text, so this was acceptable for **BERT**. Generative models need an exact round trip, which is one reason they moved to byte-level BPE and **SentencePiece**.

**Strengths**

- Merges chosen by association (likelihood-style), not only frequency
- Encoding needs only a word list; greedy MaxMatch is simple and has a linear-time variant
- ## marks word-internal pieces, so word boundaries are explicit
- Proven at scale: BERT and its descendants, GNMT

**Limitations**

- Whole word → [UNK] on any unmatched character; no byte fallback
- Depends on language-specific pre-tokenization (whitespace, punctuation, CJK rules)
- Not lossless: case, accents and spacing are lost in uncased pipelines
- Google's training procedure was never released; open trainers differ (Hugging Face's is frequency-based)

**Question:** Pair A occurs 20 times, and its parts occur 100 and 50 times. Pair B occurs 4 times, and its parts occur 4 and 5 times. Which do BPE and the WordPiece ratio score pick?

- A. Both pick A
- B. Both pick B
- C. BPE picks A; WordPiece picks B (0.2 vs 0.004)
- D. BPE picks B; WordPiece picks A

**Answer:** C. BPE picks A; WordPiece picks B (0.2 vs 0.004) — BPE compares raw counts, 20 against 4. WordPiece compares $20/(100\cdot50)=0.004$ against $4/(4\cdot5)=0.2$, so B's parts almost never occur apart and B wins.

**Question:** A WordPiece vocabulary contains ##s, low, ##er and lower, but no word-initial s. What does BERT's encoder output for the word slower?

- A. s ##lower
- B. ##s lower
- C. [UNK] ##lower
- D. [UNK]

**Answer:** D. [UNK] — At position 0 the encoder looks for `slower`, `slowe`, …, `s` without the `##` prefix. None of them exists, because `##s` can only continue a word. When MaxMatch fails at any position, the whole word becomes a single `[UNK]`.

**Key points**

- WordPiece (Schuster & Nakajima, ICASSP 2012) grows a vocabulary from characters by adding the merge that most increases training likelihood. The common proxy score is $n(ab)/(n(a)n(b))\propto p(ab)/(p(a)p(b))$.
- Unlike BPE's raw pair count, the ratio score favours pairs whose parts rarely appear apart.
- BERT marks continuation pieces with `##`, so `s` and `##s` are different entries. bert-base-uncased has 30,522 entries, with [CLS]=101, [SEP]=102 and [UNK]=100.
- Encoding is greedy longest-match-first per word, not a replay of merges. If any position fails to match, the whole word becomes [UNK].
- The pipeline is lossy (lowercasing, accent stripping, whitespace splitting), which is acceptable for encoders but not for generative models.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
