# SentencePiece

> SentencePiece (Kudo & Richardson 2018) trains and runs a unigram or BPE subword model on raw text. It NFKC-normalizes, writes spaces as ▁ so that decoding is concatenation, and can fall back to UTF-8 byte pieces. T5 (32k, unigram) and Llama 1/2 (32k, BPE) use it.

Source: https://www.mlfoundry.in/learn/sentencepiece  
Section: Deep Learning › Large Language Models (LLMs)  
Level: core · 14 min read · updated 2026-10-13

**SentencePiece:** is a tokenizer library and model format by Taku Kudo and John Richardson, "SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing" (EMNLP 2018, system demonstrations). It treats input as a raw stream of Unicode characters with no language-specific pre-tokenization. Spaces are an ordinary symbol, written `▁` (U+2581). The library bundles four parts: a **normalizer**, a **trainer** for a subword model (**Unigram tokenization** by default, or **Byte-pair encoding (BPE)**), an **encoder** and a **decoder**. All four are fixed by one self-contained `.model` file. The guarantee is that $\mathrm{Decode}(\mathrm{Encode}(\mathrm{Normalize}(x)))=\mathrm{Normalize}(x)$.

Before 2018, subword pipelines had three stages. A language-specific tokenizer such as the Moses scripts split text into words and punctuation. A subword model such as BPE or **WordPiece** split those words further. A matching detokenizer then tried to undo the first stage with rules about which tokens take a space before them. That pipeline is language-dependent and lossy. `Hello, world!` and `Hello , world !` produce the same tokens, so the detokenizer has to guess. For Chinese, Japanese or Thai, which have no spaces, the first stage needs a separate word segmenter.

Kudo and Richardson removed the first stage. If whitespace is just another character, the subword model can learn where words begin directly from raw sentences. Decoding becomes a string operation with no rules: concatenate the pieces and turn `▁` back into spaces. The same code then works for English, Japanese and a mixed-script corpus, and the tokenizer that was trained is exactly the one that runs at inference.

## Whitespace as a symbol

The pipeline for one sentence: `Hello world.` is escaped to `▁Hello▁world.`, the model splits it into `▁Hello` `▁world` `.`, and decoding concatenates them and turns each `▁` back into a space. The symbol is U+2581 LOWER ONE EIGHTH BLOCK, a visible stand-in for U+0020. With `add_dummy_prefix=true` (the default) a `▁` is also put before the text, so a sentence-initial word gets the same piece as the same word mid-sentence. Decoding removes it again.

$$
\mathrm{Decode}\big(\mathrm{Encode}(\mathrm{Normalize}(x))\big)=\mathrm{Normalize}(x),\qquad \mathrm{Decode}(t_1\dots t_m)=\mathrm{unescape}\big(t_1\Vert t_2\Vert\cdots\Vert t_m\big)
$$

- $\mathrm{Normalize}$: the only lossy step: NFKC folding and whitespace cleanup under the default `nmt_nfkc` rules
- $t_1\Vert\cdots\Vert t_m$: string concatenation of the pieces, after byte pieces are reassembled into UTF-8 characters
- $\mathrm{unescape}$: replace every `▁` with a space and drop the dummy prefix. No language rules are involved

Every piece now records whether a space came before it. `▁world` and `world` are different vocabulary entries, and the second only occurs inside a longer word. That is exactly the information a rule-based detokenizer had to guess. The convention is the reverse of **WordPiece**'s `##`. WordPiece marks a piece that does *not* start a word, while SentencePiece marks one that does, and it marks it with the actual space character it replaces.

Two defaults limit the guarantee. `remove_extra_whitespaces=true` strips leading and trailing spaces and collapses runs, so `New  York` is normalized to `New York` before encoding. `split_by_whitespace=true` prevents a piece from spanning a `▁` in the middle, so `▁New▁York` can never become one token. Turning these off gives an exact round trip of the raw bytes, which matters for code and for whitespace-sensitive data.

> Figure: The SentencePiece pipeline. Encoding takes raw Unicode, applies the normalization rules stored in the model (default nmt_nfkc: NFKC plus whitespace cleanup), escapes each space as ▁ with a dummy prefix, and segments the stream with the unigram or BPE model. A character with no piece is either <unk> or, with byte_fallback, its UTF-8 bytes. In Llama's layout byte b has id 3 + b, so 🦙 becomes <0xF0><0x9F><0xA6><0x99> = 243, 162, 169, 156. Decoding needs no rules: map ids to pieces, join, turn byte runs back into characters, and replace ▁ with a space. One .model file carries the vocabulary, the scores and the normalization table.

## Normalization, the model and byte fallback

**Normalization.** Unicode allows the same visible text to be written in several ways, such as full-width `ｈｅｌｌｏ`, the ligature `ﬁ`, or the circled `①`. SentencePiece maps these to one form before training and encoding. The default rule set, `nmt_nfkc`, is Unicode NFKC plus a few cleanups such as whitespace normalization. Other built-in rules are `nfkc`, `nfkc_cf` (with case folding) and `identity`, and you can supply a custom TSV of rewrite rules. The rules are compiled into the `.model` file, so every user of the model normalizes identically. Normalization is the one irreversible step. `ｈｅｌｌｏ①` encodes as `hello1` and decodes as `hello1`.

**The model.** With `model_type=unigram`, the default, training runs Kudo's EM-and-prune procedure and encoding takes the Viterbi path (see **Unigram tokenization**). With `model_type=bpe` it learns merges from the raw stream (see **Byte-pair encoding (BPE)**). Both operate on the escaped stream, so `▁` takes part in the statistics like any letter. There are also `char` and `word` model types. Optional sampling (`enable_sampling`, `alpha`, `nbest_size`) implements subword regularization for unigram and BPE-dropout for BPE.

**Coverage and byte fallback.** Training keeps the characters that cover `character_coverage` of the corpus, by default 0.9995. The README recommends 1.0 for languages with small alphabets. Rare characters outside that set map to `<unk>` (id 0), which destroys information. The later option `byte_fallback=true` reserves 256 pieces `<0x00>`…`<0xFF>` and encodes any unknown character as its UTF-8 bytes. Every string is then encodable and decodable with no `<unk>` at all. Llama's tokenizer uses this, as its paper states: numbers are split into digits, and unknown UTF-8 characters fall back to bytes.

**Defaults and deployed models**

- **Paper:** Kudo & Richardson, EMNLP 2018 (demo track); C++ library with Python bindings; Apache 2.0
- **Model types:** `unigram` (default), `bpe`, `char`, `word`
- **Normalization:** `nmt_nfkc` (default), `nfkc`, `nfkc_cf`, `identity`, or custom TSV rules; `remove_extra_whitespaces=true`, `add_dummy_prefix=true`
- **Special ids:** `<unk>`=0, `<s>`=1, `</s>`=2 by default; `<pad>` disabled (id −1) unless set
- **Coverage:** `character_coverage=0.9995` by default; `byte_fallback` adds 256 byte pieces
- **T5:** one 32,000-piece unigram vocabulary shared by English, German, French and Romanian; mT5 uses 250,000 pieces
- **Llama 1 / 2:** 32,000-piece BPE model, `byte_fallback`, digits split; byte pieces are ids 3–258, so newline `<0x0A>` is id 13. Llama 3 switched to a 128K tiktoken-style BPE
- **Others:** **ALBERT** (30,000), XLNet (32,000), Gemma (256k), and many multilingual MT systems

> Figure: A toy SentencePiece model with 336 ids, laid out like Llama's: 3 specials, 256 byte pieces (ids 3–258) and 77 pieces with unigram scores, decoded by Viterbi. The default input Hello, world! becomes ▁Hello,▁world! and then 4 pieces, ▁Hello(260) ,(275) ▁world(262) !(277), which decode back to the input exactly. The whitespace-split baseline gives [Hello] [,] [world] [!], which joins to Hello , world ! and has lost the spacing. Turn off add_dummy_prefix and the first word becomes He·ll·o (6 tokens), because Hello without ▁ is not a piece. For New␣␣York, nmt_nfkc collapses the double space to give ▁New ▁York. With normalization off you get ▁New ▁ ▁York, which round-trips the raw text exactly. Full-width ｈｅｌｌｏ① (6 code points, 18 UTF-8 bytes) normalizes to hello1, which is 2 tokens. Unnormalized it costs ▁ plus 18 byte tokens. I ♥ 🦙 needs 10 tokens: the heart is 229 156 168 and the llama is 243 162 169 156. Without byte_fallback each becomes <unk> (id 0) and decodes to ⁇, so the round trip fails. 東京に住む needs no spaces: ▁東京 に 住む, while the whitespace baseline sees one 5-character word.

## Worked example: encoding and decoding one string

Take `I ♥ 🦙` with the figure's toy model and the default options. It is 5 code points and 10 UTF-8 bytes.

Each stage for “I ♥ 🦙” (checked against a Python re-implementation)

| stage | result |
| --- | --- |
| normalize (nmt_nfkc) | I ♥ 🦙 (unchanged: both symbols are already NFKC) |
| escape + dummy prefix | ▁I▁♥▁🦙 |
| Viterbi pieces | ▁I · ▁ · ♥ (unknown) · ▁ · 🦙 (unknown) |
| byte fallback | ♥ = E2 99 A5, 🦙 = F0 9F A6 99 |
| tokens | ▁I ▁ <0xE2> <0x99> <0xA5> ▁ <0xF0> <0x9F> <0xA6> <0x99> (10) |
| byte ids (3 + b) | 229 156 168 · 243 162 169 156 |
| decode | ▁I▁ + bytes→♥ + ▁ + bytes→🦙 = ▁I▁♥▁🦙 → “I ♥ 🦙” |

**What each step guarantees**

1. **Escape.** The dummy prefix makes the string start with `▁`, so `I` is matched as `▁I`, the same piece it would be mid-sentence.
2. **Segment.** ♥ and 🦙 have no pieces. The unigram model gives each one unknown arc with score $\min(\text{scores})-10$, so the Viterbi path isolates them and keeps the `▁` before each as its own piece.
3. **Fallback.** Each unknown character is replaced by its UTF-8 bytes. A 3-byte and a 4-byte character cost 7 tokens. That is the price of never emitting `<unk>`.
4. **Decode.** Byte runs are reassembled into UTF-8 before `▁` is mapped to a space, so the output is exactly `I ♥ 🦙`. With `byte_fallback=false` the tokens are `▁I ▁ <unk> ▁ <unk>`, and each `<unk>` decodes to " ⁇ ". The emoji is gone and the round trip fails.
5. **Contrast.** A whitespace pre-tokenizer would give `[Hello] [,] [world] [!]` for `Hello, world!`. Joining with spaces produces `Hello , world !`. SentencePiece's `▁Hello , ▁world !` already contains the answer: `,` and `!` have no `▁`, so no space goes before them.

```python
# --- the library ----------------------------------------------------------
# import sentencepiece as spm
# spm.SentencePieceTrainer.train(
#     input="corpus.txt", model_prefix="sp", vocab_size=32000,
#     model_type="unigram",            # or "bpe" (Llama 1/2)
#     character_coverage=0.9995,       # 1.0 for small alphabets
#     normalization_rule_name="nmt_nfkc",
#     byte_fallback=True, split_digits=True)
# sp = spm.SentencePieceProcessor(model_file="sp.model")
# ids = sp.encode("Hello, world!")                 # list of ints
# pieces = sp.encode("Hello, world!", out_type=str)  # e.g. ['▁Hello', ',', '▁world', '!']
# assert sp.decode(ids) == "Hello, world!"

# --- what it does, in miniature ----------------------------------------------
import math, re, unicodedata

SP = "▁"                                         # U+2581, the whitespace meta symbol

def normalize(text):                            # roughly nmt_nfkc: NFKC + whitespace cleanup
    return re.sub(" +", " ", unicodedata.normalize("NFKC", text)).strip()

def escape(text, dummy_prefix=True):
    return (" " + text if dummy_prefix else text).replace(" ", SP)

def encode(text, scores, byte_fallback=True):
    s = escape(normalize(text))
    unk = min(scores.values()) - 10             # SentencePiece's penalty for an unknown character
    best, back = [0.0] + [-math.inf] * len(s), [0] * (len(s) + 1)
    for j in range(1, len(s) + 1):              # Viterbi over code points
        for i in range(max(0, j - 16), j):
            if s[i:j] in scores and best[i] + scores[s[i:j]] > best[j]:
                best[j], back[j] = best[i] + scores[s[i:j]], i
        if s[j - 1] not in scores and best[j - 1] + unk > best[j]:
            best[j], back[j] = best[j - 1] + unk, j - 1
    pieces, j = [], len(s)
    while j > 0:
        pieces.append(s[back[j]:j]); j = back[j]
    out = []
    for p in reversed(pieces):
        if p in scores:
            out.append(p)
        elif byte_fallback:                     # unknown character → its UTF-8 bytes
            out += [f"<0x{b:02X}>" for b in p.encode("utf-8")]
        else:
            out.append("<unk>")
    return out

def decode(pieces):
    buf, text = bytearray(), ""
    for p in pieces + [""]:
        if re.fullmatch(r"<0x[0-9A-F]{2}>", p):
            buf.append(int(p[3:5], 16)); continue
        text += buf.decode("utf-8") + (" ⁇ " if p == "<unk>" else p); buf.clear()
    text = text.replace(SP, " ")
    return text[1:] if text.startswith(" ") else text

scores = {SP: -3.0, SP + "Hello": -8.0, SP + "world": -8.2, SP + "I": -5.2, ",": -3.6, "!": -4.6,
          SP + "hello": -8.4, "1": -5.6}
for x in ["Hello, world!", "I ♥ 🦙", "ｈｅｌｌｏ①"]:
    toks = encode(x, scores)
    print(toks, repr(decode(toks)), decode(toks) == normalize(x))
# ['▁Hello', ',', '▁world', '!'] 'Hello, world!' True
# ['▁I', '▁', '<0xE2>', '<0x99>', '<0xA5>', '▁', '<0xF0>', '<0x9F>', '<0xA6>', '<0x99>'] 'I ♥ 🦙' True
# ['▁hello', '1'] 'hello1' True
print(encode("I ♥ 🦙", scores, byte_fallback=False))
# ['▁I', '▁', '<unk>', '▁', '<unk>']
```

Top: the SentencePiece API (commented out, not run here). Bottom: a small pure-Python re-implementation of normalization, escaping, Viterbi with unknown-character arcs, byte fallback and decoding. That part was run, and its output is in the comments.

> **Lossless means lossless after normalization:** The guarantee is $\mathrm{Decode}(\mathrm{Encode}(\mathrm{Normalize}(x)))=\mathrm{Normalize}(x)$, not $=x$. With the defaults, double spaces collapse, leading and trailing spaces vanish, and NFKC rewrites full-width letters, ligatures and compatibility characters. For source code, tables or anything whitespace-sensitive, use `identity` normalization and `remove_extra_whitespaces=false`. Also enable `byte_fallback`, or `<unk>` will silently drop characters.

> **The space is part of the token:** Because `▁` is a character, "word starts here" is learned rather than imposed. The model can learn `▁New`, `York` and `▁York` as different units, and a language without spaces simply never uses `▁` mid-sentence. Detokenization needs no knowledge of punctuation or script, so one tokenizer serves a multilingual model, as in mT5's 250,000-piece vocabulary.

> **SentencePiece is not an algorithm:** "SentencePiece tokenizer" names a *library and format*, not a segmentation method. The model inside is unigram or BPE. T5's and ALBERT's vocabularies are unigram, and Llama 1/2's is BPE. When comparing tokenizers, ask which model type, which normalization, whether byte fallback is on, and whether digits are split.

**Strengths**

- Language-independent: trains on raw sentences, no word segmenter needed for CJK or Thai
- Lossless round trip of the normalized text; decoding is plain concatenation
- One self-contained .model file fixes normalization, vocabulary and options
- Byte fallback removes <unk> entirely; sampling supports subword regularization

**Limitations**

- Default normalization is not reversible (NFKC, whitespace collapsing)
- Without byte fallback, rare characters become <unk>
- ▁ doubles many entries (▁word vs word), using vocabulary budget
- Byte fallback makes rare scripts and emoji cost 2–4 tokens per character

**Question:** Why can SentencePiece decode with plain string concatenation, while a Moses + BPE pipeline needs a rule-based detokenizer?

- A. SentencePiece uses a larger vocabulary
- B. Whitespace is encoded as the symbol ▁ inside the pieces, so the token sequence records exactly where spaces were
- C. SentencePiece only supports languages without spaces
- D. SentencePiece stores the original string alongside the ids

**Answer:** B. Whitespace is encoded as the symbol ▁ inside the pieces, so the token sequence records exactly where spaces were — With pre-tokenization, `Hello, world!` and `Hello , world !` produce the same tokens, so the spacing is lost and must be guessed. SentencePiece keeps spaces as `▁`. In `▁Hello , ▁world !` only the pieces with `▁` get a space before them.

**Question:** A Llama 2 tokenizer (32,000 BPE pieces, byte_fallback) meets a character that is not in its vocabulary, encoded in UTF-8 as 3 bytes. What does it emit?

- A. One <unk> token, id 0
- B. Three byte tokens, with ids 3 + each byte value
- C. Nothing; the character is dropped
- D. The nearest piece by edit distance

**Answer:** B. Three byte tokens, with ids 3 + each byte value — With `byte_fallback` the unknown character is decomposed into its UTF-8 bytes, which are pieces `<0x00>`…`<0xFF>` at ids 3–258. A 3-byte character such as ♥ (E2 99 A5) becomes ids 229, 156 and 168, and decoding reassembles it exactly.

**Key points**

- SentencePiece (Kudo & Richardson, EMNLP 2018) is a normalizer, trainer, encoder and decoder in one `.model` file. It works on raw text with no pre-tokenizer, and the model is unigram (default) or BPE.
- Spaces become `▁` (U+2581) with a dummy prefix, so $\mathrm{Decode}(\mathrm{Encode}(\mathrm{Normalize}(x)))=\mathrm{Normalize}(x)$ and decoding is just concatenation.
- The default `nmt_nfkc` normalization (NFKC plus whitespace cleanup) is the only lossy step. Use `identity` and keep extra whitespace for code.
- `byte_fallback` encodes unknown characters as UTF-8 byte pieces (ids 3–258 in Llama), so no `<unk>` is ever produced.
- Used by T5 (32k unigram), mT5 (250k), ALBERT (30k), XLNet and Llama 1/2 (32k BPE with digit splitting).

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
