SentencePiece

SentencePiece (Kudo & Richardson 2018) trains and runs a unigram or BPE subword model on raw text. It NFKC-normalizes, writes spaces as ▁ so that decoding is concatenation, and can fall back to UTF-8 byte pieces. T5 (32k, unigram) and Llama 1/2 (32k, BPE) use it.

14 min read1173 wordscore0 viewsupdated 2026-10-13

Definition

SentencePiece is a tokenizer library and model format by Taku Kudo and John Richardson, "SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing" (EMNLP 2018, system demonstrations). It treats input as a raw stream of Unicode characters with no language-specific pre-tokenization. Spaces are an ordinary symbol, written ▁ (U+2581). The library bundles four parts: a normalizer, a trainer for a subword model (Unigram tokenization by default, or Byte-pair encoding (BPE)), an encoder and a decoder. All four are fixed by one self-contained .model file. The guarantee is that Decode(Encode(Normalize(x)))=Normalize(x)\mathrm{Decode}(\mathrm{Encode}(\mathrm{Normalize}(x)))=\mathrm{Normalize}(x).

Before 2018, subword pipelines had three stages. A language-specific tokenizer such as the Moses scripts split text into words and punctuation. A subword model such as BPE or WordPiece split those words further. A matching detokenizer then tried to undo the first stage with rules about which tokens take a space before them. That pipeline is language-dependent and lossy. Hello, world! and Hello , world ! produce the same tokens, so the detokenizer has to guess. For Chinese, Japanese or Thai, which have no spaces, the first stage needs a separate word segmenter.

Kudo and Richardson removed the first stage. If whitespace is just another character, the subword model can learn where words begin directly from raw sentences. Decoding becomes a string operation with no rules: concatenate the pieces and turn ▁ back into spaces. The same code then works for English, Japanese and a mixed-script corpus, and the tokenizer that was trained is exactly the one that runs at inference.

Whitespace as a symbol

The pipeline for one sentence: Hello world. is escaped to ▁Hello▁world., the model splits it into ▁Hello ▁world ., and decoding concatenates them and turns each ▁ back into a space. The symbol is U+2581 LOWER ONE EIGHTH BLOCK, a visible stand-in for U+0020. With add_dummy_prefix=true (the default) a ▁ is also put before the text, so a sentence-initial word gets the same piece as the same word mid-sentence. Decoding removes it again.

Decode(Encode(Normalize(x)))=Normalize(x),Decode(t1…tm)=unescape(t1∥t2∥⋯∥tm)\mathrm{Decode}\big(\mathrm{Encode}(\mathrm{Normalize}(x))\big)=\mathrm{Normalize}(x),\qquad \mathrm{Decode}(t_1\dots t_m)=\mathrm{unescape}\big(t_1\Vert t_2\Vert\cdots\Vert t_m\big)
Normalize\mathrm{Normalize}
the only lossy step: NFKC folding and whitespace cleanup under the default nmt_nfkc rules
t1∥⋯∥tmt_1\Vert\cdots\Vert t_m
string concatenation of the pieces, after byte pieces are reassembled into UTF-8 characters
unescape\mathrm{unescape}
replace every ▁ with a space and drop the dummy prefix. No language rules are involved

Every piece now records whether a space came before it. ▁world and world are different vocabulary entries, and the second only occurs inside a longer word. That is exactly the information a rule-based detokenizer had to guess. The convention is the reverse of WordPiece's ##. WordPiece marks a piece that does not start a word, while SentencePiece marks one that does, and it marks it with the actual space character it replaces.

Two defaults limit the guarantee. remove_extra_whitespaces=true strips leading and trailing spaces and collapses runs, so New York is normalized to New York before encoding. split_by_whitespace=true prevents a piece from spanning a ▁ in the middle, so ▁New▁York can never become one token. Turning these off gives an exact round trip of the raw bytes, which matters for code and for whitespace-sensitive data.

ENCODE · raw Unicode in, no language-specific pre-tokenizer (Kudo & Richardson 2018)raw textHello world.🦙normalizernmt_nfkc: NFKC +whitespace cleanupescape spaces' ' → ▁ (U+2581)+ dummy prefix ▁modelunigram (default)or BPEpieces → idsunknown char →UTF-8 byte piecesescaped stream▁Hello▁world.pieces▁Hello▁world.“🦙” has no piece▁<0xF0><0x9F><0xA6><0x99>Llama: byte b has id 3 + b, so <0xF0> = 243DECODE · a pure string operation, no rules about spacing or punctuationidsid → piececoncatenatebyte runs → UTF-8 chars▁ → spacestrip the dummy prefixtext“Hello world.”Decode(Encode(Normalize(x))) = Normalize(x): lossless up to normalizationone self-contained .model filepieces + scores (or merge ranks)normalization rule table (precompiled)options: byte_fallback, split_digits, …examplesT5: 32,000 pieces, unigramLlama 1/2: 32,000, BPE + byte fallback, digits splitALBERT: 30,000 · XLNet: 32,000
fig.The SentencePiece pipeline. Encoding takes raw Unicode, applies the normalization rules stored in the model (default nmt_nfkc: NFKC plus whitespace cleanup), escapes each space as ▁ with a dummy prefix, and segments the stream with the unigram or BPE model. A character with no piece is either <unk> or, with byte_fallback, its UTF-8 bytes. In Llama's layout byte b has id 3 + b, so 🦙 becomes <0xF0><0x9F><0xA6><0x99> = 243, 162, 169, 156. Decoding needs no rules: map ids to pieces, join, turn byte runs back into characters, and replace ▁ with a space. One .model file carries the vocabulary, the scores and the normalization table.

Normalization, the model and byte fallback

Normalization. Unicode allows the same visible text to be written in several ways, such as full-width hello, the ligature fi, or the circled ①. SentencePiece maps these to one form before training and encoding. The default rule set, nmt_nfkc, is Unicode NFKC plus a few cleanups such as whitespace normalization. Other built-in rules are nfkc, nfkc_cf (with case folding) and identity, and you can supply a custom TSV of rewrite rules. The rules are compiled into the .model file, so every user of the model normalizes identically. Normalization is the one irreversible step. hello① encodes as hello1 and decodes as hello1.

The model. With model_type=unigram, the default, training runs Kudo's EM-and-prune procedure and encoding takes the Viterbi path (see Unigram tokenization). With model_type=bpe it learns merges from the raw stream (see Byte-pair encoding (BPE)). Both operate on the escaped stream, so ▁ takes part in the statistics like any letter. There are also char and word model types. Optional sampling (enable_sampling, alpha, nbest_size) implements subword regularization for unigram and BPE-dropout for BPE.

Coverage and byte fallback. Training keeps the characters that cover character_coverage of the corpus, by default 0.9995. The README recommends 1.0 for languages with small alphabets. Rare characters outside that set map to <unk> (id 0), which destroys information. The later option byte_fallback=true reserves 256 pieces <0x00>…<0xFF> and encodes any unknown character as its UTF-8 bytes. Every string is then encodable and decodable with no <unk> at all. Llama's tokenizer uses this, as its paper states: numbers are split into digits, and unknown UTF-8 characters fall back to bytes.

Defaults and deployed models

Paper
Kudo & Richardson, EMNLP 2018 (demo track); C++ library with Python bindings; Apache 2.0
Model types
unigram (default), bpe, char, word
Normalization
nmt_nfkc (default), nfkc, nfkc_cf, identity, or custom TSV rules; remove_extra_whitespaces=true, add_dummy_prefix=true
Special ids
<unk>=0, <s>=1, </s>=2 by default; <pad> disabled (id −1) unless set
Coverage
character_coverage=0.9995 by default; byte_fallback adds 256 byte pieces
T5
one 32,000-piece unigram vocabulary shared by English, German, French and Romanian; mT5 uses 250,000 pieces
Llama 1 / 2
32,000-piece BPE model, byte_fallback, digits split; byte pieces are ids 3–258, so newline <0x0A> is id 13. Llama 3 switched to a 128K tiktoken-style BPE
Others
ALBERT (30,000), XLNet (32,000), Gemma (256k), and many multilingual MT systems
toy model: 336 ids = 3 specials + 256 byte pieces (ids 3–258) + 77 pieces · unigram Viterbiraw input“Hello,␣world!”13 code points · 13 UTF-8 bytesnmt_nfkc“Hello,␣world!”unchangedspaces → ▁▁Hello,▁world!dummy prefix ▁ addedpieces / ids▁Hello260,275▁world262!2774 tokensdecode“Hello,␣world!”✓ lossless: decode(encode(x′)) = x′ · = raw inputwhitespace split[Hello] [,] [world] [!]join with spaces“Hello␣,␣world␣!”✗ lost: the split cannot tell where the spaces were
input
options
fig.A toy SentencePiece model with 336 ids, laid out like Llama's: 3 specials, 256 byte pieces (ids 3–258) and 77 pieces with unigram scores, decoded by Viterbi. The default input Hello, world! becomes ▁Hello,▁world! and then 4 pieces, ▁Hello(260) ,(275) ▁world(262) !(277), which decode back to the input exactly. The whitespace-split baseline gives [Hello] [,] [world] [!], which joins to Hello , world ! and has lost the spacing. Turn off add_dummy_prefix and the first word becomes He·ll·o (6 tokens), because Hello without ▁ is not a piece. For New␣␣York, nmt_nfkc collapses the double space to give ▁New ▁York. With normalization off you get ▁New ▁ ▁York, which round-trips the raw text exactly. Full-width hello① (6 code points, 18 UTF-8 bytes) normalizes to hello1, which is 2 tokens. Unnormalized it costs ▁ plus 18 byte tokens. I ♥ 🦙 needs 10 tokens: the heart is 229 156 168 and the llama is 243 162 169 156. Without byte_fallback each becomes <unk> (id 0) and decodes to ⁇, so the round trip fails. 東京に住む needs no spaces: ▁東京 に 住む, while the whitespace baseline sees one 5-character word.

Worked example: encoding and decoding one string

Take I ♥ 🦙 with the figure's toy model and the default options. It is 5 code points and 10 UTF-8 bytes.

stageresult
normalize (nmt_nfkc)I ♥ 🦙 (unchanged: both symbols are already NFKC)
escape + dummy prefix▁I▁♥▁🦙
Viterbi pieces▁I · ▁ · ♥ (unknown) · ▁ · 🦙 (unknown)
byte fallback♥ = E2 99 A5, 🦙 = F0 9F A6 99
tokens▁I ▁ <0xE2> <0x99> <0xA5> ▁ <0xF0> <0x9F> <0xA6> <0x99> (10)
byte ids (3 + b)229 156 168 · 243 162 169 156
decode▁I▁ + bytes→♥ + ▁ + bytes→🦙 = ▁I▁♥▁🦙 → “I ♥ 🦙”
Each stage for “I ♥ 🦙” (checked against a Python re-implementation)

What each step guarantees

  1. 1Escape. The dummy prefix makes the string start with ▁, so I is matched as ▁I, the same piece it would be mid-sentence.
  2. 2Segment. ♥ and 🦙 have no pieces. The unigram model gives each one unknown arc with score min⁡(scores)−10\min(\text{scores})-10, so the Viterbi path isolates them and keeps the ▁ before each as its own piece.
  3. 3Fallback. Each unknown character is replaced by its UTF-8 bytes. A 3-byte and a 4-byte character cost 7 tokens. That is the price of never emitting <unk>.
  4. 4Decode. Byte runs are reassembled into UTF-8 before ▁ is mapped to a space, so the output is exactly I ♥ 🦙. With byte_fallback=false the tokens are ▁I ▁ <unk> ▁ <unk>, and each <unk> decodes to " ⁇ ". The emoji is gone and the round trip fails.
  5. 5Contrast. A whitespace pre-tokenizer would give [Hello] [,] [world] [!] for Hello, world!. Joining with spaces produces Hello , world !. SentencePiece's ▁Hello , ▁world ! already contains the answer: , and ! have no ▁, so no space goes before them.
python
1# --- the library ----------------------------------------------------------
2# import sentencepiece as spm
3# spm.SentencePieceTrainer.train(
4# input="corpus.txt", model_prefix="sp", vocab_size=32000,
5# model_type="unigram", # or "bpe" (Llama 1/2)
6# character_coverage=0.9995, # 1.0 for small alphabets
7# normalization_rule_name="nmt_nfkc",
8# byte_fallback=True, split_digits=True)
9# sp = spm.SentencePieceProcessor(model_file="sp.model")
10# ids = sp.encode("Hello, world!") # list of ints
11# pieces = sp.encode("Hello, world!", out_type=str) # e.g. ['▁Hello', ',', '▁world', '!']
12# assert sp.decode(ids) == "Hello, world!"
13
14# --- what it does, in miniature ----------------------------------------------
15import math, re, unicodedata
16
17SP = "▁" # U+2581, the whitespace meta symbol
18
19def normalize(text): # roughly nmt_nfkc: NFKC + whitespace cleanup
20 return re.sub(" +", " ", unicodedata.normalize("NFKC", text)).strip()
21
22def escape(text, dummy_prefix=True):
23 return (" " + text if dummy_prefix else text).replace(" ", SP)
24
25def encode(text, scores, byte_fallback=True):
26 s = escape(normalize(text))
27 unk = min(scores.values()) - 10 # SentencePiece's penalty for an unknown character
28 best, back = [0.0] + [-math.inf] * len(s), [0] * (len(s) + 1)
29 for j in range(1, len(s) + 1): # Viterbi over code points
30 for i in range(max(0, j - 16), j):
31 if s[i:j] in scores and best[i] + scores[s[i:j]] > best[j]:
32 best[j], back[j] = best[i] + scores[s[i:j]], i
33 if s[j - 1] not in scores and best[j - 1] + unk > best[j]:
34 best[j], back[j] = best[j - 1] + unk, j - 1
35 pieces, j = [], len(s)
36 while j > 0:
37 pieces.append(s[back[j]:j]); j = back[j]
38 out = []
39 for p in reversed(pieces):
40 if p in scores:
41 out.append(p)
42 elif byte_fallback: # unknown character → its UTF-8 bytes
43 out += [f"<0x{b:02X}>" for b in p.encode("utf-8")]
44 else:
45 out.append("<unk>")
46 return out
47
48def decode(pieces):
49 buf, text = bytearray(), ""
50 for p in pieces + [""]:
51 if re.fullmatch(r"<0x[0-9A-F]{2}>", p):
52 buf.append(int(p[3:5], 16)); continue
53 text += buf.decode("utf-8") + (" ⁇ " if p == "<unk>" else p); buf.clear()
54 text = text.replace(SP, " ")
55 return text[1:] if text.startswith(" ") else text
56
57scores = {SP: -3.0, SP + "Hello": -8.0, SP + "world": -8.2, SP + "I": -5.2, ",": -3.6, "!": -4.6,
58 SP + "hello": -8.4, "1": -5.6}
59for x in ["Hello, world!", "I ♥ 🦙", "hello①"]:
60 toks = encode(x, scores)
61 print(toks, repr(decode(toks)), decode(toks) == normalize(x))
62# ['▁Hello', ',', '▁world', '!'] 'Hello, world!' True
63# ['▁I', '▁', '<0xE2>', '<0x99>', '<0xA5>', '▁', '<0xF0>', '<0x9F>', '<0xA6>', '<0x99>'] 'I ♥ 🦙' True
64# ['▁hello', '1'] 'hello1' True
65print(encode("I ♥ 🦙", scores, byte_fallback=False))
66# ['▁I', '▁', '<unk>', '▁', '<unk>']
Top: the SentencePiece API (commented out, not run here). Bottom: a small pure-Python re-implementation of normalization, escaping, Viterbi with unknown-character arcs, byte fallback and decoding. That part was run, and its output is in the comments.

Use when

  • +Language-independent: trains on raw sentences, no word segmenter needed for CJK or Thai
  • +Lossless round trip of the normalized text; decoding is plain concatenation
  • +One self-contained .model file fixes normalization, vocabulary and options
  • +Byte fallback removes <unk> entirely; sampling supports subword regularization

Avoid when

  • −Default normalization is not reversible (NFKC, whitespace collapsing)
  • −Without byte fallback, rare characters become <unk>
  • −▁ doubles many entries (▁word vs word), using vocabulary budget
  • −Byte fallback makes rare scripts and emoji cost 2–4 tokens per character

Check yourself

Why can SentencePiece decode with plain string concatenation, while a Moses + BPE pipeline needs a rule-based detokenizer?

Check yourself

A Llama 2 tokenizer (32,000 BPE pieces, byte_fallback) meets a character that is not in its vocabulary, encoded in UTF-8 as 3 bytes. What does it emit?

Remember

  • ✓SentencePiece (Kudo & Richardson, EMNLP 2018) is a normalizer, trainer, encoder and decoder in one .model file. It works on raw text with no pre-tokenizer, and the model is unigram (default) or BPE.
  • ✓Spaces become ▁ (U+2581) with a dummy prefix, so Decode(Encode(Normalize(x)))=Normalize(x)\mathrm{Decode}(\mathrm{Encode}(\mathrm{Normalize}(x)))=\mathrm{Normalize}(x) and decoding is just concatenation.
  • ✓The default nmt_nfkc normalization (NFKC plus whitespace cleanup) is the only lossy step. Use identity and keep extra whitespace for code.
  • ✓byte_fallback encodes unknown characters as UTF-8 byte pieces (ids 3–258 in Llama), so no <unk> is ever produced.
  • ✓Used by T5 (32k unigram), mT5 (250k), ALBERT (30k), XLNet and Llama 1/2 (32k BPE with digit splitting).
← back to Deep Learning roadmap