Definition
SentencePiece is a tokenizer library and model format by Taku Kudo and John Richardson, "SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing" (EMNLP 2018, system demonstrations). It treats input as a raw stream of Unicode characters with no language-specific pre-tokenization. Spaces are an ordinary symbol, written ▁ (U+2581). The library bundles four parts: a normalizer, a trainer for a subword model (Unigram tokenization by default, or Byte-pair encoding (BPE)), an encoder and a decoder. All four are fixed by one self-contained .model file. The guarantee is that .
Before 2018, subword pipelines had three stages. A language-specific tokenizer such as the Moses scripts split text into words and punctuation. A subword model such as BPE or WordPiece split those words further. A matching detokenizer then tried to undo the first stage with rules about which tokens take a space before them. That pipeline is language-dependent and lossy. Hello, world! and Hello , world ! produce the same tokens, so the detokenizer has to guess. For Chinese, Japanese or Thai, which have no spaces, the first stage needs a separate word segmenter.
Kudo and Richardson removed the first stage. If whitespace is just another character, the subword model can learn where words begin directly from raw sentences. Decoding becomes a string operation with no rules: concatenate the pieces and turn ▁ back into spaces. The same code then works for English, Japanese and a mixed-script corpus, and the tokenizer that was trained is exactly the one that runs at inference.
Whitespace as a symbol
The pipeline for one sentence: Hello world. is escaped to ▁Hello▁world., the model splits it into ▁Hello ▁world ., and decoding concatenates them and turns each ▁ back into a space. The symbol is U+2581 LOWER ONE EIGHTH BLOCK, a visible stand-in for U+0020. With add_dummy_prefix=true (the default) a ▁ is also put before the text, so a sentence-initial word gets the same piece as the same word mid-sentence. Decoding removes it again.
- the only lossy step: NFKC folding and whitespace cleanup under the default
nmt_nfkcrules - string concatenation of the pieces, after byte pieces are reassembled into UTF-8 characters
- replace every
▁with a space and drop the dummy prefix. No language rules are involved
Every piece now records whether a space came before it. ▁world and world are different vocabulary entries, and the second only occurs inside a longer word. That is exactly the information a rule-based detokenizer had to guess. The convention is the reverse of WordPiece's ##. WordPiece marks a piece that does not start a word, while SentencePiece marks one that does, and it marks it with the actual space character it replaces.
Two defaults limit the guarantee. remove_extra_whitespaces=true strips leading and trailing spaces and collapses runs, so New York is normalized to New York before encoding. split_by_whitespace=true prevents a piece from spanning a ▁ in the middle, so ▁New▁York can never become one token. Turning these off gives an exact round trip of the raw bytes, which matters for code and for whitespace-sensitive data.
Normalization, the model and byte fallback
Normalization. Unicode allows the same visible text to be written in several ways, such as full-width hello, the ligature fi, or the circled ①. SentencePiece maps these to one form before training and encoding. The default rule set, nmt_nfkc, is Unicode NFKC plus a few cleanups such as whitespace normalization. Other built-in rules are nfkc, nfkc_cf (with case folding) and identity, and you can supply a custom TSV of rewrite rules. The rules are compiled into the .model file, so every user of the model normalizes identically. Normalization is the one irreversible step. hello① encodes as hello1 and decodes as hello1.
The model. With model_type=unigram, the default, training runs Kudo's EM-and-prune procedure and encoding takes the Viterbi path (see Unigram tokenization). With model_type=bpe it learns merges from the raw stream (see Byte-pair encoding (BPE)). Both operate on the escaped stream, so ▁ takes part in the statistics like any letter. There are also char and word model types. Optional sampling (enable_sampling, alpha, nbest_size) implements subword regularization for unigram and BPE-dropout for BPE.
Coverage and byte fallback. Training keeps the characters that cover character_coverage of the corpus, by default 0.9995. The README recommends 1.0 for languages with small alphabets. Rare characters outside that set map to <unk> (id 0), which destroys information. The later option byte_fallback=true reserves 256 pieces <0x00>…<0xFF> and encodes any unknown character as its UTF-8 bytes. Every string is then encodable and decodable with no <unk> at all. Llama's tokenizer uses this, as its paper states: numbers are split into digits, and unknown UTF-8 characters fall back to bytes.
Defaults and deployed models
- Paper
- Kudo & Richardson, EMNLP 2018 (demo track); C++ library with Python bindings; Apache 2.0
- Model types
unigram(default),bpe,char,word- Normalization
nmt_nfkc(default),nfkc,nfkc_cf,identity, or custom TSV rules;remove_extra_whitespaces=true,add_dummy_prefix=true- Special ids
<unk>=0,<s>=1,</s>=2 by default;<pad>disabled (id −1) unless set- Coverage
character_coverage=0.9995by default;byte_fallbackadds 256 byte pieces- T5
- one 32,000-piece unigram vocabulary shared by English, German, French and Romanian; mT5 uses 250,000 pieces
- Llama 1 / 2
- 32,000-piece BPE model,
byte_fallback, digits split; byte pieces are ids 3–258, so newline<0x0A>is id 13. Llama 3 switched to a 128K tiktoken-style BPE - Others
- ALBERT (30,000), XLNet (32,000), Gemma (256k), and many multilingual MT systems
Worked example: encoding and decoding one string
Take I ♥ 🦙 with the figure's toy model and the default options. It is 5 code points and 10 UTF-8 bytes.
| stage | result |
|---|---|
| normalize (nmt_nfkc) | I ♥ 🦙 (unchanged: both symbols are already NFKC) |
| escape + dummy prefix | ▁I▁♥▁🦙 |
| Viterbi pieces | ▁I · ▁ · ♥ (unknown) · ▁ · 🦙 (unknown) |
| byte fallback | ♥ = E2 99 A5, 🦙 = F0 9F A6 99 |
| tokens | ▁I ▁ <0xE2> <0x99> <0xA5> ▁ <0xF0> <0x9F> <0xA6> <0x99> (10) |
| byte ids (3 + b) | 229 156 168 · 243 162 169 156 |
| decode | ▁I▁ + bytes→♥ + ▁ + bytes→🦙 = ▁I▁♥▁🦙 → “I ♥ 🦙” |
What each step guarantees
- 1Escape. The dummy prefix makes the string start with
▁, soIis matched as▁I, the same piece it would be mid-sentence. - 2Segment. ♥ and 🦙 have no pieces. The unigram model gives each one unknown arc with score , so the Viterbi path isolates them and keeps the
▁before each as its own piece. - 3Fallback. Each unknown character is replaced by its UTF-8 bytes. A 3-byte and a 4-byte character cost 7 tokens. That is the price of never emitting
<unk>. - 4Decode. Byte runs are reassembled into UTF-8 before
▁is mapped to a space, so the output is exactlyI ♥ 🦙. Withbyte_fallback=falsethe tokens are▁I ▁ <unk> ▁ <unk>, and each<unk>decodes to " ⁇ ". The emoji is gone and the round trip fails. - 5Contrast. A whitespace pre-tokenizer would give
[Hello] [,] [world] [!]forHello, world!. Joining with spaces producesHello , world !. SentencePiece's▁Hello , ▁world !already contains the answer:,and!have no▁, so no space goes before them.
1# --- the library ----------------------------------------------------------2# import sentencepiece as spm3# spm.SentencePieceTrainer.train(4# input="corpus.txt", model_prefix="sp", vocab_size=32000,5# model_type="unigram", # or "bpe" (Llama 1/2)6# character_coverage=0.9995, # 1.0 for small alphabets7# normalization_rule_name="nmt_nfkc",8# byte_fallback=True, split_digits=True)9# sp = spm.SentencePieceProcessor(model_file="sp.model")10# ids = sp.encode("Hello, world!") # list of ints11# pieces = sp.encode("Hello, world!", out_type=str) # e.g. ['▁Hello', ',', '▁world', '!']12# assert sp.decode(ids) == "Hello, world!"1314# --- what it does, in miniature ----------------------------------------------15import math, re, unicodedata1617SP = "▁" # U+2581, the whitespace meta symbol1819def normalize(text): # roughly nmt_nfkc: NFKC + whitespace cleanup20 return re.sub(" +", " ", unicodedata.normalize("NFKC", text)).strip()2122def escape(text, dummy_prefix=True):23 return (" " + text if dummy_prefix else text).replace(" ", SP)2425def encode(text, scores, byte_fallback=True):26 s = escape(normalize(text))27 unk = min(scores.values()) - 10 # SentencePiece's penalty for an unknown character28 best, back = [0.0] + [-math.inf] * len(s), [0] * (len(s) + 1)29 for j in range(1, len(s) + 1): # Viterbi over code points30 for i in range(max(0, j - 16), j):31 if s[i:j] in scores and best[i] + scores[s[i:j]] > best[j]:32 best[j], back[j] = best[i] + scores[s[i:j]], i33 if s[j - 1] not in scores and best[j - 1] + unk > best[j]:34 best[j], back[j] = best[j - 1] + unk, j - 135 pieces, j = [], len(s)36 while j > 0:37 pieces.append(s[back[j]:j]); j = back[j]38 out = []39 for p in reversed(pieces):40 if p in scores:41 out.append(p)42 elif byte_fallback: # unknown character → its UTF-8 bytes43 out += [f"<0x{b:02X}>" for b in p.encode("utf-8")]44 else:45 out.append("<unk>")46 return out4748def decode(pieces):49 buf, text = bytearray(), ""50 for p in pieces + [""]:51 if re.fullmatch(r"<0x[0-9A-F]{2}>", p):52 buf.append(int(p[3:5], 16)); continue53 text += buf.decode("utf-8") + (" ⁇ " if p == "<unk>" else p); buf.clear()54 text = text.replace(SP, " ")55 return text[1:] if text.startswith(" ") else text5657scores = {SP: -3.0, SP + "Hello": -8.0, SP + "world": -8.2, SP + "I": -5.2, ",": -3.6, "!": -4.6,58 SP + "hello": -8.4, "1": -5.6}59for x in ["Hello, world!", "I ♥ 🦙", "hello①"]:60 toks = encode(x, scores)61 print(toks, repr(decode(toks)), decode(toks) == normalize(x))62# ['▁Hello', ',', '▁world', '!'] 'Hello, world!' True63# ['▁I', '▁', '<0xE2>', '<0x99>', '<0xA5>', '▁', '<0xF0>', '<0x9F>', '<0xA6>', '<0x99>'] 'I ♥ 🦙' True64# ['▁hello', '1'] 'hello1' True65print(encode("I ♥ 🦙", scores, byte_fallback=False))66# ['▁I', '▁', '<unk>', '▁', '<unk>']Use when
- +Language-independent: trains on raw sentences, no word segmenter needed for CJK or Thai
- +Lossless round trip of the normalized text; decoding is plain concatenation
- +One self-contained .model file fixes normalization, vocabulary and options
- +Byte fallback removes <unk> entirely; sampling supports subword regularization
Avoid when
- −Default normalization is not reversible (NFKC, whitespace collapsing)
- −Without byte fallback, rare characters become <unk>
- −▁ doubles many entries (▁word vs word), using vocabulary budget
- −Byte fallback makes rare scripts and emoji cost 2–4 tokens per character
Check yourself
Why can SentencePiece decode with plain string concatenation, while a Moses + BPE pipeline needs a rule-based detokenizer?
Check yourself
A Llama 2 tokenizer (32,000 BPE pieces, byte_fallback) meets a character that is not in its vocabulary, encoded in UTF-8 as 3 bytes. What does it emit?
Remember
- ✓SentencePiece (Kudo & Richardson, EMNLP 2018) is a normalizer, trainer, encoder and decoder in one
.modelfile. It works on raw text with no pre-tokenizer, and the model is unigram (default) or BPE. - ✓Spaces become
▁(U+2581) with a dummy prefix, so and decoding is just concatenation. - ✓The default
nmt_nfkcnormalization (NFKC plus whitespace cleanup) is the only lossy step. Useidentityand keep extra whitespace for code. - ✓
byte_fallbackencodes unknown characters as UTF-8 byte pieces (ids 3–258 in Llama), so no<unk>is ever produced. - ✓Used by T5 (32k unigram), mT5 (250k), ALBERT (30k), XLNet and Llama 1/2 (32k BPE with digit splitting).