T5

T5 (Raffel et al., JMLR 2020) casts every NLP task as text-to-text for one encoder–decoder Transformer, pre-trained on the C4 web corpus by span corruption: 15% of tokens, in spans of mean length 3, each replaced by a sentinel. It uses bucketed relative position biases; sizes run 60M–11B.

15 min read1458 wordscore0 viewsupdated 2026-10-13

Definition

T5 (Text-to-Text Transfer Transformer) is the model and study of Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li and Peter J. Liu, "Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer" (Google, arXiv 2019; JMLR 2020). Every task, whether translation, classification, regression, question answering or summarization, is written as text in, text out, so one encoder–decoder model, one loss and one decoding procedure serve all of them. The paper is also a large controlled study of transfer learning. It compares architectures, pre-training objectives, datasets, fine-tuning methods and scaling. Its best choices are combined into models of up to 11B parameters, pre-trained on the new C4 corpus with a span corruption objective.

By 2019 BERT, GPT-2, XLNet and RoBERTa had shown that unsupervised pre-training transfers well. They differed in so many ways at once that it was hard to tell what mattered. BERT needed a task-specific head for each output type: a classifier on [CLS], a span-start/end predictor for SQuAD. GPT-style models were awkward for tasks with a long input and a short answer. The T5 team wanted a single framework in which every factor could be varied while everything else stayed fixed. The text-to-text format provided it, and the resulting models set the state of the art on many benchmarks.

Text-to-text

input text (task prefix + input)target textT5enc–dectranslate English to German: That is good.Das ist gut.cola sentence: The course is jumping well.not acceptablestsb sentence1: The rhino grazed on the grass.sentence2: A rhino is grazing in a field.3.8summarize: state authorities dispatched emergencycrews tuesday to survey the damage after …six people hospitalized aftera storm in attala county.STS-B scores are rounded to 0.2 and emitted as strings; one loss: teacher-forced cross-entropyencoder × 12fully visibleself-attentionFFN 768→3072→768ReLU, dropout 0.1decoder × 12causal self-attncross-attentionFFN 768→3072→768output tied to emb.keys/valuesT5-Base: d_model 768, 12 heads × d_kv 64, ≈220MLayerNorm: scale only, no bias, at sub-block inputSentencePiece vocab: 32k pieces + 100 sentinelsno absolute positions; no 1/√d_k in attentionrelative position bias: logit = q·k + b_h[bucket(j − i)]012345678121623324664918 exact8 log-spaced up to 128cell = first distance |j − i| in the bucket;the last bucket holds every distance ≥ 91encoder: 16 buckets per direction = 32decoder (causal): 32 buckets for the pastone learned scalar per bucket and head,shared by every layer of the stack
fig.Top: the paper's own examples (its Figure 1). A short task prefix tells the model what to do, and the answer is always a string: a German sentence, the word "not acceptable" for CoLA, the number "3.8" for STS-B similarity, or a summary. Bottom: T5-Base, 12 encoder and 12 decoder blocks with d_model = 768, a 3072-wide ReLU feed-forward layer and 12 heads of size 64, about 220M parameters. Positions enter only as a learned scalar bias on the attention logits, indexed by one of 32 relative-distance buckets: 16 per direction in the encoder, 8 exact and 8 log-spaced up to 128.

For classification, the model outputs the label word ("entailment", "not acceptable"). If it outputs a word that is not a label, the prediction is counted as wrong; the paper reports this never happened with trained models. For the STS-B regression task, most labels were annotated in steps of 0.2, so every score is rounded to the nearest 0.2 and written as a string. Regression becomes classification over a small set of strings such as "3.8". Training always uses the same loss, teacher-forced cross-entropy on the target tokens, and inference uses greedy decoding (beam search for long outputs in the final models).

L(θ)=−∑t=1∣y∣log⁡pθ(yt∣y<t, Enc(x)),x="prefix: input",  y=answer as text\mathcal L(\theta)=-\sum_{t=1}^{|y|}\log p_\theta\big(y_t\mid y_{<t},\,\mathrm{Enc}(x)\big),\qquad x=\text{"prefix: input"},\ \ y=\text{answer as text}
xx
task prefix plus input, e.g. "translate English to German: That is good."
yy
target string, e.g. "Das ist gut."; for pre-training, the sentinel-delimited spans
Enc(x)\mathrm{Enc}(x)
fully visible encoder states, read by every decoder layer through cross-attention

The architecture

T5 is close to the original Encoder–decoder transformer, with three changes. Layer normalization is simplified to a rescaling with no additive bias, and it is applied to the input of each sub-block (pre-norm). Dropout (0.1) is applied throughout. The output projection shares its weights with the input embedding. And the sinusoidal position encoding is replaced by a relative position bias: each head adds a learned scalar to the attention logit, chosen by the bucket of the offset between key and query. The bias table is shared across all layers.

aij=softmax⁡j(qi⊤kj+bh[β(j−i)]),β(n)={∣n∣∣n∣<8min⁡ ⁣(15, 8+⌊ln⁡(∣n∣/8)ln⁡(128/8)⋅8⌋)∣n∣≥8  + 16⋅[ n>0 ]a_{ij}=\operatorname{softmax}_j\big(q_i^\top k_j+b_h\big[\beta(j-i)\big]\big),\qquad \beta(n)=\begin{cases}|n| & |n|<8\\ \min\!\Big(15,\ 8+\Big\lfloor \dfrac{\ln(|n|/8)}{\ln(128/8)}\cdot 8\Big\rfloor\Big) & |n|\ge 8\end{cases}\ \ +\ 16\cdot[\,n>0\,]
bh[⋅]b_h[\cdot]
one learned scalar per bucket for head h; 32 buckets, shared by every layer
β(n)\beta(n)
encoder (bidirectional) bucket of the offset n = j − i: 8 exact buckets, 8 log-spaced up to 128, per direction
[n>0][n>0]
keys to the right of the query use the second half of the buckets
qi⊤kjq_i^\top k_j
no 1/dk1/\sqrt{d_k} scaling in the reference implementation; it is folded into the initialisation

The decoder is causal, so it uses all 32 buckets for offsets into the past: 16 exact and 16 log-spaced. Every offset beyond the last boundary shares one bucket, so the model can run on inputs longer than it was trained on. Compare ALiBi, which replaces the learned table by a fixed linear penalty, and RoPE, which rotates queries and keys instead of adding a bias.

C4 and span corruption

The Colossal Clean Crawled Corpus (C4) is about 750 GB of English text built from the April 2019 Common Crawl dump. The filters were heuristic. Keep only lines that end in terminal punctuation. Drop pages with fewer than 3 sentences and lines with fewer than 5 words. Remove pages containing words from a list of bad words, any line mentioning Javascript, pages containing "lorem ipsum" or a curly bracket, and citation markers. Deduplicate any three-sentence span that occurs more than once. Keep pages that langdetect classifies as English with probability at least 0.99.

The pre-training objective is a denoising one. Pick 15% of the tokens in spans whose mean length is 3. Replace each span in the input by a unique sentinel token, and train the decoder to output each sentinel followed by the tokens it replaced. The paper's example: "Thank you for inviting me to your party last week." becomes the input "Thank you <X> me to your party <Y> week." and the target "<X> for inviting <Y> last <Z>". The target contains only the dropped tokens plus sentinels, so it is much shorter than the input. The decoder, which costs as much per token as the encoder, does far less work than if it had to rewrite the whole text, as in BERT-style denoising with a full-sequence target.

N=round⁡(0.15 n),k=round⁡(N/3),∣x∣=n−N+k+1,∣y∣=N+k+1N=\operatorname{round}(0.15\,n),\qquad k=\operatorname{round}(N/3),\qquad |x|=n-N+k+1,\qquad |y|=N+k+1
nn
raw tokens in one example
N, kN,\ k
corrupted tokens and number of spans; the N tokens and the n − N clean tokens are each split at random into k pieces and interleaved
∣x∣, ∣y∣|x|,\ |y|
encoder input and decoder target lengths: one sentinel per span and one EOS each

T5 in numbers

Baseline (≈ T5-Base)
12 + 12 blocks, dmodel=768d_{model}=768, dff=3072d_{ff}=3072, 12 heads, dkv=64d_{kv}=64, dropout 0.1, about 220M parameters (twice BERT-base, as it has two stacks)
Vocabulary
SentencePiece, 32,000 wordpieces trained on C4 mixed with German, French and Romanian; 100 sentinel ids are added in the released vocabulary
Baseline pre-training
219=524,2882^{19}=524{,}288 steps on C4, 128 sequences of 512 tokens per batch (2162^{16} tokens), about 235≈342^{35}\approx34B tokens; AdaFactor with an inverse-square-root schedule 1/max⁡(n,104)1/\sqrt{\max(n,10^4)}
Baseline fine-tuning
2182^{18} steps per task, same batch size, constant learning rate 0.001
Released sizes
Small ≈ 60M, Base ≈ 220M, Large ≈ 770M, 3B and 11B parameters
Large, 3B, 11B
24 + 24 blocks. Large: dmodel=1024d_{model}=1024, dff=4096d_{ff}=4096, 16 heads. 3B: dff=16,384d_{ff}=16{,}384, 32 heads of 128. 11B: dff=65,536d_{ff}=65{,}536, 128 heads of 128
Final pre-training
1M steps with batch 2112^{11} sequences of 512 tokens, about 1 trillion tokens, then fine-tuning per task
Results
state of the art at the time on GLUE (average 90.3), SuperGLUE (88.9, against a human baseline of 89.8), SQuAD and CNN/Daily Mail; not on WMT translation, where back-translation systems remained ahead
N = round(34·0.15) = 5 noisy tokens in k = round(N/3) = 2 spansencoder inputThankyouforinvitingmetoyourpartylastweek.The<X0>,themusicwasloud,andwe<X1>midnightbeforewalkinghomeunderafullmoon.</s>decoder target<X0>foodwasgreat<X1>danceduntil</s>original34 + </s>corrupted5 (15%)spans2, mean 2.50input length32target length8 (23% of original)one pre-training example, 512 inputsraw text needed 568 tokensN = 85 noisy, k = 28 sentinelsinput 512 · target 114 tokenstarget = N + k + 1, input = n − N + k + 1■ sentinel ■ corrupted token ■ [MASK] ■ random word ■ EOSwords stand in for SentencePiece pieces; sentinels are the vocabulary's 100 extra ids <extra_id_0>, … the decoder only predicts the target, so shorter targets mean cheaper pre-training steps
objective
fig.A 34-word passage (words stand in for SentencePiece pieces) corrupted live. At the default, 15% corruption, mean span 3 and seed 2, T5 picks N = round(34·0.15) = 5 tokens in k = round(5/3) = 2 spans. The input shrinks to 32 tokens with two sentinels, and the target is "<X0> food was great <X1> danced until </s>": 8 tokens, 23% of the original 35. Mean span 1 gives 5 one-token spans and an 11-token target; mean span 10 gives one span and a 7-token target. At 50% corruption the target grows to 24 tokens, as long as the input. The i.i.d. objective with the same seed happens to corrupt 7 tokens in 6 spans (14-token target), and BERT-style always predicts the whole 35-token passage. The right panel scales this to one real pre-training example. A 512-token input needs 568 raw tokens (85 corrupted, 28 sentinels) and gives a 114-token target. Mean span 1 needs 155 target tokens, mean span 10 needs 99, and 50% corruption needs 513.

Worked example: lengths of one pre-training example

T5 packs C4 into examples whose corrupted input is exactly 512 tokens. The reference code searches for the largest raw length nn whose input n−N+k+1n-N+k+1 still fits. The table runs the formula for the paper's setting and for the alternatives it compared.

corruptionmean spanraw ncorrupted Nspans ktarget |y|
15%3 (T5)5688528114
15%1 (i.i.d.-like)5117777155
15%1059189999
25%361415451206
50%3767384128513
BERT-stylen/a51177n/a512
Raw length n for a 512-token input, and the resulting target length (computed with the formula above)

Reading the first row

  1. 1Try n=568n=568: N=round⁡(85.2)=85N=\operatorname{round}(85.2)=85 and k=round⁡(85/3)=28k=\operatorname{round}(85/3)=28.
  2. 2Input: 568−85+28+1=512568-85+28+1=512, exactly the budget. With n=569n=569, NN is still 85 and the input would be 513, so 568 is the largest raw length that fits.
  3. 3Target: 85+28+1=11485+28+1=114 tokens, 22% of the input length. A BERT-style objective that reconstructs the full text needs a 512-token target (511 tokens plus EOS).
  4. 4Mean span 1 needs a sentinel for almost every corrupted token: 77 sentinels and a 155-token target. Longer spans save sentinels, but the paper found that very long spans (mean 10) begin to hurt quality.
  5. 5Relative positions in the same example: an offset of −50-50 (key 50 tokens to the left) falls in encoder bucket 13, which covers offsets 46–63. An offset of +50+50 uses bucket 16+13=2916+13=29. In the causal decoder, −50-50 is bucket 24.

What the ablations found

Architecture. An encoder–decoder trained with a denoising objective did best. Sharing the parameters between encoder and decoder performed nearly as well with half the parameters. A decoder-only language model and a prefix LM were worse in this text-to-text setting. Halving the depth of both stacks hurt.

Objective. BERT-style denoising beat prefix language modelling and deshuffling. Among denoising variants the differences were small. Replacing spans with sentinels and dropping corrupted tokens both did well and give short targets. Corruption rates of 10%, 15% and 25% performed similarly, while 50% hurt. Mean span lengths of 2, 3 and 5 were close to i.i.d. corruption, slightly better on some benchmarks, and 10 was worse. Span corruption was chosen mainly because its targets are short, which makes training faster.

Data. Filtered C4 beat unfiltered Common Crawl. In-domain corpora, such as RealNews-like or Wikipedia plus Toronto Books, helped the matching downstream tasks. Shrinking C4 so that it is repeated many times during pre-training hurt. The degradation was small at 64 repeats and became clear at 256 or more, together with signs of memorization in the training loss.

Training strategy and scale. Fine-tuning all parameters beat adapter layers and gradual unfreezing. Multi-task pre-training followed by fine-tuning matched unsupervised pre-training followed by fine-tuning. Given 4× the compute, training a bigger model and training longer both helped, and ensembling gave further gains. The final recipe combines all of this: span corruption on C4, larger models, much longer training, multi-task pre-training and per-task fine-tuning.

python
1import numpy as np
2
3def random_spans_noise_mask(length, noise_density=0.15, mean_span=3.0, seed=0):
4 """Port of T5's random_spans_noise_mask. True marks a corrupted token."""
5 rng = np.random.default_rng(seed)
6 n_noise = int(np.clip(round(length * noise_density), 1, length - 1))
7 n_spans = max(int(round(n_noise / mean_span)), 1)
8
9 def segment(n_items, n_segs): # random split into n_segs non-empty pieces
10 cuts = np.sort(rng.choice(np.arange(1, n_items), n_segs - 1, replace=False))
11 return np.diff(np.concatenate([[0], cuts, [n_items]]))
12
13 noise, clean = segment(n_noise, n_spans), segment(length - n_noise, n_spans)
14 mask = np.repeat(np.tile([False, True], n_spans), np.stack([clean, noise], 1).reshape(-1))
15 return mask # clean, noise, clean, noise, ... (ends in noise)
16
17def sentinelize(tokens, drop):
18 """Keep tokens where drop is False; each run of dropped tokens becomes one sentinel."""
19 out, sid = [], 0
20 for i, t in enumerate(tokens):
21 if not drop[i]:
22 out.append(t)
23 elif i == 0 or not drop[i - 1]:
24 out.append(f"<extra_id_{sid}>")
25 sid += 1
26 return out + ["</s>"]
27
28def span_corrupt(tokens, **kw):
29 mask = random_spans_noise_mask(len(tokens), **kw)
30 return sentinelize(tokens, mask), sentinelize(tokens, ~mask) # (inputs, targets)
31
32words = "Thank you for inviting me to your party last week .".split()
33inputs, targets = span_corrupt(words, noise_density=0.3, mean_span=1.5)
34print(" ".join(inputs)) # Thank you for inviting me <extra_id_0> party last week <extra_id_1> </s>
35print(" ".join(targets)) # <extra_id_0> to your <extra_id_1> . </s>
36
37inputs, targets = span_corrupt([str(i) for i in range(568)]) # the raw length T5 uses for 512 inputs
38print(len(inputs), len(targets)) # 512 114
39
40# --- the same objective and the text-to-text interface with the released model ---
41from transformers import T5TokenizerFast, T5ForConditionalGeneration
42
43tok = T5TokenizerFast.from_pretrained("t5-base")
44model = T5ForConditionalGeneration.from_pretrained("t5-base")
45
46enc = tok("Thank you <extra_id_0> me to your party <extra_id_1> week.", return_tensors="pt")
47labels = tok("<extra_id_0> for inviting <extra_id_1> last <extra_id_2>", return_tensors="pt").input_ids
48loss = model(**enc, labels=labels).loss # span-corruption loss; the decoder input is labels shifted right
49
50x = tok("translate English to German: That is good.", return_tensors="pt")
51print(tok.decode(model.generate(**x, max_new_tokens=20)[0], skip_special_tokens=True))
Span corruption in numpy, a port of T5's random_spans_noise_mask (run with numpy; outputs shown), followed by the same idea with the released checkpoint in Hugging Face transformers (not run here)

Use when

  • +One model, one loss, one decoding procedure for every task: no task-specific heads
  • +Span corruption gives short targets, so pre-training is cheap per token of text
  • +Bucketed relative position biases are tiny (32 scalars per head) and extend to longer inputs
  • +A carefully controlled study: its ablations remain a reference for pre-training design choices

Avoid when

  • −Classification and regression become generation: label strings must be decoded, and STS-B is quantised to steps of 0.2
  • −Two stacks: about twice the parameters of an encoder-only model of the same width and depth
  • −Encoder–decoder models are less convenient than decoder-only LLMs for open-ended chat and long generation
  • −Original checkpoints are tied to their training prefixes; new tasks need fine-tuning

Check yourself

Span corruption with a 15% rate and mean span length 3 turns 568 raw tokens into a 512-token input. How long is the target?

Check yourself

How does T5 encode position?

Remember

  • ✓T5 writes every task as text-to-text with a task prefix, so one encoder–decoder model and one cross-entropy loss handle classification, regression, QA, translation and summarization.
  • ✓Span corruption: 15% of tokens in spans of mean length 3, each replaced by a sentinel; the target is the sentinels and their spans, N+k+1N+k+1 tokens (114 for a 512-token input).
  • ✓C4 is about 750 GB of heuristically cleaned English Common Crawl text (April 2019).
  • ✓Positions: one learned scalar per head on each logit, indexed by 32 relative-distance buckets (log-spaced up to 128), shared across layers; LayerNorm without bias, pre-norm.
  • ✓Ablations: encoder–decoder plus denoising was best; objective variants mattered little; clean, unrepeated data and scale mattered more. Sizes run from 60M to 11B, trained on about 1T tokens.
← back to Deep Learning roadmap