Definition
T5 (Text-to-Text Transfer Transformer) is the model and study of Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li and Peter J. Liu, "Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer" (Google, arXiv 2019; JMLR 2020). Every task, whether translation, classification, regression, question answering or summarization, is written as text in, text out, so one encoder–decoder model, one loss and one decoding procedure serve all of them. The paper is also a large controlled study of transfer learning. It compares architectures, pre-training objectives, datasets, fine-tuning methods and scaling. Its best choices are combined into models of up to 11B parameters, pre-trained on the new C4 corpus with a span corruption objective.
By 2019 BERT, GPT-2, XLNet and RoBERTa had shown that unsupervised pre-training transfers well. They differed in so many ways at once that it was hard to tell what mattered. BERT needed a task-specific head for each output type: a classifier on [CLS], a span-start/end predictor for SQuAD. GPT-style models were awkward for tasks with a long input and a short answer. The T5 team wanted a single framework in which every factor could be varied while everything else stayed fixed. The text-to-text format provided it, and the resulting models set the state of the art on many benchmarks.
Text-to-text
For classification, the model outputs the label word ("entailment", "not acceptable"). If it outputs a word that is not a label, the prediction is counted as wrong; the paper reports this never happened with trained models. For the STS-B regression task, most labels were annotated in steps of 0.2, so every score is rounded to the nearest 0.2 and written as a string. Regression becomes classification over a small set of strings such as "3.8". Training always uses the same loss, teacher-forced cross-entropy on the target tokens, and inference uses greedy decoding (beam search for long outputs in the final models).
- task prefix plus input, e.g. "translate English to German: That is good."
- target string, e.g. "Das ist gut."; for pre-training, the sentinel-delimited spans
- fully visible encoder states, read by every decoder layer through cross-attention
The architecture
T5 is close to the original Encoder–decoder transformer, with three changes. Layer normalization is simplified to a rescaling with no additive bias, and it is applied to the input of each sub-block (pre-norm). Dropout (0.1) is applied throughout. The output projection shares its weights with the input embedding. And the sinusoidal position encoding is replaced by a relative position bias: each head adds a learned scalar to the attention logit, chosen by the bucket of the offset between key and query. The bias table is shared across all layers.
- one learned scalar per bucket for head h; 32 buckets, shared by every layer
- encoder (bidirectional) bucket of the offset n = j − i: 8 exact buckets, 8 log-spaced up to 128, per direction
- keys to the right of the query use the second half of the buckets
- no scaling in the reference implementation; it is folded into the initialisation
The decoder is causal, so it uses all 32 buckets for offsets into the past: 16 exact and 16 log-spaced. Every offset beyond the last boundary shares one bucket, so the model can run on inputs longer than it was trained on. Compare ALiBi, which replaces the learned table by a fixed linear penalty, and RoPE, which rotates queries and keys instead of adding a bias.
C4 and span corruption
The Colossal Clean Crawled Corpus (C4) is about 750 GB of English text built from the April 2019 Common Crawl dump. The filters were heuristic. Keep only lines that end in terminal punctuation. Drop pages with fewer than 3 sentences and lines with fewer than 5 words. Remove pages containing words from a list of bad words, any line mentioning Javascript, pages containing "lorem ipsum" or a curly bracket, and citation markers. Deduplicate any three-sentence span that occurs more than once. Keep pages that langdetect classifies as English with probability at least 0.99.
The pre-training objective is a denoising one. Pick 15% of the tokens in spans whose mean length is 3. Replace each span in the input by a unique sentinel token, and train the decoder to output each sentinel followed by the tokens it replaced. The paper's example: "Thank you for inviting me to your party last week." becomes the input "Thank you <X> me to your party <Y> week." and the target "<X> for inviting <Y> last <Z>". The target contains only the dropped tokens plus sentinels, so it is much shorter than the input. The decoder, which costs as much per token as the encoder, does far less work than if it had to rewrite the whole text, as in BERT-style denoising with a full-sequence target.
- raw tokens in one example
- corrupted tokens and number of spans; the N tokens and the n − N clean tokens are each split at random into k pieces and interleaved
- encoder input and decoder target lengths: one sentinel per span and one EOS each
T5 in numbers
- Baseline (≈ T5-Base)
- 12 + 12 blocks, , , 12 heads, , dropout 0.1, about 220M parameters (twice BERT-base, as it has two stacks)
- Vocabulary
- SentencePiece, 32,000 wordpieces trained on C4 mixed with German, French and Romanian; 100 sentinel ids are added in the released vocabulary
- Baseline pre-training
- steps on C4, 128 sequences of 512 tokens per batch ( tokens), about B tokens; AdaFactor with an inverse-square-root schedule
- Baseline fine-tuning
- steps per task, same batch size, constant learning rate 0.001
- Released sizes
- Small ≈ 60M, Base ≈ 220M, Large ≈ 770M, 3B and 11B parameters
- Large, 3B, 11B
- 24 + 24 blocks. Large: , , 16 heads. 3B: , 32 heads of 128. 11B: , 128 heads of 128
- Final pre-training
- 1M steps with batch sequences of 512 tokens, about 1 trillion tokens, then fine-tuning per task
- Results
- state of the art at the time on GLUE (average 90.3), SuperGLUE (88.9, against a human baseline of 89.8), SQuAD and CNN/Daily Mail; not on WMT translation, where back-translation systems remained ahead
Worked example: lengths of one pre-training example
T5 packs C4 into examples whose corrupted input is exactly 512 tokens. The reference code searches for the largest raw length whose input still fits. The table runs the formula for the paper's setting and for the alternatives it compared.
| corruption | mean span | raw n | corrupted N | spans k | target |y| |
|---|---|---|---|---|---|
| 15% | 3 (T5) | 568 | 85 | 28 | 114 |
| 15% | 1 (i.i.d.-like) | 511 | 77 | 77 | 155 |
| 15% | 10 | 591 | 89 | 9 | 99 |
| 25% | 3 | 614 | 154 | 51 | 206 |
| 50% | 3 | 767 | 384 | 128 | 513 |
| BERT-style | n/a | 511 | 77 | n/a | 512 |
Reading the first row
- 1Try : and .
- 2Input: , exactly the budget. With , is still 85 and the input would be 513, so 568 is the largest raw length that fits.
- 3Target: tokens, 22% of the input length. A BERT-style objective that reconstructs the full text needs a 512-token target (511 tokens plus EOS).
- 4Mean span 1 needs a sentinel for almost every corrupted token: 77 sentinels and a 155-token target. Longer spans save sentinels, but the paper found that very long spans (mean 10) begin to hurt quality.
- 5Relative positions in the same example: an offset of (key 50 tokens to the left) falls in encoder bucket 13, which covers offsets 46–63. An offset of uses bucket . In the causal decoder, is bucket 24.
What the ablations found
Architecture. An encoder–decoder trained with a denoising objective did best. Sharing the parameters between encoder and decoder performed nearly as well with half the parameters. A decoder-only language model and a prefix LM were worse in this text-to-text setting. Halving the depth of both stacks hurt.
Objective. BERT-style denoising beat prefix language modelling and deshuffling. Among denoising variants the differences were small. Replacing spans with sentinels and dropping corrupted tokens both did well and give short targets. Corruption rates of 10%, 15% and 25% performed similarly, while 50% hurt. Mean span lengths of 2, 3 and 5 were close to i.i.d. corruption, slightly better on some benchmarks, and 10 was worse. Span corruption was chosen mainly because its targets are short, which makes training faster.
Data. Filtered C4 beat unfiltered Common Crawl. In-domain corpora, such as RealNews-like or Wikipedia plus Toronto Books, helped the matching downstream tasks. Shrinking C4 so that it is repeated many times during pre-training hurt. The degradation was small at 64 repeats and became clear at 256 or more, together with signs of memorization in the training loss.
Training strategy and scale. Fine-tuning all parameters beat adapter layers and gradual unfreezing. Multi-task pre-training followed by fine-tuning matched unsupervised pre-training followed by fine-tuning. Given 4× the compute, training a bigger model and training longer both helped, and ensembling gave further gains. The final recipe combines all of this: span corruption on C4, larger models, much longer training, multi-task pre-training and per-task fine-tuning.
1import numpy as np23def random_spans_noise_mask(length, noise_density=0.15, mean_span=3.0, seed=0):4 """Port of T5's random_spans_noise_mask. True marks a corrupted token."""5 rng = np.random.default_rng(seed)6 n_noise = int(np.clip(round(length * noise_density), 1, length - 1))7 n_spans = max(int(round(n_noise / mean_span)), 1)89 def segment(n_items, n_segs): # random split into n_segs non-empty pieces10 cuts = np.sort(rng.choice(np.arange(1, n_items), n_segs - 1, replace=False))11 return np.diff(np.concatenate([[0], cuts, [n_items]]))1213 noise, clean = segment(n_noise, n_spans), segment(length - n_noise, n_spans)14 mask = np.repeat(np.tile([False, True], n_spans), np.stack([clean, noise], 1).reshape(-1))15 return mask # clean, noise, clean, noise, ... (ends in noise)1617def sentinelize(tokens, drop):18 """Keep tokens where drop is False; each run of dropped tokens becomes one sentinel."""19 out, sid = [], 020 for i, t in enumerate(tokens):21 if not drop[i]:22 out.append(t)23 elif i == 0 or not drop[i - 1]:24 out.append(f"<extra_id_{sid}>")25 sid += 126 return out + ["</s>"]2728def span_corrupt(tokens, **kw):29 mask = random_spans_noise_mask(len(tokens), **kw)30 return sentinelize(tokens, mask), sentinelize(tokens, ~mask) # (inputs, targets)3132words = "Thank you for inviting me to your party last week .".split()33inputs, targets = span_corrupt(words, noise_density=0.3, mean_span=1.5)34print(" ".join(inputs)) # Thank you for inviting me <extra_id_0> party last week <extra_id_1> </s>35print(" ".join(targets)) # <extra_id_0> to your <extra_id_1> . </s>3637inputs, targets = span_corrupt([str(i) for i in range(568)]) # the raw length T5 uses for 512 inputs38print(len(inputs), len(targets)) # 512 1143940# --- the same objective and the text-to-text interface with the released model ---41from transformers import T5TokenizerFast, T5ForConditionalGeneration4243tok = T5TokenizerFast.from_pretrained("t5-base")44model = T5ForConditionalGeneration.from_pretrained("t5-base")4546enc = tok("Thank you <extra_id_0> me to your party <extra_id_1> week.", return_tensors="pt")47labels = tok("<extra_id_0> for inviting <extra_id_1> last <extra_id_2>", return_tensors="pt").input_ids48loss = model(**enc, labels=labels).loss # span-corruption loss; the decoder input is labels shifted right4950x = tok("translate English to German: That is good.", return_tensors="pt")51print(tok.decode(model.generate(**x, max_new_tokens=20)[0], skip_special_tokens=True))Use when
- +One model, one loss, one decoding procedure for every task: no task-specific heads
- +Span corruption gives short targets, so pre-training is cheap per token of text
- +Bucketed relative position biases are tiny (32 scalars per head) and extend to longer inputs
- +A carefully controlled study: its ablations remain a reference for pre-training design choices
Avoid when
- −Classification and regression become generation: label strings must be decoded, and STS-B is quantised to steps of 0.2
- −Two stacks: about twice the parameters of an encoder-only model of the same width and depth
- −Encoder–decoder models are less convenient than decoder-only LLMs for open-ended chat and long generation
- −Original checkpoints are tied to their training prefixes; new tasks need fine-tuning
Check yourself
Span corruption with a 15% rate and mean span length 3 turns 568 raw tokens into a 512-token input. How long is the target?
Check yourself
How does T5 encode position?
Remember
- ✓T5 writes every task as text-to-text with a task prefix, so one encoder–decoder model and one cross-entropy loss handle classification, regression, QA, translation and summarization.
- ✓Span corruption: 15% of tokens in spans of mean length 3, each replaced by a sentinel; the target is the sentinels and their spans, tokens (114 for a 512-token input).
- ✓C4 is about 750 GB of heuristically cleaned English Common Crawl text (April 2019).
- ✓Positions: one learned scalar per head on each logit, indexed by 32 relative-distance buckets (log-spaced up to 128), shared across layers; LayerNorm without bias, pre-norm.
- ✓Ablations: encoder–decoder plus denoising was best; objective variants mattered little; clean, unrepeated data and scale mattered more. Sizes run from 60M to 11B, trained on about 1T tokens.