# T5

> T5 (Raffel et al., JMLR 2020) casts every NLP task as text-to-text for one encoder–decoder Transformer, pre-trained on the C4 web corpus by span corruption: 15% of tokens, in spans of mean length 3, each replaced by a sentinel. It uses bucketed relative position biases; sizes run 60M–11B.

Source: https://www.mlfoundry.in/learn/t5  
Section: Deep Learning › Large Language Models (LLMs)  
Level: core · 15 min read · updated 2026-10-13

**T5:** (Text-to-Text Transfer Transformer) is the model and study of Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li and Peter J. Liu, "Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer" (Google, arXiv 2019; JMLR 2020). Every task, whether translation, classification, regression, question answering or summarization, is written as **text in, text out**, so one encoder–decoder model, one loss and one decoding procedure serve all of them. The paper is also a large controlled study of transfer learning. It compares architectures, pre-training objectives, datasets, fine-tuning methods and scaling. Its best choices are combined into models of up to 11B parameters, pre-trained on the new **C4** corpus with a **span corruption** objective.

By 2019 **BERT**, GPT-2, XLNet and RoBERTa had shown that unsupervised pre-training transfers well. They differed in so many ways at once that it was hard to tell what mattered. BERT needed a task-specific head for each output type: a classifier on [CLS], a span-start/end predictor for SQuAD. GPT-style models were awkward for tasks with a long input and a short answer. The T5 team wanted a single framework in which every factor could be varied while everything else stayed fixed. The text-to-text format provided it, and the resulting models set the state of the art on many benchmarks.

## Text-to-text

> Figure: Top: the paper's own examples (its Figure 1). A short task prefix tells the model what to do, and the answer is always a string: a German sentence, the word "not acceptable" for CoLA, the number "3.8" for STS-B similarity, or a summary. Bottom: T5-Base, 12 encoder and 12 decoder blocks with d_model = 768, a 3072-wide ReLU feed-forward layer and 12 heads of size 64, about 220M parameters. Positions enter only as a learned scalar bias on the attention logits, indexed by one of 32 relative-distance buckets: 16 per direction in the encoder, 8 exact and 8 log-spaced up to 128.

For classification, the model outputs the label word ("entailment", "not acceptable"). If it outputs a word that is not a label, the prediction is counted as wrong; the paper reports this never happened with trained models. For the STS-B regression task, most labels were annotated in steps of 0.2, so every score is rounded to the nearest 0.2 and written as a string. Regression becomes classification over a small set of strings such as "3.8". Training always uses the same loss, teacher-forced cross-entropy on the target tokens, and inference uses greedy decoding (beam search for long outputs in the final models).

$$
\mathcal L(\theta)=-\sum_{t=1}^{|y|}\log p_\theta\big(y_t\mid y_{<t},\,\mathrm{Enc}(x)\big),\qquad x=\text{"prefix: input"},\ \ y=\text{answer as text}
$$

- $x$: task prefix plus input, e.g. "translate English to German: That is good."
- $y$: target string, e.g. "Das ist gut."; for pre-training, the sentinel-delimited spans
- $\mathrm{Enc}(x)$: fully visible encoder states, read by every decoder layer through cross-attention

## The architecture

T5 is close to the original **Encoder–decoder transformer**, with three changes. Layer normalization is simplified to a rescaling with no additive bias, and it is applied to the *input* of each sub-block (pre-norm). Dropout (0.1) is applied throughout. The output projection shares its weights with the input embedding. And the sinusoidal position encoding is replaced by a **relative position bias**: each head adds a learned scalar to the attention logit, chosen by the bucket of the offset between key and query. The bias table is shared across all layers.

$$
a_{ij}=\operatorname{softmax}_j\big(q_i^\top k_j+b_h\big[\beta(j-i)\big]\big),\qquad \beta(n)=\begin{cases}|n| & |n|<8\\ \min\!\Big(15,\ 8+\Big\lfloor \dfrac{\ln(|n|/8)}{\ln(128/8)}\cdot 8\Big\rfloor\Big) & |n|\ge 8\end{cases}\ \ +\ 16\cdot[\,n>0\,]
$$

- $b_h[\cdot]$: one learned scalar per bucket for head h; 32 buckets, shared by every layer
- $\beta(n)$: encoder (bidirectional) bucket of the offset n = j − i: 8 exact buckets, 8 log-spaced up to 128, per direction
- $[n>0]$: keys to the right of the query use the second half of the buckets
- $q_i^\top k_j$: no $1/\sqrt{d_k}$ scaling in the reference implementation; it is folded into the initialisation

The decoder is causal, so it uses all 32 buckets for offsets into the past: 16 exact and 16 log-spaced. Every offset beyond the last boundary shares one bucket, so the model can run on inputs longer than it was trained on. Compare **ALiBi**, which replaces the learned table by a fixed linear penalty, and **RoPE**, which rotates queries and keys instead of adding a bias.

## C4 and span corruption

The **Colossal Clean Crawled Corpus (C4)** is about 750 GB of English text built from the April 2019 Common Crawl dump. The filters were heuristic. Keep only lines that end in terminal punctuation. Drop pages with fewer than 3 sentences and lines with fewer than 5 words. Remove pages containing words from a list of bad words, any line mentioning Javascript, pages containing "lorem ipsum" or a curly bracket, and citation markers. Deduplicate any three-sentence span that occurs more than once. Keep pages that langdetect classifies as English with probability at least 0.99.

The pre-training objective is a denoising one. Pick 15% of the tokens in **spans** whose mean length is 3. Replace each span in the input by a unique **sentinel** token, and train the decoder to output each sentinel followed by the tokens it replaced. The paper's example: "Thank you for inviting me to your party last week." becomes the input "Thank you <X> me to your party <Y> week." and the target "<X> for inviting <Y> last <Z>". The target contains only the dropped tokens plus sentinels, so it is much shorter than the input. The decoder, which costs as much per token as the encoder, does far less work than if it had to rewrite the whole text, as in BERT-style denoising with a full-sequence target.

$$
N=\operatorname{round}(0.15\,n),\qquad k=\operatorname{round}(N/3),\qquad |x|=n-N+k+1,\qquad |y|=N+k+1
$$

- $n$: raw tokens in one example
- $N,\ k$: corrupted tokens and number of spans; the N tokens and the n − N clean tokens are each split at random into k pieces and interleaved
- $|x|,\ |y|$: encoder input and decoder target lengths: one sentinel per span and one EOS each

**T5 in numbers**

- **Baseline (≈ T5-Base):** 12 + 12 blocks, $d_{model}=768$, $d_{ff}=3072$, 12 heads, $d_{kv}=64$, dropout 0.1, about 220M parameters (twice BERT-base, as it has two stacks)
- **Vocabulary:** SentencePiece, 32,000 wordpieces trained on C4 mixed with German, French and Romanian; 100 sentinel ids are added in the released vocabulary
- **Baseline pre-training:** $2^{19}=524{,}288$ steps on C4, 128 sequences of 512 tokens per batch ($2^{16}$ tokens), about $2^{35}\approx34$B tokens; AdaFactor with an inverse-square-root schedule $1/\sqrt{\max(n,10^4)}$
- **Baseline fine-tuning:** $2^{18}$ steps per task, same batch size, constant learning rate 0.001
- **Released sizes:** Small ≈ 60M, Base ≈ 220M, Large ≈ 770M, 3B and 11B parameters
- **Large, 3B, 11B:** 24 + 24 blocks. Large: $d_{model}=1024$, $d_{ff}=4096$, 16 heads. 3B: $d_{ff}=16{,}384$, 32 heads of 128. 11B: $d_{ff}=65{,}536$, 128 heads of 128
- **Final pre-training:** 1M steps with batch $2^{11}$ sequences of 512 tokens, about 1 trillion tokens, then fine-tuning per task
- **Results:** state of the art at the time on GLUE (average 90.3), SuperGLUE (88.9, against a human baseline of 89.8), SQuAD and CNN/Daily Mail; not on WMT translation, where back-translation systems remained ahead

> Figure: A 34-word passage (words stand in for SentencePiece pieces) corrupted live. At the default, 15% corruption, mean span 3 and seed 2, T5 picks N = round(34·0.15) = 5 tokens in k = round(5/3) = 2 spans. The input shrinks to 32 tokens with two sentinels, and the target is "<X0> food was great <X1> danced until </s>": 8 tokens, 23% of the original 35. Mean span 1 gives 5 one-token spans and an 11-token target; mean span 10 gives one span and a 7-token target. At 50% corruption the target grows to 24 tokens, as long as the input. The i.i.d. objective with the same seed happens to corrupt 7 tokens in 6 spans (14-token target), and BERT-style always predicts the whole 35-token passage. The right panel scales this to one real pre-training example. A 512-token input needs 568 raw tokens (85 corrupted, 28 sentinels) and gives a 114-token target. Mean span 1 needs 155 target tokens, mean span 10 needs 99, and 50% corruption needs 513.

## Worked example: lengths of one pre-training example

T5 packs C4 into examples whose *corrupted* input is exactly 512 tokens. The reference code searches for the largest raw length $n$ whose input $n-N+k+1$ still fits. The table runs the formula for the paper's setting and for the alternatives it compared.

Raw length n for a 512-token input, and the resulting target length (computed with the formula above)

| corruption | mean span | raw n | corrupted N | spans k | target \|y\| |
| --- | --- | --- | --- | --- | --- |
| 15% | 3 (T5) | 568 | 85 | 28 | 114 |
| 15% | 1 (i.i.d.-like) | 511 | 77 | 77 | 155 |
| 15% | 10 | 591 | 89 | 9 | 99 |
| 25% | 3 | 614 | 154 | 51 | 206 |
| 50% | 3 | 767 | 384 | 128 | 513 |
| BERT-style | n/a | 511 | 77 | n/a | 512 |

**Reading the first row**

1. Try $n=568$: $N=\operatorname{round}(85.2)=85$ and $k=\operatorname{round}(85/3)=28$.
2. Input: $568-85+28+1=512$, exactly the budget. With $n=569$, $N$ is still 85 and the input would be 513, so 568 is the largest raw length that fits.
3. Target: $85+28+1=114$ tokens, 22% of the input length. A BERT-style objective that reconstructs the full text needs a 512-token target (511 tokens plus EOS).
4. Mean span 1 needs a sentinel for almost every corrupted token: 77 sentinels and a 155-token target. Longer spans save sentinels, but the paper found that very long spans (mean 10) begin to hurt quality.
5. Relative positions in the same example: an offset of $-50$ (key 50 tokens to the left) falls in encoder bucket 13, which covers offsets 46–63. An offset of $+50$ uses bucket $16+13=29$. In the causal decoder, $-50$ is bucket 24.

## What the ablations found

**Architecture.** An encoder–decoder trained with a denoising objective did best. Sharing the parameters between encoder and decoder performed nearly as well with half the parameters. A decoder-only language model and a prefix LM were worse in this text-to-text setting. Halving the depth of both stacks hurt.

**Objective.** BERT-style denoising beat prefix language modelling and deshuffling. Among denoising variants the differences were small. Replacing spans with sentinels and dropping corrupted tokens both did well and give short targets. Corruption rates of 10%, 15% and 25% performed similarly, while 50% hurt. Mean span lengths of 2, 3 and 5 were close to i.i.d. corruption, slightly better on some benchmarks, and 10 was worse. Span corruption was chosen mainly because its targets are short, which makes training faster.

**Data.** Filtered C4 beat unfiltered Common Crawl. In-domain corpora, such as RealNews-like or Wikipedia plus Toronto Books, helped the matching downstream tasks. Shrinking C4 so that it is repeated many times during pre-training hurt. The degradation was small at 64 repeats and became clear at 256 or more, together with signs of memorization in the training loss.

**Training strategy and scale.** Fine-tuning all parameters beat adapter layers and gradual unfreezing. Multi-task pre-training followed by fine-tuning matched unsupervised pre-training followed by fine-tuning. Given 4× the compute, training a bigger model and training longer both helped, and ensembling gave further gains. The final recipe combines all of this: span corruption on C4, larger models, much longer training, multi-task pre-training and per-task fine-tuning.

```python
import numpy as np

def random_spans_noise_mask(length, noise_density=0.15, mean_span=3.0, seed=0):
    """Port of T5's random_spans_noise_mask. True marks a corrupted token."""
    rng = np.random.default_rng(seed)
    n_noise = int(np.clip(round(length * noise_density), 1, length - 1))
    n_spans = max(int(round(n_noise / mean_span)), 1)

    def segment(n_items, n_segs):                 # random split into n_segs non-empty pieces
        cuts = np.sort(rng.choice(np.arange(1, n_items), n_segs - 1, replace=False))
        return np.diff(np.concatenate([[0], cuts, [n_items]]))

    noise, clean = segment(n_noise, n_spans), segment(length - n_noise, n_spans)
    mask = np.repeat(np.tile([False, True], n_spans), np.stack([clean, noise], 1).reshape(-1))
    return mask                                   # clean, noise, clean, noise, ... (ends in noise)

def sentinelize(tokens, drop):
    """Keep tokens where drop is False; each run of dropped tokens becomes one sentinel."""
    out, sid = [], 0
    for i, t in enumerate(tokens):
        if not drop[i]:
            out.append(t)
        elif i == 0 or not drop[i - 1]:
            out.append(f"<extra_id_{sid}>")
            sid += 1
    return out + ["</s>"]

def span_corrupt(tokens, **kw):
    mask = random_spans_noise_mask(len(tokens), **kw)
    return sentinelize(tokens, mask), sentinelize(tokens, ~mask)   # (inputs, targets)

words = "Thank you for inviting me to your party last week .".split()
inputs, targets = span_corrupt(words, noise_density=0.3, mean_span=1.5)
print(" ".join(inputs))    # Thank you for inviting me <extra_id_0> party last week <extra_id_1> </s>
print(" ".join(targets))   # <extra_id_0> to your <extra_id_1> . </s>

inputs, targets = span_corrupt([str(i) for i in range(568)])    # the raw length T5 uses for 512 inputs
print(len(inputs), len(targets))                                # 512 114

# --- the same objective and the text-to-text interface with the released model ---
from transformers import T5TokenizerFast, T5ForConditionalGeneration

tok = T5TokenizerFast.from_pretrained("t5-base")
model = T5ForConditionalGeneration.from_pretrained("t5-base")

enc = tok("Thank you <extra_id_0> me to your party <extra_id_1> week.", return_tensors="pt")
labels = tok("<extra_id_0> for inviting <extra_id_1> last <extra_id_2>", return_tensors="pt").input_ids
loss = model(**enc, labels=labels).loss       # span-corruption loss; the decoder input is labels shifted right

x = tok("translate English to German: That is good.", return_tensors="pt")
print(tok.decode(model.generate(**x, max_new_tokens=20)[0], skip_special_tokens=True))
```

Span corruption in numpy, a port of T5's random_spans_noise_mask (run with numpy; outputs shown), followed by the same idea with the released checkpoint in Hugging Face transformers (not run here)

> **The prefix is part of the model's training:** T5's original checkpoints were pre-trained on a multi-task mixture that included the supervised tasks with their prefixes. "translate English to German:" works out of the box because the model saw it in training, not because it understands instructions. A new prefix carries no meaning until you fine-tune on it. The later T5 v1.1 checkpoints were pre-trained on C4 only, so they need fine-tuning before any prefix works. For instruction-following, see **Instruction tuning**: Flan-T5 fine-tuned T5 on a large collection of tasks phrased as instructions.

> **Short targets are the point of span corruption:** With a 15% rate and mean span 3, the decoder predicts 114 tokens per 512-token input instead of 512. Since the decoder is half the network, pre-training with full-sequence reconstruction would spend most decoder compute copying visible tokens. Quality was roughly the same across the span variants, so the paper picked the cheapest one. Sentinels also tell the decoder exactly which gap it is filling, which the plain **BERT** [MASK] token does not.

> **Descendants:** mT5 (Xue et al. 2021) trained the same recipe on mC4 in 101 languages. T5 v1.1 switched to GEGLU feed-forward layers, untied the output layer, and dropped dropout during pre-training. ByT5 operates on raw bytes, and UL2 mixes span corruption with prefix-LM objectives. Flan-T5 instruction-tuned the checkpoints. See **Encoder–decoder models** for how this family compares with decoder-only LLMs, and **BART** for a contemporaneous denoising seq2seq model.

**Strengths**

- One model, one loss, one decoding procedure for every task: no task-specific heads
- Span corruption gives short targets, so pre-training is cheap per token of text
- Bucketed relative position biases are tiny (32 scalars per head) and extend to longer inputs
- A carefully controlled study: its ablations remain a reference for pre-training design choices

**Limitations**

- Classification and regression become generation: label strings must be decoded, and STS-B is quantised to steps of 0.2
- Two stacks: about twice the parameters of an encoder-only model of the same width and depth
- Encoder–decoder models are less convenient than decoder-only LLMs for open-ended chat and long generation
- Original checkpoints are tied to their training prefixes; new tasks need fine-tuning

**Question:** Span corruption with a 15% rate and mean span length 3 turns 568 raw tokens into a 512-token input. How long is the target?

- A. 512 tokens
- B. 85 tokens
- C. 114 tokens
- D. 568 tokens

**Answer:** C. 114 tokens — $N=85$ corrupted tokens in $k=28$ spans. The target is each sentinel followed by its span, plus EOS: $85+28+1=114$.

**Question:** How does T5 encode position?

- A. Learned absolute position embeddings added to the token embeddings
- B. Sinusoidal encodings as in the original Transformer
- C. A learned scalar per head added to each attention logit, indexed by a bucket of the relative offset; 32 buckets, log-spaced beyond 8 and capped at 128, shared across layers
- D. Rotary position embeddings applied to queries and keys

**Answer:** C. A learned scalar per head added to each attention logit, indexed by a bucket of the relative offset; 32 buckets, log-spaced beyond 8 and capped at 128, shared across layers — Each head learns one bias per bucket. In the encoder each direction gets 16 buckets: offsets 0–7 exactly, then 8 log-spaced buckets up to 128. Everything farther shares the last bucket.

**Key points**

- T5 writes every task as text-to-text with a task prefix, so one encoder–decoder model and one cross-entropy loss handle classification, regression, QA, translation and summarization.
- Span corruption: 15% of tokens in spans of mean length 3, each replaced by a sentinel; the target is the sentinels and their spans, $N+k+1$ tokens (114 for a 512-token input).
- C4 is about 750 GB of heuristically cleaned English Common Crawl text (April 2019).
- Positions: one learned scalar per head on each logit, indexed by 32 relative-distance buckets (log-spaced up to 128), shared across layers; LayerNorm without bias, pre-norm.
- Ablations: encoder–decoder plus denoising was best; objective variants mattered little; clean, unrepeated data and scale mattered more. Sizes run from 60M to 11B, trained on about 1T tokens.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
