# BART

> BART (Lewis et al., ACL 2020) is a denoising autoencoder: a bidirectional encoder reads a corrupted document and an autoregressive decoder rebuilds it. BART-large uses text infilling (Poisson λ = 3 spans, one <mask> each) on 30% of tokens plus sentence permutation; it led on CNN/DM and XSum.

Source: https://www.mlfoundry.in/learn/bart  
Section: Deep Learning › Large Language Models (LLMs)  
Level: core · 14 min read · updated 2026-10-13

**BART:** is the sequence-to-sequence pre-training method of Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov and Luke Zettlemoyer (Facebook AI), "BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension" (arXiv 2019; ACL 2020). It corrupts a document with an arbitrary noising function and trains a standard Transformer encoder–decoder to reconstruct the original text. The encoder is bidirectional, as in **BERT**. The decoder is left-to-right, as in GPT. The training loss is the cross-entropy between the decoder's output and the original document.

BERT's masked-LM pre-training gave excellent encoders, but they cannot generate text naturally: each mask is predicted independently, and the number of masks fixes the output length. GPT's left-to-right LM generates well, but each token only sees the past, which is weaker for understanding tasks. MASS (Song et al. 2019) and UniLM (Dong et al. 2019) had combined the two in different ways. BART's idea is to separate the corruption from the model. Any transformation of the input is allowed, because the decoder always rewrites the *whole* document. That makes it possible to compare noising schemes directly, including ones that change the length or the order of the text.

## Architecture

> Figure: Left: the paper's Figure 1(c). The corrupted document A _ B _ E goes into the bidirectional encoder. The decoder sees the original shifted right (<s> A B C D), attends to the final encoder layer, and predicts A B C D E. Right: the five noising functions applied to the two-sentence document A B C . D E ., as in the paper's Figure 2. Token masking and deletion act on single tokens. Text infilling replaces a span of any length, even zero, with one mask. Permutation and rotation leave every token intact and change only the order.

BART is a standard **Encoder–decoder transformer**, with GELU activations instead of ReLU and parameters initialised from $\mathcal N(0,0.02)$. BART-base has 6 layers in the encoder and 6 in the decoder; BART-large has 12 and 12. Compared with BERT, each decoder layer also cross-attends to the final encoder layer, and there is no extra feed-forward layer before the word prediction. The result is roughly 10% more parameters than a BERT of the same size.

$$
\mathcal L(\theta)=-\,\mathbb E_{x\sim\mathcal D}\,\mathbb E_{\tilde x\sim g(x)}\sum_{t=1}^{|x|}\log p_\theta\big(x_t\mid x_{<t},\ \mathrm{Enc}(\tilde x)\big)
$$

- $x$: an original document from the corpus
- $g$: a noising function: masking, deletion, infilling, permutation, rotation or a combination
- $\tilde x$: the corrupted document, read by the bidirectional encoder; its length may differ from x
- $x_{<t}$: the original prefix, fed to the causal decoder by teacher forcing

## Five noising functions

**Token masking**, as in BERT, replaces random tokens with [MASK]. **Token deletion** removes random tokens, so the model must also decide *which positions* are missing. **Text infilling** samples spans with lengths drawn from a Poisson distribution with $\lambda=3$ and replaces each span with a *single* [MASK]. A span of length 0 inserts a mask, and the model has to learn that nothing is missing there. **Sentence permutation** splits the document at full stops and shuffles the sentences. **Document rotation** picks a token uniformly at random and rotates the document to start with it, so the model must find the true start.

$$
P(\ell=k)=\frac{e^{-\lambda}\lambda^k}{k!},\quad \lambda=3:\quad P(0)=0.050,\ \ P(1)=0.149,\ \ P(2)=P(3)=0.224,\ \ \mathbb E[\ell]=3
$$

- $\ell$: length of one masked span
- $P(0)$: about 5% of masks hide nothing: an inserted [MASK]
- $\mathbb E[\ell]=3$: each mask hides 3 tokens on average, so a 30% budget needs about 0.1 masks per token

Text infilling generalises two earlier objectives. SpanBERT also masks spans, but replaces each with as many [MASK] tokens as it has words, which tells the model the span length. With one mask per span, BART must predict **how many** tokens are missing as well as what they are. This is a natural task for a decoder that generates freely, and it fits generation tasks where the output length is unknown.

**BART-large in the paper**

- **Architecture:** 12-layer encoder and 12-layer decoder, hidden size 1024 (16 heads), GELU, about 400M parameters
- **Tokenizer:** the GPT-2 byte-level BPE
- **Data:** 160GB of news, books, stories and web text, the same corpus as RoBERTa
- **Optimisation:** batch size 8000, 500,000 steps; dropout switched off for the final 10% of training steps
- **Noising:** text infilling on 30% of the tokens in each document, plus permutation of all sentences
- **Comprehension:** similar to RoBERTa on SQuAD and GLUE, so the generative decoder costs nothing on discriminative tasks
- **Summarization:** CNN/DailyMail ROUGE-1/2/L 44.16/21.28/40.90; XSum 45.14/22.27/37.25, about 6 ROUGE points above the previous best on XSum
- **Translation:** WMT16 Romanian–English: about 1.1 BLEU over a back-translation Transformer baseline, using a new randomly initialised source encoder in front of BART

> Figure: A 34-token, four-sentence document noised live. The top row is the decoder target, always the full original. The bottom row is the encoder input; underline colours show which sentence each token came from, and the number under each <mask> is how many tokens it hides. At the default (text infilling, 30%, λ = 3, seed 1), 10 tokens are hidden in spans of 1, 6 and 3. The encoder sees 27 tokens with only 3 masks, so nothing in the input says that one mask hides six words. With the same seed, token masking keeps all 34 positions with 10 masks, and deletion leaves 24 tokens with no placeholders. λ = 1 needs 8 masks (spans 1, 3, 2, 2, 0, 1, 0, 1, with two empty masks), so the input has 32 tokens; λ = 6 needs 2 masks (1 and 9), leaving 26 tokens. Sentence permutation and rotation hide nothing: they only reorder, and rotation with seed 1 starts the document at "funding". Infilling plus permutation, the BART-large recipe, gives spans 7, 2 and 1 in a 27-token input. The panel at the right shows the Poisson(λ) pmf behind the span lengths; P(0) = 0.050 at λ = 3 and 0.368 at λ = 1.

## Worked example: text infilling on 30% of a document

Take the figure's document: $n=34$ tokens and a 30% budget, $\operatorname{round}(0.3\cdot34)=10$ tokens to hide. Draw span lengths from Poisson(3) until they cover 10 tokens, trimming the last draw. Seed 1 in the figure gives spans 1, 6 and 3.

Encoder input for the same document under each noising function (seed 1 in the figure; 30% where a ratio applies)

| noise | <mask> tokens | tokens hidden | input length | the model must infer |
| --- | --- | --- | --- | --- |
| none | 0 | 0 | 34 | nothing (identity) |
| token masking | 10 | 10 | 34 | the 10 tokens; positions are given |
| token deletion | 0 | 10 | 24 | positions and tokens |
| text infilling | 3 | 10 | 27 | span lengths (1, 6, 3) and tokens |
| sentence permutation | 0 | 0 | 34 | sentence order |
| document rotation | 0 | 0 | 34 | the start of the document |

**Counting the infilled input**

1. Hidden tokens: $1+6+3=10$. Clean tokens: $34-10=24$.
2. Each span becomes one mask: $24+3=27$ encoder tokens, 21% fewer than the original.
3. The decoder still predicts all 34 tokens, plus the end-of-sequence token. The loss counts every target position, not just the masked ones. Unlike BERT, BART pays for reconstructing visible text too.
4. At BART-large scale, 30% of a 1024-token document is 307 tokens. With $\mathbb E[\ell]=3$ that is about 102 masks, so the encoder input is about $1024-307+102=819$ tokens.
5. Poisson(3) puts 5.0% of its mass at 0, 14.9% at 1, 22.4% at 2 and at 3, and 16.8% at 4. Most masks hide 1 to 4 tokens, but long gaps do occur: $P(\ell\ge7)=0.034$.

## What the comparison of objectives showed

The paper trained base-size models for 1M steps on the BERT corpus (books and Wikipedia) with each noising function, and with re-implementations of other objectives: a left-to-right LM, an XLNet-style permuted LM, BERT's masked LM, UniLM's multitask masked LM and MASS. Results varied a lot by task, but several patterns were clear.

**Token masking matters.** Rotation and sentence permutation alone performed poorly. The successful methods all corrupted tokens, by masking, deleting or infilling. **Deletion beat masking on generation tasks.** **Left-to-right pre-training helps generation**: the masked LM and permuted LM did worse on generation than objectives with an autoregressive decoder. **Bidirectional encoders matter for SQuAD**, where right context is needed. A pure language model was best on ELI5, a long-form QA task with weakly constrained outputs. Across tasks, **text infilling** gave the most consistently strong results, which is why BART-large uses it, together with sentence permutation.

For fine-tuning, BART uses the whole model. For classification, the same input goes to the encoder and the decoder, and the final decoder token's hidden state is fed to a linear classifier, like BERT's [CLS] but at the end so that it attends to the full input. Summarization and generative QA fine-tune directly as seq2seq tasks. For translation into English, a new randomly initialised encoder maps the foreign-language input into representations that BART can denoise into English. It is trained first with most of BART frozen, then end to end.

```python
import numpy as np

def text_infilling(tokens, mask_ratio=0.3, lam=3.0, seed=0, mask="<mask>"):
    """BART text infilling: Poisson(lam) span lengths cover mask_ratio of the tokens;
    each span (even an empty one) becomes a single <mask>. Spans never overlap here."""
    rng = np.random.default_rng(seed)
    n = len(tokens)
    goal = int(round(mask_ratio * n))
    lens = []
    while sum(lens) < goal:
        lens.append(int(rng.poisson(lam)))
    lens[-1] -= sum(lens) - goal                   # trim so exactly `goal` tokens are hidden
    clean = n - goal
    gaps = rng.choice(clean + 1, size=len(lens), replace=False)   # one span per gap
    span_at = dict(zip(sorted(gaps.tolist()), lens))
    out, p = [], 0
    for g in range(clean + 1):
        if g in span_at:
            out.append(mask)
            p += span_at[g]                         # skip the hidden tokens
        if g < clean:
            out.append(tokens[p])
            p += 1
    return out, lens

def sentence_permutation(text, seed=0):
    sents = [s.strip() + " ." for s in text.split(" .") if s.strip()]
    return " ".join(np.random.default_rng(seed).permutation(sents))

doc = ("The committee met on Monday . It approved the new budget after a long debate . "
       "Critics said the plan cuts funding for public schools . A final vote is expected next week .")
noisy, spans = text_infilling(sentence_permutation(doc, seed=2).split(), seed=3)
print(" ".join(noisy))
# A final vote is expected next <mask> . Critics said the plan cuts funding for <mask> .
#   The <mask> on Monday . It approved <mask> budget <mask> debate .
print("span lengths", spans, "| input", len(noisy), "tokens, original", len(doc.split()))
# span lengths [1, 2, 2, 2, 3] | input 29 tokens, original 34

# --- released checkpoints ---
from transformers import BartTokenizer, BartForConditionalGeneration

tok = BartTokenizer.from_pretrained("facebook/bart-large")
bart = BartForConditionalGeneration.from_pretrained("facebook/bart-large")
batch = tok("The committee <mask> on Monday.", return_tensors="pt")
out = bart.generate(batch["input_ids"], max_new_tokens=20, num_beams=4)
print(tok.decode(out[0], skip_special_tokens=True))   # the decoder rewrites the whole sentence

# pre-training loss for one (corrupted, original) pair: labels are the original document
labels = tok(doc, return_tensors="pt").input_ids
loss = bart(input_ids=tok(" ".join(noisy), return_tensors="pt").input_ids, labels=labels).loss

summarizer = BartForConditionalGeneration.from_pretrained("facebook/bart-large-cnn")   # fine-tuned on CNN/DM
stok = BartTokenizer.from_pretrained("facebook/bart-large-cnn")
article = stok(["<a long news article>"], truncation=True, max_length=1024, return_tensors="pt")
summary = summarizer.generate(article["input_ids"], num_beams=4, max_length=142)
```

Sentence permutation and text infilling in numpy (run; outputs shown), then mask filling and summarization with released BART checkpoints in Hugging Face transformers (not run here)

> **The <mask> count is not the gap length:** With text infilling, one [MASK] can stand for zero, one or many tokens. Code that assumes BERT-style one-for-one masks breaks here. Examples are computing accuracy only "at masked positions", or aligning input and output token by token. BART's decoder output is a new sequence that need not align with the input. Evaluate it as generation: compare the full reconstruction, or the task output, with the reference.

> **Why a decoder makes any noise possible:** A BERT-style model predicts one output per input position, so the corruption must preserve length and order. BART's autoregressive decoder rewrites the whole document, so the noise can delete, insert, reorder or rotate freely. This separation of noise and model is what allowed the paper to compare objectives, and it is why the same pre-trained model fine-tunes directly for summarization, where input and output lengths differ by a factor of ten.

> **BART, T5 and their descendants:** **T5**, published the same month, also pre-trains an encoder–decoder by denoising. Its target lists only the dropped spans after sentinels, while BART reconstructs the entire document. T5's target is cheaper, while BART's makes the decoder a full language model of the text. mBART (Liu et al. 2020) applied BART's objective to 25 languages for translation pre-training. Fine-tuned BART checkpoints such as bart-large-cnn and bart-large-xsum remain common summarization baselines. See **Encoder–decoder models** for the family as a whole.

**Strengths**

- One pre-trained model is strong at both generation (summarization, dialogue, generative QA) and comprehension (comparable to RoBERTa)
- Any document corruption can be used, including ones that change length or order
- Text infilling teaches the model to predict missing content and its length
- State-of-the-art summarization on release: 44.16/21.28/40.90 ROUGE on CNN/DM and about 6 points better than prior work on XSum

**Limitations**

- The decoder reconstructs the whole document, so pre-training targets are long (T5's sentinel targets are about 4× shorter)
- Sentence permutation and rotation alone teach little; the gains come from token-level corruption
- Translation from other languages needs an extra source encoder and two-stage fine-tuning, and gains were small
- Inputs up to 1024 tokens (learned positions) limit long-document summarization without architectural changes

- **Abstractive summarization:** bart-large-cnn and bart-large-xsum are standard news-summarization baselines and are often fine-tuned for domain summaries
- **Zero-shot classification:** BART-large fine-tuned on MNLI is widely used as an NLI-based zero-shot classifier, scoring each candidate label as a hypothesis
- **Text correction and rewriting:** denoising pre-training suits grammatical error correction, paraphrasing and data-to-text after fine-tuning

**Question:** Under text infilling, a document has 24 visible tokens and 3 <mask> tokens. What do you know about how many tokens are hidden?

- A. Exactly 3, one per mask
- B. Nothing from the input alone: each mask hides a Poisson-distributed number of tokens, possibly 0
- C. Exactly 9, since λ = 3
- D. The same number as the visible tokens

**Answer:** B. Nothing from the input alone: each mask hides a Poisson-distributed number of tokens, possibly 0 — Each span length is drawn from Poisson(3) and replaced by a single mask. On average a mask hides 3 tokens, but it may hide none (probability 0.050) or many. In the figure's default, three masks hide 1, 6 and 3 tokens. Predicting the length is part of the task.

**Question:** Which finding from BART's comparison of noising functions is correct?

- A. Document rotation alone was the best objective
- B. Sentence permutation alone matched text infilling
- C. Objectives that corrupt tokens (masking, deletion, infilling) did well, while permutation or rotation alone did poorly; text infilling was the most consistent
- D. Bidirectional encoders were unnecessary for SQuAD

**Answer:** C. Objectives that corrupt tokens (masking, deletion, infilling) did well, while permutation or rotation alone did poorly; text infilling was the most consistent — Reordering alone teaches little about content. Token deletion beat masking on generation, and text infilling was the most consistently strong, so BART-large combines it with sentence permutation.

**Key points**

- BART: corrupt a document with any noise, encode it bidirectionally, and reconstruct the full original with an autoregressive decoder trained by cross-entropy.
- Five noising functions: token masking, token deletion, text infilling (span lengths $\sim$ Poisson($\lambda=3$), each span → one [MASK], including length 0), sentence permutation and document rotation.
- Text infilling was the most consistently strong. BART-large uses it on 30% of tokens plus permutation of all sentences.
- BART-large: 12 + 12 layers, $d=1024$, GPT-2 BPE, 160GB of text, batch 8000, 500k steps; about 10% more parameters than a BERT of the same size.
- Matches RoBERTa on comprehension; on release it was state of the art for summarization (CNN/DM 44.16/21.28/40.90 ROUGE, XSum about +6 ROUGE).

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
