BART

BART (Lewis et al., ACL 2020) is a denoising autoencoder: a bidirectional encoder reads a corrupted document and an autoregressive decoder rebuilds it. BART-large uses text infilling (Poisson λ = 3 spans, one <mask> each) on 30% of tokens plus sentence permutation; it led on CNN/DM and XSum.

14 min read1159 wordscore0 viewsupdated 2026-10-13

Definition

BART is the sequence-to-sequence pre-training method of Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov and Luke Zettlemoyer (Facebook AI), "BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension" (arXiv 2019; ACL 2020). It corrupts a document with an arbitrary noising function and trains a standard Transformer encoder–decoder to reconstruct the original text. The encoder is bidirectional, as in BERT. The decoder is left-to-right, as in GPT. The training loss is the cross-entropy between the decoder's output and the original document.

BERT's masked-LM pre-training gave excellent encoders, but they cannot generate text naturally: each mask is predicted independently, and the number of masks fixes the output length. GPT's left-to-right LM generates well, but each token only sees the past, which is weaker for understanding tasks. MASS (Song et al. 2019) and UniLM (Dong et al. 2019) had combined the two in different ways. BART's idea is to separate the corruption from the model. Any transformation of the input is allowed, because the decoder always rewrites the whole document. That makes it possible to compare noising schemes directly, including ones that change the length or the order of the text.

Architecture

original documentABCDEbidirectionalencoderfull self-attentionbase 6 · large 12 layerslarge: d=1024, 16 headsautoregressivedecodercausal self-attention+ cross-attention tofinal encoder layersame depth as encoderA_B_Ecorrupted document<s>ABCDtarget shifted rightloss: −log p(original | corrupted), decoder cross-entropyGELU; init N(0, 0.02); about 10% more parameters than aBERT of the same size (cross-attention in every decoder layer)large: 160GB, batch 8000, 500k steps, GPT-2 BPEnoising functionsoriginalABC.DE.token maskingA_C._E.each _ hides one tokentoken deletionA.C.E.where is the gap?text infillingA_.D_E._ hides 2 tokens, then 0sentence permutationDE.ABC.restore sentence orderdocument rotationC.DE.ABfind the true starttext infilling: span lengths ~ Poisson(λ = 3)
fig.Left: the paper's Figure 1(c). The corrupted document A _ B _ E goes into the bidirectional encoder. The decoder sees the original shifted right (<s> A B C D), attends to the final encoder layer, and predicts A B C D E. Right: the five noising functions applied to the two-sentence document A B C . D E ., as in the paper's Figure 2. Token masking and deletion act on single tokens. Text infilling replaces a span of any length, even zero, with one mask. Permutation and rotation leave every token intact and change only the order.

BART is a standard Encoder–decoder transformer, with GELU activations instead of ReLU and parameters initialised from N(0,0.02)\mathcal N(0,0.02). BART-base has 6 layers in the encoder and 6 in the decoder; BART-large has 12 and 12. Compared with BERT, each decoder layer also cross-attends to the final encoder layer, and there is no extra feed-forward layer before the word prediction. The result is roughly 10% more parameters than a BERT of the same size.

L(θ)=− Ex∼D Ex~∼g(x)∑t=1∣x∣log⁡pθ(xt∣x<t, Enc(x~))\mathcal L(\theta)=-\,\mathbb E_{x\sim\mathcal D}\,\mathbb E_{\tilde x\sim g(x)}\sum_{t=1}^{|x|}\log p_\theta\big(x_t\mid x_{<t},\ \mathrm{Enc}(\tilde x)\big)
xx
an original document from the corpus
gg
a noising function: masking, deletion, infilling, permutation, rotation or a combination
x~\tilde x
the corrupted document, read by the bidirectional encoder; its length may differ from x
x<tx_{<t}
the original prefix, fed to the causal decoder by teacher forcing

Five noising functions

Token masking, as in BERT, replaces random tokens with [MASK]. Token deletion removes random tokens, so the model must also decide which positions are missing. Text infilling samples spans with lengths drawn from a Poisson distribution with λ=3\lambda=3 and replaces each span with a single [MASK]. A span of length 0 inserts a mask, and the model has to learn that nothing is missing there. Sentence permutation splits the document at full stops and shuffles the sentences. Document rotation picks a token uniformly at random and rotates the document to start with it, so the model must find the true start.

P(ℓ=k)=e−λλkk!,λ=3:P(0)=0.050,  P(1)=0.149,  P(2)=P(3)=0.224,  E[ℓ]=3P(\ell=k)=\frac{e^{-\lambda}\lambda^k}{k!},\quad \lambda=3:\quad P(0)=0.050,\ \ P(1)=0.149,\ \ P(2)=P(3)=0.224,\ \ \mathbb E[\ell]=3
ℓ\ell
length of one masked span
P(0)P(0)
about 5% of masks hide nothing: an inserted [MASK]
E[ℓ]=3\mathbb E[\ell]=3
each mask hides 3 tokens on average, so a 30% budget needs about 0.1 masks per token

Text infilling generalises two earlier objectives. SpanBERT also masks spans, but replaces each with as many [MASK] tokens as it has words, which tells the model the span length. With one mask per span, BART must predict how many tokens are missing as well as what they are. This is a natural task for a decoder that generates freely, and it fits generation tasks where the output length is unknown.

BART-large in the paper

Architecture
12-layer encoder and 12-layer decoder, hidden size 1024 (16 heads), GELU, about 400M parameters
Tokenizer
the GPT-2 byte-level BPE
Data
160GB of news, books, stories and web text, the same corpus as RoBERTa
Optimisation
batch size 8000, 500,000 steps; dropout switched off for the final 10% of training steps
Noising
text infilling on 30% of the tokens in each document, plus permutation of all sentences
Comprehension
similar to RoBERTa on SQuAD and GLUE, so the generative decoder costs nothing on discriminative tasks
Summarization
CNN/DailyMail ROUGE-1/2/L 44.16/21.28/40.90; XSum 45.14/22.27/37.25, about 6 ROUGE points above the previous best on XSum
Translation
WMT16 Romanian–English: about 1.1 BLEU over a back-translation Transformer baseline, using a new randomly initialised source encoder in front of BART
span lengths ~ Poisson(λ = 3) until 10 of 34 tokens are covered; each span → one <mask>original document = decoder target (autoregressive, teacher-forced)ThecommitteemetonMonday.Itapprovedthenewbudgetafteralongdebate.Criticssaidtheplancutsfundingforpublicschools.Afinalvoteisexpectednextweek.corrupted document = encoder input (bidirectional)ThecommitteemetonMonday.Itapprovedthenewbudgetafteralongdebate<mask>1Criticssaid<mask>6schools.<mask>3isexpectednextweek.original34 tokens, 4 sentencesencoder input27 tokens<mask> tokens3tokens hidden10 (29%)span lengths1, 6, 3Poisson(3) pmf · ● spans drawn here012345678910+P(0) = 0.050: a <mask> that hides nothingunderline = source sentence · ■ hidden from the encoder · ■ <mask> (number = tokens it hides)one <mask> per span: the model must also predict how many tokens are missingBART-large: text infilling on 30% of tokens + permutation of all sentences
noise
fig.A 34-token, four-sentence document noised live. The top row is the decoder target, always the full original. The bottom row is the encoder input; underline colours show which sentence each token came from, and the number under each <mask> is how many tokens it hides. At the default (text infilling, 30%, λ = 3, seed 1), 10 tokens are hidden in spans of 1, 6 and 3. The encoder sees 27 tokens with only 3 masks, so nothing in the input says that one mask hides six words. With the same seed, token masking keeps all 34 positions with 10 masks, and deletion leaves 24 tokens with no placeholders. λ = 1 needs 8 masks (spans 1, 3, 2, 2, 0, 1, 0, 1, with two empty masks), so the input has 32 tokens; λ = 6 needs 2 masks (1 and 9), leaving 26 tokens. Sentence permutation and rotation hide nothing: they only reorder, and rotation with seed 1 starts the document at "funding". Infilling plus permutation, the BART-large recipe, gives spans 7, 2 and 1 in a 27-token input. The panel at the right shows the Poisson(λ) pmf behind the span lengths; P(0) = 0.050 at λ = 3 and 0.368 at λ = 1.

Worked example: text infilling on 30% of a document

Take the figure's document: n=34n=34 tokens and a 30% budget, round⁡(0.3⋅34)=10\operatorname{round}(0.3\cdot34)=10 tokens to hide. Draw span lengths from Poisson(3) until they cover 10 tokens, trimming the last draw. Seed 1 in the figure gives spans 1, 6 and 3.

noise<mask> tokenstokens hiddeninput lengththe model must infer
none0034nothing (identity)
token masking101034the 10 tokens; positions are given
token deletion01024positions and tokens
text infilling31027span lengths (1, 6, 3) and tokens
sentence permutation0034sentence order
document rotation0034the start of the document
Encoder input for the same document under each noising function (seed 1 in the figure; 30% where a ratio applies)

Counting the infilled input

  1. 1Hidden tokens: 1+6+3=101+6+3=10. Clean tokens: 34−10=2434-10=24.
  2. 2Each span becomes one mask: 24+3=2724+3=27 encoder tokens, 21% fewer than the original.
  3. 3The decoder still predicts all 34 tokens, plus the end-of-sequence token. The loss counts every target position, not just the masked ones. Unlike BERT, BART pays for reconstructing visible text too.
  4. 4At BART-large scale, 30% of a 1024-token document is 307 tokens. With E[ℓ]=3\mathbb E[\ell]=3 that is about 102 masks, so the encoder input is about 1024−307+102=8191024-307+102=819 tokens.
  5. 5Poisson(3) puts 5.0% of its mass at 0, 14.9% at 1, 22.4% at 2 and at 3, and 16.8% at 4. Most masks hide 1 to 4 tokens, but long gaps do occur: P(ℓ≥7)=0.034P(\ell\ge7)=0.034.

What the comparison of objectives showed

The paper trained base-size models for 1M steps on the BERT corpus (books and Wikipedia) with each noising function, and with re-implementations of other objectives: a left-to-right LM, an XLNet-style permuted LM, BERT's masked LM, UniLM's multitask masked LM and MASS. Results varied a lot by task, but several patterns were clear.

Token masking matters. Rotation and sentence permutation alone performed poorly. The successful methods all corrupted tokens, by masking, deleting or infilling. Deletion beat masking on generation tasks. Left-to-right pre-training helps generation: the masked LM and permuted LM did worse on generation than objectives with an autoregressive decoder. Bidirectional encoders matter for SQuAD, where right context is needed. A pure language model was best on ELI5, a long-form QA task with weakly constrained outputs. Across tasks, text infilling gave the most consistently strong results, which is why BART-large uses it, together with sentence permutation.

For fine-tuning, BART uses the whole model. For classification, the same input goes to the encoder and the decoder, and the final decoder token's hidden state is fed to a linear classifier, like BERT's [CLS] but at the end so that it attends to the full input. Summarization and generative QA fine-tune directly as seq2seq tasks. For translation into English, a new randomly initialised encoder maps the foreign-language input into representations that BART can denoise into English. It is trained first with most of BART frozen, then end to end.

python
1import numpy as np
2
3def text_infilling(tokens, mask_ratio=0.3, lam=3.0, seed=0, mask="<mask>"):
4 """BART text infilling: Poisson(lam) span lengths cover mask_ratio of the tokens;
5 each span (even an empty one) becomes a single <mask>. Spans never overlap here."""
6 rng = np.random.default_rng(seed)
7 n = len(tokens)
8 goal = int(round(mask_ratio * n))
9 lens = []
10 while sum(lens) < goal:
11 lens.append(int(rng.poisson(lam)))
12 lens[-1] -= sum(lens) - goal # trim so exactly `goal` tokens are hidden
13 clean = n - goal
14 gaps = rng.choice(clean + 1, size=len(lens), replace=False) # one span per gap
15 span_at = dict(zip(sorted(gaps.tolist()), lens))
16 out, p = [], 0
17 for g in range(clean + 1):
18 if g in span_at:
19 out.append(mask)
20 p += span_at[g] # skip the hidden tokens
21 if g < clean:
22 out.append(tokens[p])
23 p += 1
24 return out, lens
25
26def sentence_permutation(text, seed=0):
27 sents = [s.strip() + " ." for s in text.split(" .") if s.strip()]
28 return " ".join(np.random.default_rng(seed).permutation(sents))
29
30doc = ("The committee met on Monday . It approved the new budget after a long debate . "
31 "Critics said the plan cuts funding for public schools . A final vote is expected next week .")
32noisy, spans = text_infilling(sentence_permutation(doc, seed=2).split(), seed=3)
33print(" ".join(noisy))
34# A final vote is expected next <mask> . Critics said the plan cuts funding for <mask> .
35# The <mask> on Monday . It approved <mask> budget <mask> debate .
36print("span lengths", spans, "| input", len(noisy), "tokens, original", len(doc.split()))
37# span lengths [1, 2, 2, 2, 3] | input 29 tokens, original 34
38
39# --- released checkpoints ---
40from transformers import BartTokenizer, BartForConditionalGeneration
41
42tok = BartTokenizer.from_pretrained("facebook/bart-large")
43bart = BartForConditionalGeneration.from_pretrained("facebook/bart-large")
44batch = tok("The committee <mask> on Monday.", return_tensors="pt")
45out = bart.generate(batch["input_ids"], max_new_tokens=20, num_beams=4)
46print(tok.decode(out[0], skip_special_tokens=True)) # the decoder rewrites the whole sentence
47
48# pre-training loss for one (corrupted, original) pair: labels are the original document
49labels = tok(doc, return_tensors="pt").input_ids
50loss = bart(input_ids=tok(" ".join(noisy), return_tensors="pt").input_ids, labels=labels).loss
51
52summarizer = BartForConditionalGeneration.from_pretrained("facebook/bart-large-cnn") # fine-tuned on CNN/DM
53stok = BartTokenizer.from_pretrained("facebook/bart-large-cnn")
54article = stok(["<a long news article>"], truncation=True, max_length=1024, return_tensors="pt")
55summary = summarizer.generate(article["input_ids"], num_beams=4, max_length=142)
Sentence permutation and text infilling in numpy (run; outputs shown), then mask filling and summarization with released BART checkpoints in Hugging Face transformers (not run here)

Use when

  • +One pre-trained model is strong at both generation (summarization, dialogue, generative QA) and comprehension (comparable to RoBERTa)
  • +Any document corruption can be used, including ones that change length or order
  • +Text infilling teaches the model to predict missing content and its length
  • +State-of-the-art summarization on release: 44.16/21.28/40.90 ROUGE on CNN/DM and about 6 points better than prior work on XSum

Avoid when

  • −The decoder reconstructs the whole document, so pre-training targets are long (T5's sentinel targets are about 4× shorter)
  • −Sentence permutation and rotation alone teach little; the gains come from token-level corruption
  • −Translation from other languages needs an extra source encoder and two-stage fine-tuning, and gains were small
  • −Inputs up to 1024 tokens (learned positions) limit long-document summarization without architectural changes

Where it's used

Abstractive summarization

bart-large-cnn and bart-large-xsum are standard news-summarization baselines and are often fine-tuned for domain summaries

Zero-shot classification

BART-large fine-tuned on MNLI is widely used as an NLI-based zero-shot classifier, scoring each candidate label as a hypothesis

Text correction and rewriting

denoising pre-training suits grammatical error correction, paraphrasing and data-to-text after fine-tuning

Check yourself

Under text infilling, a document has 24 visible tokens and 3 <mask> tokens. What do you know about how many tokens are hidden?

Check yourself

Which finding from BART's comparison of noising functions is correct?

Remember

  • ✓BART: corrupt a document with any noise, encode it bidirectionally, and reconstruct the full original with an autoregressive decoder trained by cross-entropy.
  • ✓Five noising functions: token masking, token deletion, text infilling (span lengths ∼\sim Poisson(λ=3\lambda=3), each span → one [MASK], including length 0), sentence permutation and document rotation.
  • ✓Text infilling was the most consistently strong. BART-large uses it on 30% of tokens plus permutation of all sentences.
  • ✓BART-large: 12 + 12 layers, d=1024d=1024, GPT-2 BPE, 160GB of text, batch 8000, 500k steps; about 10% more parameters than a BERT of the same size.
  • ✓Matches RoBERTa on comprehension; on release it was state of the art for summarization (CNN/DM 44.16/21.28/40.90 ROUGE, XSum about +6 ROUGE).
← back to Deep Learning roadmap