Definition
BART is the sequence-to-sequence pre-training method of Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov and Luke Zettlemoyer (Facebook AI), "BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension" (arXiv 2019; ACL 2020). It corrupts a document with an arbitrary noising function and trains a standard Transformer encoder–decoder to reconstruct the original text. The encoder is bidirectional, as in BERT. The decoder is left-to-right, as in GPT. The training loss is the cross-entropy between the decoder's output and the original document.
BERT's masked-LM pre-training gave excellent encoders, but they cannot generate text naturally: each mask is predicted independently, and the number of masks fixes the output length. GPT's left-to-right LM generates well, but each token only sees the past, which is weaker for understanding tasks. MASS (Song et al. 2019) and UniLM (Dong et al. 2019) had combined the two in different ways. BART's idea is to separate the corruption from the model. Any transformation of the input is allowed, because the decoder always rewrites the whole document. That makes it possible to compare noising schemes directly, including ones that change the length or the order of the text.
Architecture
BART is a standard Encoder–decoder transformer, with GELU activations instead of ReLU and parameters initialised from . BART-base has 6 layers in the encoder and 6 in the decoder; BART-large has 12 and 12. Compared with BERT, each decoder layer also cross-attends to the final encoder layer, and there is no extra feed-forward layer before the word prediction. The result is roughly 10% more parameters than a BERT of the same size.
- an original document from the corpus
- a noising function: masking, deletion, infilling, permutation, rotation or a combination
- the corrupted document, read by the bidirectional encoder; its length may differ from x
- the original prefix, fed to the causal decoder by teacher forcing
Five noising functions
Token masking, as in BERT, replaces random tokens with [MASK]. Token deletion removes random tokens, so the model must also decide which positions are missing. Text infilling samples spans with lengths drawn from a Poisson distribution with and replaces each span with a single [MASK]. A span of length 0 inserts a mask, and the model has to learn that nothing is missing there. Sentence permutation splits the document at full stops and shuffles the sentences. Document rotation picks a token uniformly at random and rotates the document to start with it, so the model must find the true start.
- length of one masked span
- about 5% of masks hide nothing: an inserted [MASK]
- each mask hides 3 tokens on average, so a 30% budget needs about 0.1 masks per token
Text infilling generalises two earlier objectives. SpanBERT also masks spans, but replaces each with as many [MASK] tokens as it has words, which tells the model the span length. With one mask per span, BART must predict how many tokens are missing as well as what they are. This is a natural task for a decoder that generates freely, and it fits generation tasks where the output length is unknown.
BART-large in the paper
- Architecture
- 12-layer encoder and 12-layer decoder, hidden size 1024 (16 heads), GELU, about 400M parameters
- Tokenizer
- the GPT-2 byte-level BPE
- Data
- 160GB of news, books, stories and web text, the same corpus as RoBERTa
- Optimisation
- batch size 8000, 500,000 steps; dropout switched off for the final 10% of training steps
- Noising
- text infilling on 30% of the tokens in each document, plus permutation of all sentences
- Comprehension
- similar to RoBERTa on SQuAD and GLUE, so the generative decoder costs nothing on discriminative tasks
- Summarization
- CNN/DailyMail ROUGE-1/2/L 44.16/21.28/40.90; XSum 45.14/22.27/37.25, about 6 ROUGE points above the previous best on XSum
- Translation
- WMT16 Romanian–English: about 1.1 BLEU over a back-translation Transformer baseline, using a new randomly initialised source encoder in front of BART
Worked example: text infilling on 30% of a document
Take the figure's document: tokens and a 30% budget, tokens to hide. Draw span lengths from Poisson(3) until they cover 10 tokens, trimming the last draw. Seed 1 in the figure gives spans 1, 6 and 3.
| noise | <mask> tokens | tokens hidden | input length | the model must infer |
|---|---|---|---|---|
| none | 0 | 0 | 34 | nothing (identity) |
| token masking | 10 | 10 | 34 | the 10 tokens; positions are given |
| token deletion | 0 | 10 | 24 | positions and tokens |
| text infilling | 3 | 10 | 27 | span lengths (1, 6, 3) and tokens |
| sentence permutation | 0 | 0 | 34 | sentence order |
| document rotation | 0 | 0 | 34 | the start of the document |
Counting the infilled input
- 1Hidden tokens: . Clean tokens: .
- 2Each span becomes one mask: encoder tokens, 21% fewer than the original.
- 3The decoder still predicts all 34 tokens, plus the end-of-sequence token. The loss counts every target position, not just the masked ones. Unlike BERT, BART pays for reconstructing visible text too.
- 4At BART-large scale, 30% of a 1024-token document is 307 tokens. With that is about 102 masks, so the encoder input is about tokens.
- 5Poisson(3) puts 5.0% of its mass at 0, 14.9% at 1, 22.4% at 2 and at 3, and 16.8% at 4. Most masks hide 1 to 4 tokens, but long gaps do occur: .
What the comparison of objectives showed
The paper trained base-size models for 1M steps on the BERT corpus (books and Wikipedia) with each noising function, and with re-implementations of other objectives: a left-to-right LM, an XLNet-style permuted LM, BERT's masked LM, UniLM's multitask masked LM and MASS. Results varied a lot by task, but several patterns were clear.
Token masking matters. Rotation and sentence permutation alone performed poorly. The successful methods all corrupted tokens, by masking, deleting or infilling. Deletion beat masking on generation tasks. Left-to-right pre-training helps generation: the masked LM and permuted LM did worse on generation than objectives with an autoregressive decoder. Bidirectional encoders matter for SQuAD, where right context is needed. A pure language model was best on ELI5, a long-form QA task with weakly constrained outputs. Across tasks, text infilling gave the most consistently strong results, which is why BART-large uses it, together with sentence permutation.
For fine-tuning, BART uses the whole model. For classification, the same input goes to the encoder and the decoder, and the final decoder token's hidden state is fed to a linear classifier, like BERT's [CLS] but at the end so that it attends to the full input. Summarization and generative QA fine-tune directly as seq2seq tasks. For translation into English, a new randomly initialised encoder maps the foreign-language input into representations that BART can denoise into English. It is trained first with most of BART frozen, then end to end.
1import numpy as np23def text_infilling(tokens, mask_ratio=0.3, lam=3.0, seed=0, mask="<mask>"):4 """BART text infilling: Poisson(lam) span lengths cover mask_ratio of the tokens;5 each span (even an empty one) becomes a single <mask>. Spans never overlap here."""6 rng = np.random.default_rng(seed)7 n = len(tokens)8 goal = int(round(mask_ratio * n))9 lens = []10 while sum(lens) < goal:11 lens.append(int(rng.poisson(lam)))12 lens[-1] -= sum(lens) - goal # trim so exactly `goal` tokens are hidden13 clean = n - goal14 gaps = rng.choice(clean + 1, size=len(lens), replace=False) # one span per gap15 span_at = dict(zip(sorted(gaps.tolist()), lens))16 out, p = [], 017 for g in range(clean + 1):18 if g in span_at:19 out.append(mask)20 p += span_at[g] # skip the hidden tokens21 if g < clean:22 out.append(tokens[p])23 p += 124 return out, lens2526def sentence_permutation(text, seed=0):27 sents = [s.strip() + " ." for s in text.split(" .") if s.strip()]28 return " ".join(np.random.default_rng(seed).permutation(sents))2930doc = ("The committee met on Monday . It approved the new budget after a long debate . "31 "Critics said the plan cuts funding for public schools . A final vote is expected next week .")32noisy, spans = text_infilling(sentence_permutation(doc, seed=2).split(), seed=3)33print(" ".join(noisy))34# A final vote is expected next <mask> . Critics said the plan cuts funding for <mask> .35# The <mask> on Monday . It approved <mask> budget <mask> debate .36print("span lengths", spans, "| input", len(noisy), "tokens, original", len(doc.split()))37# span lengths [1, 2, 2, 2, 3] | input 29 tokens, original 343839# --- released checkpoints ---40from transformers import BartTokenizer, BartForConditionalGeneration4142tok = BartTokenizer.from_pretrained("facebook/bart-large")43bart = BartForConditionalGeneration.from_pretrained("facebook/bart-large")44batch = tok("The committee <mask> on Monday.", return_tensors="pt")45out = bart.generate(batch["input_ids"], max_new_tokens=20, num_beams=4)46print(tok.decode(out[0], skip_special_tokens=True)) # the decoder rewrites the whole sentence4748# pre-training loss for one (corrupted, original) pair: labels are the original document49labels = tok(doc, return_tensors="pt").input_ids50loss = bart(input_ids=tok(" ".join(noisy), return_tensors="pt").input_ids, labels=labels).loss5152summarizer = BartForConditionalGeneration.from_pretrained("facebook/bart-large-cnn") # fine-tuned on CNN/DM53stok = BartTokenizer.from_pretrained("facebook/bart-large-cnn")54article = stok(["<a long news article>"], truncation=True, max_length=1024, return_tensors="pt")55summary = summarizer.generate(article["input_ids"], num_beams=4, max_length=142)Use when
- +One pre-trained model is strong at both generation (summarization, dialogue, generative QA) and comprehension (comparable to RoBERTa)
- +Any document corruption can be used, including ones that change length or order
- +Text infilling teaches the model to predict missing content and its length
- +State-of-the-art summarization on release: 44.16/21.28/40.90 ROUGE on CNN/DM and about 6 points better than prior work on XSum
Avoid when
- −The decoder reconstructs the whole document, so pre-training targets are long (T5's sentinel targets are about 4× shorter)
- −Sentence permutation and rotation alone teach little; the gains come from token-level corruption
- −Translation from other languages needs an extra source encoder and two-stage fine-tuning, and gains were small
- −Inputs up to 1024 tokens (learned positions) limit long-document summarization without architectural changes
Where it's used
Abstractive summarization
bart-large-cnn and bart-large-xsum are standard news-summarization baselines and are often fine-tuned for domain summaries
Zero-shot classification
BART-large fine-tuned on MNLI is widely used as an NLI-based zero-shot classifier, scoring each candidate label as a hypothesis
Text correction and rewriting
denoising pre-training suits grammatical error correction, paraphrasing and data-to-text after fine-tuning
Check yourself
Under text infilling, a document has 24 visible tokens and 3 <mask> tokens. What do you know about how many tokens are hidden?
Check yourself
Which finding from BART's comparison of noising functions is correct?
Remember
- ✓BART: corrupt a document with any noise, encode it bidirectionally, and reconstruct the full original with an autoregressive decoder trained by cross-entropy.
- ✓Five noising functions: token masking, token deletion, text infilling (span lengths Poisson(), each span → one [MASK], including length 0), sentence permutation and document rotation.
- ✓Text infilling was the most consistently strong. BART-large uses it on 30% of tokens plus permutation of all sentences.
- ✓BART-large: 12 + 12 layers, , GPT-2 BPE, 160GB of text, batch 8000, 500k steps; about 10% more parameters than a BERT of the same size.
- ✓Matches RoBERTa on comprehension; on release it was state of the art for summarization (CNN/DM 44.16/21.28/40.90 ROUGE, XSum about +6 ROUGE).