Definition
Longformer is a Transformer whose self-attention combines three patterns: a sliding window in which each token attends to neighbours on each side, an optional dilated sliding window whose keys are positions apart, and global attention on a few task-chosen positions, which attend to every token and are attended by every token. The cost per layer is instead of , so the released encoder processes documents of 4,096 tokens. It was introduced by Beltagy, Peters and Cohan, "Longformer: The Long-Document Transformer" (Allen Institute for AI, arXiv 2020).
BERT and RoBERTa read at most 512 tokens. For longer documents the usual workaround was to cut the text into 512-token pieces, encode each piece separately and combine the predictions with task-specific machinery. Information could not flow between pieces inside the encoder. Iz Beltagy, Matthew Peters and Arman Cohan asked for a drop-in replacement for full Self-attention that scales linearly with length, keeps the RoBERTa weights usable, and still lets a task route information across the whole document. Earlier long-sequence models such as Transformer-XL and the Sparse attention of Child et al. (2019) had been evaluated mainly on autoregressive language modelling. Longformer was evaluated both on character-level language modelling and as a pretrained encoder fine-tuned on long-document question answering, coreference and classification.
Three attention patterns
- window size: keys on each side of the query, keys including itself (512 in the pretrained model)
- dilation: is the plain sliding window; leaves gaps of positions, like a Dilated convolution
- global positions chosen per task, e.g. [CLS] for classification or all question tokens for QA
- the keys query attends to; softmax runs only over this set. Global attention is symmetric: sees everything and everyone sees
A sliding window alone is local, but stacking layers widens the view, as in a CNN. After layers a token has read positions. With dilation the window spans positions using the same keys, and the top layer reaches . Dilation alone, though, only ever reaches positions in the same residue class modulo . In practice it is used on some heads and mixed with undilated heads. For tasks that need one token to summarise the whole input, waiting for the receptive field to grow is not enough. A global token reads every position in one layer and can broadcast the result back in the next.
- projections used for the sliding-window scores (and for ordinary tokens attending to global keys)
- a second set of projections for the global-token rows; initialised to the values of Q_s, K_s, V_s
- the band-plus-global mask; real implementations never materialise the matrix
The paper uses two sets of projections. Global attention plays a different role from local attention: one is a document-wide summary and the other a local context model. Separate let the two specialise, and the paper's ablation found they help on downstream tasks. The scores are the usual Scaled dot-product attention; only the set of keys changes.
Setup in the paper
- Pretrained encoder
- continued MLM pretraining from the RoBERTa checkpoint; window in every layer (the same per-token cost as RoBERTa's 512-token full attention); 65K gradient updates at sequence length 4,096
- Position embeddings
- RoBERTa's 512 learned positions copied 8 times to fill 4,096; before any further training this already gave a usable MLM loss, while random position embeddings did not
- Global tokens
- [CLS] for classification; all question tokens for QA (WikiHop, TriviaQA, HotpotQA)
- Character LM
- window size increasing from lower to higher layers; dilation only on 2 heads of the higher layers; staged training that doubles sequence length and window across phases
- Implementations
- loop (memory-efficient, slow), chunks (no dilation; used for pretraining and fine-tuning), and a custom CUDA kernel written with TVM that supports dilation
- Encoder–decoder
- LED (Longformer-Encoder-Decoder), initialised from BART, with inputs up to 16K tokens for long-document summarisation such as arXiv
Worked example: counting scores at n = 4096
| pattern | scores | fraction of n² | fp32 memory |
|---|---|---|---|
| full attention | 16,777,216 | 100% | 64 MiB |
| sliding window, d = 1 | 2,035,456 | 12.13% | 7.76 MiB |
| dilated window, d = 2 | 1,969,664 | 11.74% | 7.51 MiB |
| window + 1 global ([CLS]) | 2,043,134 | 12.18% | 7.79 MiB |
| window + 64 global (a long question) | 2,522,816 | 15.04% | 9.62 MiB |
Where the numbers come from
- 1An interior query reads keys. Near the edges the window is cut: token reads only keys. Total .
- 2Ratio to full attention: . Doubling doubles the banded count but quadruples , so the fraction halves with every doubling of length.
- 3One global token at position 0: its row grows from 257 keys to 4,096 (), and the 3,839 queries whose window does not contain it gain one key each (). Total .
- 4Memory for all 12 heads and 12 layers if the probabilities are kept for the backward pass: B GiB for full attention against B GiB for Longformer.
- 5Receptive field with and 12 layers: positions, so even without global tokens the top layer can see the whole input.
Results
| model | WikiHop | TriviaQA | HotpotQA | OntoNotes | IMDB | Hyperpartisan |
|---|---|---|---|---|---|---|
| RoBERTa-base (512 tokens, chunked) | 72.4 | 74.3 | 63.5 | 78.4 | 95.3 | 87.4 |
| Longformer-base (4,096 tokens) | 75.0 | 75.2 | 64.4 | 78.6 | 95.7 | 94.8 |
The gains are largest where evidence is spread over long inputs (WikiHop, Hyperpartisan) and smallest where most documents already fit in 512 tokens (IMDB) or the task is local (coreference). Longformer-large set new state-of-the-art results on WikiHop and TriviaQA at publication. In character-level language modelling the small model reached 1.10 bits per character on text8 and 1.00 on enwik8, and the large model 0.99 on enwik8, matching or beating Transformer-XL-style models of comparable size.
1import torch, torch.nn as nn23def longformer_mask(n, w, d=1, global_idx=()):4 i, j = torch.arange(n)[:, None], torch.arange(n)[None, :]5 off = (i - j).abs()6 m = (off <= (w // 2) * d) & (off % d == 0) # (dilated) sliding window7 g = torch.zeros(n, dtype=torch.bool)8 g[list(global_idx)] = True9 return m | g[:, None] | g[None, :] # global rows and columns1011class LongformerSelfAttention(nn.Module):12 """Same output as the banded kernels, but O(n^2) memory: use it to check them."""13 def __init__(self, D, heads, w, d=1):14 super().__init__()15 self.h, self.w, self.d = heads, w, d16 self.qkv = nn.Linear(D, 3 * D) # Q_s, K_s, V_s17 self.qkv_g = nn.Linear(D, 3 * D) # Q_g, K_g, V_g18 self.qkv_g.load_state_dict(self.qkv.state_dict()) # initialised to match19 self.out = nn.Linear(D, D)2021 def heads(self, t): # (B, n, D) -> (B, h, n, D/h)22 B, n, D = t.shape23 return t.view(B, n, self.h, D // self.h).transpose(1, 2)2425 def forward(self, x, global_idx):26 B, n, D = x.shape27 scale = (D // self.h) ** -0.528 q, k, v = map(self.heads, self.qkv(x).chunk(3, dim=-1))29 qg, kg, vg = map(self.heads, self.qkv_g(x).chunk(3, dim=-1))30 mask = longformer_mask(n, self.w, self.d, global_idx).to(x.device)31 s = (q @ k.transpose(-1, -2)) * scale # (B, h, n, n)32 y = s.masked_fill(~mask, float("-inf")).softmax(-1) @ v33 g = list(global_idx) # global rows: all keys, global projections34 y[:, :, g] = ((qg[:, :, g] @ kg.transpose(-1, -2)) * scale).softmax(-1) @ vg35 return self.out(y.transpose(1, 2).reshape(B, n, D))3637x = torch.randn(2, 1024, 768)38attn = LongformerSelfAttention(768, heads=12, w=128)39print(attn(x, global_idx=[0]).shape) # torch.Size([2, 1024, 768])40print(longformer_mask(1024, 128, global_idx=[0]).sum().item()) # 129854 of 10485764142# The released model: window 512 in every layer, global attention chosen per input43from transformers import LongformerModel, LongformerTokenizer44tok = LongformerTokenizer.from_pretrained("allenai/longformer-base-4096")45model = LongformerModel.from_pretrained("allenai/longformer-base-4096")46enc = tok("a long document " * 3000, return_tensors="pt", truncation=True, max_length=4096)47glob = torch.zeros_like(enc.input_ids)48glob[:, 0] = 1 # <s> ([CLS]) attends globally49out = model(**enc, global_attention_mask=glob)50print(out.last_hidden_state.shape) # torch.Size([1, 4096, 768])51print(model.config.attention_window[:3]) # [512, 512, 512]Use when
- + time and memory; 12% of full attention at 4,096 tokens and falling as grows
- +Warm-starts from RoBERTa: same per-token cost, copied position embeddings, short continued pretraining
- +Global tokens give a task-controlled route for whole-document information in one layer
- +Dilation enlarges the receptive field to at no extra cost
Avoid when
- −Global positions are chosen by hand per task; a wrong choice silently hurts
- −Two projection sets for global attention add parameters and complexity
- −Banded and dilated patterns need custom kernels to be fast; the chunked version wastes about half its work
- −Long-range information between ordinary tokens still needs several layers or a global hop
Check yourself
A Longformer layer uses w = 512, d = 1 and no global tokens. What is true of query 2,000 in a 4,096-token input?
Check yourself
Why does a QA model built on Longformer make the question tokens global?
Remember
- ✓Longformer attention: window of keys per side (dilated by ) plus symmetric global tokens; cost , receptive field after layers.
- ✓longformer-base-4096: RoBERTa warm start, in every layer, 4,096 tokens, separate projections for the global rows.
- ✓At , the window computes 2,035,456 scores per head per layer, 12.13% of full attention.
- ✓Global positions are task-specific: [CLS] for classification, the question tokens for QA.
- ✓LED extends the idea to an encoder–decoder initialised from BART, with 16K-token inputs for summarisation.