Longformer

Longformer replaces full self-attention with a sliding window of w keys per query (optionally dilated) plus a few task-chosen global tokens that attend to and are attended by every position, so cost grows as O(n·w) and a RoBERTa-style encoder reads 4,096 tokens.

14 min read895 wordsadvanced0 viewsupdated 2026-10-12

Definition

Longformer is a Transformer whose self-attention combines three patterns: a sliding window in which each token attends to w/2w/2 neighbours on each side, an optional dilated sliding window whose keys are dd positions apart, and global attention on a few task-chosen positions, which attend to every token and are attended by every token. The cost per layer is O(n⋅w)O(n\cdot w) instead of O(n2)O(n^2), so the released encoder processes documents of 4,096 tokens. It was introduced by Beltagy, Peters and Cohan, "Longformer: The Long-Document Transformer" (Allen Institute for AI, arXiv 2020).

BERT and RoBERTa read at most 512 tokens. For longer documents the usual workaround was to cut the text into 512-token pieces, encode each piece separately and combine the predictions with task-specific machinery. Information could not flow between pieces inside the encoder. Iz Beltagy, Matthew Peters and Arman Cohan asked for a drop-in replacement for full Self-attention that scales linearly with length, keeps the RoBERTa weights usable, and still lets a task route information across the whole document. Earlier long-sequence models such as Transformer-XL and the Sparse attention of Child et al. (2019) had been evaluated mainly on autoregressive language modelling. Longformer was evaluated both on character-level language modelling and as a pretrained encoder fine-tuned on long-document question answering, coreference and classification.

Three attention patterns

Nwin(i)={ j: ∣i−j∣≤w2 d,  (i−j) mod d=0},N(i)={{1,…,n}i∈GNwin(i)∪Gi∉G\mathcal N_{\text{win}}(i)=\Big\{\,j:\ |i-j|\le \tfrac{w}{2}\,d,\ \ (i-j)\bmod d=0\Big\},\qquad \mathcal N(i)=\begin{cases}\{1,\dots,n\} & i\in G\\[2pt] \mathcal N_{\text{win}}(i)\cup G & i\notin G\end{cases}
ww
window size: w/2w/2 keys on each side of the query, w+1w+1 keys including itself (512 in the pretrained model)
dd
dilation: d=1d=1 is the plain sliding window; d>1d>1 leaves gaps of d−1d-1 positions, like a Dilated convolution
GG
global positions chosen per task, e.g. [CLS] for classification or all question tokens for QA
N(i)\mathcal N(i)
the keys query ii attends to; softmax runs only over this set. Global attention is symmetric: i∈Gi\in G sees everything and everyone sees ii

A sliding window alone is local, but stacking layers widens the view, as in a CNN. After ℓ\ell layers a token has read ℓ×w\ell\times w positions. With dilation the window spans w dw\,d positions using the same ww keys, and the top layer reaches ℓ×d×w\ell\times d\times w. Dilation alone, though, only ever reaches positions in the same residue class modulo dd. In practice it is used on some heads and mixed with undilated heads. For tasks that need one token to summarise the whole input, waiting for the receptive field to grow is not enough. A global token reads every position in one layer and can broadcast the result back in the next.

local: softmax(QsKs⊤dk+M)Vs,global rows: softmax(QgKg⊤dk)Vg,Mij={0j∈N(i)−∞otherwise\text{local: } \mathrm{softmax}\Big(\frac{Q_sK_s^\top}{\sqrt{d_k}}+M\Big)V_s,\qquad \text{global rows: } \mathrm{softmax}\Big(\frac{Q_g K_g^\top}{\sqrt{d_k}}\Big)V_g,\qquad M_{ij}=\begin{cases}0 & j\in\mathcal N(i)\\ -\infty & \text{otherwise}\end{cases}
Qs,Ks,VsQ_s,K_s,V_s
projections used for the sliding-window scores (and for ordinary tokens attending to global keys)
Qg,Kg,VgQ_g,K_g,V_g
a second set of projections for the global-token rows; initialised to the values of Q_s, K_s, V_s
MM
the band-plus-global mask; real implementations never materialise the n×nn\times n matrix

The paper uses two sets of projections. Global attention plays a different role from local attention: one is a document-wide summary and the other a local context model. Separate Qg,Kg,VgQ_g,K_g,V_g let the two specialise, and the paper's ablation found they help on downstream tasks. The scores are the usual Scaled dot-product attention; only the set of keys changes.

Setup in the paper

Pretrained encoder
continued MLM pretraining from the RoBERTa checkpoint; window w=512w=512 in every layer (the same per-token cost as RoBERTa's 512-token full attention); 65K gradient updates at sequence length 4,096
Position embeddings
RoBERTa's 512 learned positions copied 8 times to fill 4,096; before any further training this already gave a usable MLM loss, while random position embeddings did not
Global tokens
[CLS] for classification; all question tokens for QA (WikiHop, TriviaQA, HotpotQA)
Character LM
window size increasing from lower to higher layers; dilation only on 2 heads of the higher layers; staged training that doubles sequence length and window across phases
Implementations
loop (memory-efficient, slow), chunks (no dilation; used for pretraining and fine-tuning), and a custom CUDA kernel written with TVM that supports dilation
Encoder–decoder
LED (Longformer-Encoder-Decoder), initialised from BART, with inputs up to 16K tokens for long-document summarisation such as arXiv
one layer: query i (row) attends key j (column) · w/2 = 2 each side, dilation d = 10088161624243131■ window ■ global (row and column) · click a rowtoken 16 after L layers: darker = read in fewer layersgrey = not reachable within L = 3this figure (n = 32)keys for query 16 5 of 32scores, this layer 154 of 1024 (15.0%)read after L = 3 13 of 32 tokenslayers to read all 8L·d·w (paper) 12 positionsheads: all dilatedglobal: nonelongformer-base-4096n = 4096, w = 512, d = 1, 0 globalscores / head / layer 2,035,456full attention 16,777,216fraction 12.13%fp32 scores 7.8 vs 64 MiBper query: w + 1 = 5 window keys (+ globals) · cost O(n·w), not O(n²)
window w
dilation dheads
global tokens
fig.One layer's attention pattern on 32 tokens: rows are queries, columns keys. With w = 4, d = 1 and no global tokens each interior query reads 5 keys, the layer computes 154 of 1,024 scores (15.0%), token 16 has read 13 tokens after 3 layers, and it needs 8 layers to read all 32 (the paper's L·d·w is 12 at L = 3). Switch to d = 2 with all heads dilated: token 16 never reads more than its own 16 even positions. With half the heads dilated it reads everything after 5 layers. Add a [CLS] global token: two layers suffice, because every token reads [CLS] and [CLS] reads every token. The global row and column also appear in the count (212 scores; 374 with a 4-token question). The right panel counts the real longformer-base-4096 setting exactly: 2,035,456 scores per head per layer with no global token, 12.13% of the 16,777,216 for full attention (7.8 vs 64 MiB in fp32), and 2,043,134 with [CLS] global.

Worked example: counting scores at n = 4096

patternscoresfraction of n²fp32 memory
full attention16,777,216100%64 MiB
sliding window, d = 12,035,45612.13%7.76 MiB
dilated window, d = 21,969,66411.74%7.51 MiB
window + 1 global ([CLS])2,043,13412.18%7.79 MiB
window + 64 global (a long question)2,522,81615.04%9.62 MiB
Attention scores per head per layer, n = 4096, w = 512 (computed exactly in Node, boundary effects included)

Where the numbers come from

  1. 1An interior query reads w+1=513w+1=513 keys. Near the edges the window is cut: token i<256i<256 reads only 257+i257+i keys. Total =4096⋅513−2∑k=1256k=2,101,248−65,792=2,035,456=4096\cdot513-2\sum_{k=1}^{256}k=2{,}101{,}248-65{,}792=2{,}035{,}456.
  2. 2Ratio to full attention: 2,035,456/40962=12.13%2{,}035{,}456/4096^2=12.13\%. Doubling nn doubles the banded count but quadruples n2n^2, so the fraction halves with every doubling of length.
  3. 3One global token at position 0: its row grows from 257 keys to 4,096 (+3,839+3{,}839), and the 3,839 queries i>256i>256 whose window does not contain it gain one key each (+3,839+3{,}839). Total 2,043,1342{,}043{,}134.
  4. 4Memory for all 12 heads and 12 layers if the probabilities are kept for the backward pass: 144⋅16,777,216⋅4144\cdot16{,}777{,}216\cdot4 B =9=9 GiB for full attention against 144⋅2,043,134⋅4144\cdot2{,}043{,}134\cdot4 B ≈1.10\approx1.10 GiB for Longformer.
  5. 5Receptive field with w=512w=512 and 12 layers: 12⋅512=6,144>4,09612\cdot512=6{,}144>4{,}096 positions, so even without global tokens the top layer can see the whole input.

Results

modelWikiHopTriviaQAHotpotQAOntoNotesIMDBHyperpartisan
RoBERTa-base (512 tokens, chunked)72.474.363.578.495.387.4
Longformer-base (4,096 tokens)75.075.264.478.695.794.8
Fine-tuned base-size encoders (Beltagy et al. 2020, Table 7; QA F1 or accuracy, OntoNotes average F1, IMDB and Hyperpartisan accuracy/F1)

The gains are largest where evidence is spread over long inputs (WikiHop, Hyperpartisan) and smallest where most documents already fit in 512 tokens (IMDB) or the task is local (coreference). Longformer-large set new state-of-the-art results on WikiHop and TriviaQA at publication. In character-level language modelling the small model reached 1.10 bits per character on text8 and 1.00 on enwik8, and the large model 0.99 on enwik8, matching or beating Transformer-XL-style models of comparable size.

python
1import torch, torch.nn as nn
2
3def longformer_mask(n, w, d=1, global_idx=()):
4 i, j = torch.arange(n)[:, None], torch.arange(n)[None, :]
5 off = (i - j).abs()
6 m = (off <= (w // 2) * d) & (off % d == 0) # (dilated) sliding window
7 g = torch.zeros(n, dtype=torch.bool)
8 g[list(global_idx)] = True
9 return m | g[:, None] | g[None, :] # global rows and columns
10
11class LongformerSelfAttention(nn.Module):
12 """Same output as the banded kernels, but O(n^2) memory: use it to check them."""
13 def __init__(self, D, heads, w, d=1):
14 super().__init__()
15 self.h, self.w, self.d = heads, w, d
16 self.qkv = nn.Linear(D, 3 * D) # Q_s, K_s, V_s
17 self.qkv_g = nn.Linear(D, 3 * D) # Q_g, K_g, V_g
18 self.qkv_g.load_state_dict(self.qkv.state_dict()) # initialised to match
19 self.out = nn.Linear(D, D)
20
21 def heads(self, t): # (B, n, D) -> (B, h, n, D/h)
22 B, n, D = t.shape
23 return t.view(B, n, self.h, D // self.h).transpose(1, 2)
24
25 def forward(self, x, global_idx):
26 B, n, D = x.shape
27 scale = (D // self.h) ** -0.5
28 q, k, v = map(self.heads, self.qkv(x).chunk(3, dim=-1))
29 qg, kg, vg = map(self.heads, self.qkv_g(x).chunk(3, dim=-1))
30 mask = longformer_mask(n, self.w, self.d, global_idx).to(x.device)
31 s = (q @ k.transpose(-1, -2)) * scale # (B, h, n, n)
32 y = s.masked_fill(~mask, float("-inf")).softmax(-1) @ v
33 g = list(global_idx) # global rows: all keys, global projections
34 y[:, :, g] = ((qg[:, :, g] @ kg.transpose(-1, -2)) * scale).softmax(-1) @ vg
35 return self.out(y.transpose(1, 2).reshape(B, n, D))
36
37x = torch.randn(2, 1024, 768)
38attn = LongformerSelfAttention(768, heads=12, w=128)
39print(attn(x, global_idx=[0]).shape) # torch.Size([2, 1024, 768])
40print(longformer_mask(1024, 128, global_idx=[0]).sum().item()) # 129854 of 1048576
41
42# The released model: window 512 in every layer, global attention chosen per input
43from transformers import LongformerModel, LongformerTokenizer
44tok = LongformerTokenizer.from_pretrained("allenai/longformer-base-4096")
45model = LongformerModel.from_pretrained("allenai/longformer-base-4096")
46enc = tok("a long document " * 3000, return_tensors="pt", truncation=True, max_length=4096)
47glob = torch.zeros_like(enc.input_ids)
48glob[:, 0] = 1 # <s> ([CLS]) attends globally
49out = model(**enc, global_attention_mask=glob)
50print(out.last_hidden_state.shape) # torch.Size([1, 4096, 768])
51print(model.config.attention_window[:3]) # [512, 512, 512]
A reference Longformer attention (dense mask, for clarity) and the Hugging Face model

Use when

  • +O(n⋅w)O(n\cdot w) time and memory; 12% of full attention at 4,096 tokens and falling as nn grows
  • +Warm-starts from RoBERTa: same per-token cost, copied position embeddings, short continued pretraining
  • +Global tokens give a task-controlled route for whole-document information in one layer
  • +Dilation enlarges the receptive field to ℓ×d×w\ell\times d\times w at no extra cost

Avoid when

  • −Global positions are chosen by hand per task; a wrong choice silently hurts
  • −Two projection sets for global attention add parameters and complexity
  • −Banded and dilated patterns need custom kernels to be fast; the chunked version wastes about half its work
  • −Long-range information between ordinary tokens still needs several layers or a global hop

Check yourself

A Longformer layer uses w = 512, d = 1 and no global tokens. What is true of query 2,000 in a 4,096-token input?

Check yourself

Why does a QA model built on Longformer make the question tokens global?

Remember

  • ✓Longformer attention: window of w/2w/2 keys per side (dilated by dd) plus symmetric global tokens; cost O(n⋅w)O(n\cdot w), receptive field ℓ×d×w\ell\times d\times w after ℓ\ell layers.
  • ✓longformer-base-4096: RoBERTa warm start, w=512w=512 in every layer, 4,096 tokens, separate Qg,Kg,VgQ_g,K_g,V_g projections for the global rows.
  • ✓At n=4096n=4096, w=512w=512 the window computes 2,035,456 scores per head per layer, 12.13% of full attention.
  • ✓Global positions are task-specific: [CLS] for classification, the question tokens for QA.
  • ✓LED extends the idea to an encoder–decoder initialised from BART, with 16K-token inputs for summarisation.
← back to Deep Learning roadmap