# Longformer

> Longformer replaces full self-attention with a sliding window of w keys per query (optionally dilated) plus a few task-chosen global tokens that attend to and are attended by every position, so cost grows as O(n·w) and a RoBERTa-style encoder reads 4,096 tokens.

Source: https://www.mlfoundry.in/learn/longformer  
Section: Deep Learning › Attention and Transformer Architectures  
Level: advanced · 14 min read · updated 2026-10-12

**Longformer:** is a Transformer whose self-attention combines three patterns: a **sliding window** in which each token attends to $w/2$ neighbours on each side, an optional **dilated** sliding window whose keys are $d$ positions apart, and **global attention** on a few task-chosen positions, which attend to every token and are attended by every token. The cost per layer is $O(n\cdot w)$ instead of $O(n^2)$, so the released encoder processes documents of 4,096 tokens. It was introduced by Beltagy, Peters and Cohan, "Longformer: The Long-Document Transformer" (Allen Institute for AI, arXiv 2020).

BERT and RoBERTa read at most 512 tokens. For longer documents the usual workaround was to cut the text into 512-token pieces, encode each piece separately and combine the predictions with task-specific machinery. Information could not flow between pieces inside the encoder. Iz Beltagy, Matthew Peters and Arman Cohan asked for a drop-in replacement for full **Self-attention** that scales linearly with length, keeps the RoBERTa weights usable, and still lets a task route information across the whole document. Earlier long-sequence models such as **Transformer-XL** and the **Sparse attention** of Child et al. (2019) had been evaluated mainly on autoregressive language modelling. Longformer was evaluated both on character-level language modelling and as a pretrained encoder fine-tuned on long-document question answering, coreference and classification.

## Three attention patterns

$$
\mathcal N_{\text{win}}(i)=\Big\{\,j:\ |i-j|\le \tfrac{w}{2}\,d,\ \ (i-j)\bmod d=0\Big\},\qquad \mathcal N(i)=\begin{cases}\{1,\dots,n\} & i\in G\\[2pt] \mathcal N_{\text{win}}(i)\cup G & i\notin G\end{cases}
$$

- $w$: window size: $w/2$ keys on each side of the query, $w+1$ keys including itself (512 in the pretrained model)
- $d$: dilation: $d=1$ is the plain sliding window; $d>1$ leaves gaps of $d-1$ positions, like a **Dilated convolution**
- $G$: global positions chosen per task, e.g. [CLS] for classification or all question tokens for QA
- $\mathcal N(i)$: the keys query $i$ attends to; softmax runs only over this set. Global attention is **symmetric**: $i\in G$ sees everything and everyone sees $i$

A sliding window alone is local, but stacking layers widens the view, as in a CNN. After $\ell$ layers a token has read $\ell\times w$ positions. With dilation the window spans $w\,d$ positions using the same $w$ keys, and the top layer reaches $\ell\times d\times w$. Dilation alone, though, only ever reaches positions in the same residue class modulo $d$. In practice it is used on some heads and mixed with undilated heads. For tasks that need one token to summarise the whole input, waiting for the receptive field to grow is not enough. A global token reads every position in one layer and can broadcast the result back in the next.

$$
\text{local: } \mathrm{softmax}\Big(\frac{Q_sK_s^\top}{\sqrt{d_k}}+M\Big)V_s,\qquad \text{global rows: } \mathrm{softmax}\Big(\frac{Q_g K_g^\top}{\sqrt{d_k}}\Big)V_g,\qquad M_{ij}=\begin{cases}0 & j\in\mathcal N(i)\\ -\infty & \text{otherwise}\end{cases}
$$

- $Q_s,K_s,V_s$: projections used for the sliding-window scores (and for ordinary tokens attending to global keys)
- $Q_g,K_g,V_g$: a second set of projections for the global-token rows; initialised to the values of Q_s, K_s, V_s
- $M$: the band-plus-global mask; real implementations never materialise the $n\times n$ matrix

The paper uses **two sets of projections**. Global attention plays a different role from local attention: one is a document-wide summary and the other a local context model. Separate $Q_g,K_g,V_g$ let the two specialise, and the paper's ablation found they help on downstream tasks. The scores are the usual **Scaled dot-product attention**; only the set of keys changes.

**Setup in the paper**

- **Pretrained encoder:** continued MLM pretraining from the RoBERTa checkpoint; window $w=512$ in every layer (the same per-token cost as RoBERTa's 512-token full attention); 65K gradient updates at sequence length 4,096
- **Position embeddings:** RoBERTa's 512 learned positions copied 8 times to fill 4,096; before any further training this already gave a usable MLM loss, while random position embeddings did not
- **Global tokens:** [CLS] for classification; all question tokens for QA (WikiHop, TriviaQA, HotpotQA)
- **Character LM:** window size increasing from lower to higher layers; dilation only on 2 heads of the higher layers; staged training that doubles sequence length and window across phases
- **Implementations:** *loop* (memory-efficient, slow), *chunks* (no dilation; used for pretraining and fine-tuning), and a custom CUDA kernel written with TVM that supports dilation
- **Encoder–decoder:** LED (Longformer-Encoder-Decoder), initialised from BART, with inputs up to 16K tokens for long-document summarisation such as arXiv

> Figure: One layer's attention pattern on 32 tokens: rows are queries, columns keys. With w = 4, d = 1 and no global tokens each interior query reads 5 keys, the layer computes 154 of 1,024 scores (15.0%), token 16 has read 13 tokens after 3 layers, and it needs 8 layers to read all 32 (the paper's L·d·w is 12 at L = 3). Switch to d = 2 with all heads dilated: token 16 never reads more than its own 16 even positions. With half the heads dilated it reads everything after 5 layers. Add a [CLS] global token: two layers suffice, because every token reads [CLS] and [CLS] reads every token. The global row and column also appear in the count (212 scores; 374 with a 4-token question). The right panel counts the real longformer-base-4096 setting exactly: 2,035,456 scores per head per layer with no global token, 12.13% of the 16,777,216 for full attention (7.8 vs 64 MiB in fp32), and 2,043,134 with [CLS] global.

## Worked example: counting scores at n = 4096

Attention scores per head per layer, n = 4096, w = 512 (computed exactly in Node, boundary effects included)

| pattern | scores | fraction of n² | fp32 memory |
| --- | --- | --- | --- |
| full attention | 16,777,216 | 100% | 64 MiB |
| sliding window, d = 1 | 2,035,456 | 12.13% | 7.76 MiB |
| dilated window, d = 2 | 1,969,664 | 11.74% | 7.51 MiB |
| window + 1 global ([CLS]) | 2,043,134 | 12.18% | 7.79 MiB |
| window + 64 global (a long question) | 2,522,816 | 15.04% | 9.62 MiB |

**Where the numbers come from**

1. An interior query reads $w+1=513$ keys. Near the edges the window is cut: token $i<256$ reads only $257+i$ keys. Total $=4096\cdot513-2\sum_{k=1}^{256}k=2{,}101{,}248-65{,}792=2{,}035{,}456$.
2. Ratio to full attention: $2{,}035{,}456/4096^2=12.13\%$. Doubling $n$ doubles the banded count but quadruples $n^2$, so the fraction halves with every doubling of length.
3. One global token at position 0: its row grows from 257 keys to 4,096 ($+3{,}839$), and the 3,839 queries $i>256$ whose window does not contain it gain one key each ($+3{,}839$). Total $2{,}043{,}134$.
4. Memory for all 12 heads and 12 layers if the probabilities are kept for the backward pass: $144\cdot16{,}777{,}216\cdot4$ B $=9$ GiB for full attention against $144\cdot2{,}043{,}134\cdot4$ B $\approx1.10$ GiB for Longformer.
5. Receptive field with $w=512$ and 12 layers: $12\cdot512=6{,}144>4{,}096$ positions, so even without global tokens the top layer can see the whole input.

## Results

Fine-tuned base-size encoders (Beltagy et al. 2020, Table 7; QA F1 or accuracy, OntoNotes average F1, IMDB and Hyperpartisan accuracy/F1)

| model | WikiHop | TriviaQA | HotpotQA | OntoNotes | IMDB | Hyperpartisan |
| --- | --- | --- | --- | --- | --- | --- |
| RoBERTa-base (512 tokens, chunked) | 72.4 | 74.3 | 63.5 | 78.4 | 95.3 | 87.4 |
| Longformer-base (4,096 tokens) | 75.0 | 75.2 | 64.4 | 78.6 | 95.7 | 94.8 |

The gains are largest where evidence is spread over long inputs (WikiHop, Hyperpartisan) and smallest where most documents already fit in 512 tokens (IMDB) or the task is local (coreference). Longformer-large set new state-of-the-art results on WikiHop and TriviaQA at publication. In character-level language modelling the small model reached 1.10 bits per character on text8 and 1.00 on enwik8, and the large model 0.99 on enwik8, matching or beating Transformer-XL-style models of comparable size.

```python
import torch, torch.nn as nn

def longformer_mask(n, w, d=1, global_idx=()):
    i, j = torch.arange(n)[:, None], torch.arange(n)[None, :]
    off = (i - j).abs()
    m = (off <= (w // 2) * d) & (off % d == 0)           # (dilated) sliding window
    g = torch.zeros(n, dtype=torch.bool)
    g[list(global_idx)] = True
    return m | g[:, None] | g[None, :]                    # global rows and columns

class LongformerSelfAttention(nn.Module):
    """Same output as the banded kernels, but O(n^2) memory: use it to check them."""
    def __init__(self, D, heads, w, d=1):
        super().__init__()
        self.h, self.w, self.d = heads, w, d
        self.qkv = nn.Linear(D, 3 * D)                     # Q_s, K_s, V_s
        self.qkv_g = nn.Linear(D, 3 * D)                   # Q_g, K_g, V_g
        self.qkv_g.load_state_dict(self.qkv.state_dict())  # initialised to match
        self.out = nn.Linear(D, D)

    def heads(self, t):                                    # (B, n, D) -> (B, h, n, D/h)
        B, n, D = t.shape
        return t.view(B, n, self.h, D // self.h).transpose(1, 2)

    def forward(self, x, global_idx):
        B, n, D = x.shape
        scale = (D // self.h) ** -0.5
        q, k, v = map(self.heads, self.qkv(x).chunk(3, dim=-1))
        qg, kg, vg = map(self.heads, self.qkv_g(x).chunk(3, dim=-1))
        mask = longformer_mask(n, self.w, self.d, global_idx).to(x.device)
        s = (q @ k.transpose(-1, -2)) * scale              # (B, h, n, n)
        y = s.masked_fill(~mask, float("-inf")).softmax(-1) @ v
        g = list(global_idx)                               # global rows: all keys, global projections
        y[:, :, g] = ((qg[:, :, g] @ kg.transpose(-1, -2)) * scale).softmax(-1) @ vg
        return self.out(y.transpose(1, 2).reshape(B, n, D))

x = torch.randn(2, 1024, 768)
attn = LongformerSelfAttention(768, heads=12, w=128)
print(attn(x, global_idx=[0]).shape)                     # torch.Size([2, 1024, 768])
print(longformer_mask(1024, 128, global_idx=[0]).sum().item())   # 129854 of 1048576

# The released model: window 512 in every layer, global attention chosen per input
from transformers import LongformerModel, LongformerTokenizer
tok = LongformerTokenizer.from_pretrained("allenai/longformer-base-4096")
model = LongformerModel.from_pretrained("allenai/longformer-base-4096")
enc = tok("a long document " * 3000, return_tensors="pt", truncation=True, max_length=4096)
glob = torch.zeros_like(enc.input_ids)
glob[:, 0] = 1                                           # <s> ([CLS]) attends globally
out = model(**enc, global_attention_mask=glob)
print(out.last_hidden_state.shape)                       # torch.Size([1, 4096, 768])
print(model.config.attention_window[:3])                 # [512, 512, 512]
```

A reference Longformer attention (dense mask, for clarity) and the Hugging Face model

> **Forgetting the global attention mask:** In the Hugging Face model, `global_attention_mask` defaults to none. A classification head reading a [CLS] token that only saw its 256 right-hand neighbours receives a poor summary of a 4,096-token document. Set the mask on [CLS] for classification and on the question tokens for QA, as the paper did. Inputs are also padded internally to a multiple of the window size, so very short inputs pay for a full 512-token window.

> **Why the RoBERTa weights transfer:** With $w=512$, each token sees exactly as many keys as in RoBERTa's 512-token full attention, and the copied position embeddings make every 512-token stretch look like a RoBERTa input. Before any extra training the model therefore behaves like RoBERTa applied to overlapping windows, and continued MLM pretraining only has to teach it to use the longer context. This is why Longformer needed only 65K updates rather than pretraining from scratch.

> **Sparse is not automatically fast:** A banded matrix product is awkward on GPUs, which are built for dense tiles. The *chunks* implementation splits the sequence into overlapping blocks of size $w$ and runs dense products on them, computing about twice the needed scores but at full speed. Dilation breaks this trick and needed the custom kernel. Block-sparse patterns such as **BigBird** were designed with this hardware constraint in mind, and kernels such as **FlashAttention** later made long full attention cheap enough that many models dropped the window altogether.

**Strengths**

- $O(n\cdot w)$ time and memory; 12% of full attention at 4,096 tokens and falling as $n$ grows
- Warm-starts from RoBERTa: same per-token cost, copied position embeddings, short continued pretraining
- Global tokens give a task-controlled route for whole-document information in one layer
- Dilation enlarges the receptive field to $\ell\times d\times w$ at no extra cost

**Limitations**

- Global positions are chosen by hand per task; a wrong choice silently hurts
- Two projection sets for global attention add parameters and complexity
- Banded and dilated patterns need custom kernels to be fast; the chunked version wastes about half its work
- Long-range information between ordinary tokens still needs several layers or a global hop

**Question:** A Longformer layer uses w = 512, d = 1 and no global tokens. What is true of query 2,000 in a 4,096-token input?

- A. It attends to 513 keys: positions 1,744 to 2,256
- B. It attends to 512 keys: positions 1,488 to 2,000
- C. It attends to all 4,096 keys, but most weights are masked to zero after the softmax
- D. It attends to 1,025 keys because w is the window on each side

**Answer:** A. It attends to 513 keys: positions 1,744 to 2,256 — $w$ is the total window: $w/2=256$ keys on each side plus the query itself, so $2000\pm256$, 513 keys. The mask is applied *before* the softmax, so the other positions get exactly zero weight and, in a real kernel, are never computed.

**Question:** Why does a QA model built on Longformer make the question tokens global?

- A. Global tokens use a larger window, so they see more of their neighbourhood
- B. Every document token can compare itself to the whole question in one layer, and each question token can gather evidence from the whole document
- C. Global attention is cheaper than sliding-window attention
- D. It removes the need for position embeddings

**Answer:** B. Every document token can compare itself to the whole question in one layer, and each question token can gather evidence from the whole document — Global attention is symmetric: a global token attends to all $n$ positions and all positions attend to it. Question–document matching then happens in one layer instead of waiting $n/w$ layers for the receptive field. The cost is one extra row and column per global token, $O(n)$ each.

**Key points**

- Longformer attention: window of $w/2$ keys per side (dilated by $d$) plus symmetric global tokens; cost $O(n\cdot w)$, receptive field $\ell\times d\times w$ after $\ell$ layers.
- longformer-base-4096: RoBERTa warm start, $w=512$ in every layer, 4,096 tokens, separate $Q_g,K_g,V_g$ projections for the global rows.
- At $n=4096$, $w=512$ the window computes 2,035,456 scores per head per layer, 12.13% of full attention.
- Global positions are task-specific: [CLS] for classification, the question tokens for QA.
- LED extends the idea to an encoder–decoder initialised from BART, with 16K-token inputs for summarisation.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
