# Document classification

> Classify texts longer than a 512-token encoder window: truncate (head, or head 128 + tail 382), chunk and pool, encode words then sentences (HAN), or read it all with a long encoder like Longformer.

Source: https://www.mlfoundry.in/learn/document-classification  
Section: Deep Learning › Deep Learning for NLP  
Level: core · 8 min read · updated 2026-10-15

**Document classification:** is **Text classification** where the input is a whole document (a review, article, patent, contract or medical record), often thousands of tokens long. The modelling question is how to turn a text far longer than the encoder's context window into one vector for the classifier head. The main answers are truncation, chunk-and-pool, hierarchical models such as the Hierarchical Attention Network (Yang et al., 2016), and long-context encoders such as **Longformer** (Beltagy, Peters and Cohan, 2020).

The label set and head are the same as for sentences: one vector $h$, then $\mathrm{softmax}(Wh+b)$, or per-class sigmoids for multi-label tasks such as legal or medical coding. What changes is the length.

**BERT** and RoBERTa have 512 learned position embeddings, so they cannot read token 513. Self-attention cost also grows as $n^2$: a 4,096-token document needs 64 times the attention scores of one 512-token pass, per layer and per head. Before Transformers, bag-of-words and TF-IDF linear models handled any length, and they are still a strong baseline for topic labels.

## Four strategies

> Figure: A 3,000-token review and a 512-token encoder. (1) Truncation keeps the first 512 tokens, or the first 128 plus the last 382, and never reads the rest. (2) Chunk-and-pool runs the encoder on six 512-token chunks with shared weights and pools the six [CLS] vectors. (3) A Hierarchical Attention Network encodes words into sentence vectors and sentences into a document vector, with attention at both levels (thicker lines: higher weight). (4) A long encoder such as Longformer reads 4,096 tokens in one pass.

**Truncation** is the default in most libraries (`truncation=True, max_length=512`). It is cheap and works when the label is decided early, as in news topics. Sun, Qiu, Xu and Huang (2019), in "How to Fine-Tune BERT for Text Classification?", compared head-only, tail-only and head+tail truncation on IMDb and Sogou News. Keeping the first 128 and the last 382 tokens worked best: reviews often state the verdict at the end.

**Chunk-and-pool** splits the document into $m$ windows (optionally overlapping), encodes each one, and combines the chunk vectors $h_1,\dots,h_m$:

$$
h_{\text{doc}}=\frac1m\sum_{j=1}^{m}h_j\;\;\text{(mean)}\qquad\text{or}\qquad h_{\text{doc}}=\max_j h_j\;\;\text{(elementwise max)},\qquad m=\Big\lceil\frac{n}{510}\Big\rceil
$$

- $h_j$: [CLS] vector of chunk $j$ (510 content tokens + [CLS] + [SEP])
- $n$: document length in tokens; m is for non-overlapping chunks

Mean pooling weights every chunk equally, so a decisive paragraph in a long document is diluted by $1/m$. Max pooling keeps the strongest signal, but only if one chunk contains the evidence. Learned pooling (attention, or a small LSTM or Transformer over chunk vectors, as in Pappagari et al.'s 2019 RoBERT and ToBERT) lets the model decide.

**Hierarchical Attention Networks** (Yang, Yang, Dyer, He, Smola and Hovy, 2016) build the hierarchy into the architecture. A bidirectional GRU reads the words of each sentence and attention pools them into a sentence vector $s_i$. A second bidirectional GRU reads the sentence vectors and attention pools them into the document vector $v$:

$$
u_{it}=\tanh(W_w h_{it}+b_w),\qquad \alpha_{it}=\frac{\exp(u_{it}^\top u_w)}{\sum_{t'}\exp(u_{it'}^\top u_w)},\qquad s_i=\sum_t\alpha_{it}\,h_{it}
$$

- $h_{it}$: bi-GRU state of word $t$ in sentence $i$
- $u_w$: learned word-level context vector (“which words are informative?”)
- $s_i$: sentence vector; the same three equations with $u_s$ over sentence states give $v$, and $p=\mathrm{softmax}(W_c v+b_c)$

HAN outperformed earlier models on six large review and question datasets (Yelp 2013–2015, IMDb, Yahoo Answers, Amazon), and its two attention layers show which sentences and words drove the decision. It has no length limit, but it is trained from scratch, without the pretraining that makes Transformers strong.

**Long encoders** remove the 512 limit. Longformer replaces full attention with a sliding window ($w=512$) plus global attention on chosen tokens. For classification the [CLS] token is global, so it attends to all 4,096 positions and they all attend to it. **BigBird** (Zaheer et al., 2020) adds random attention to the same pattern. Both cost $O(n\cdot w)$, and stacked layers let information travel across windows, which chunking cannot do.

**Long-document benchmarks**

- **IMDb:** 50,000 movie reviews, binary; many exceed 512 BERT tokens. Used by Sun et al. (2019) and Longformer.
- **Hyperpartisan:** News articles labelled as hyperpartisan or not (SemEval-2019 Task 4); small and long, a standard Longformer and BigBird test.
- **Yelp, Yahoo Answers, Amazon:** Large review and Q&A datasets used by HAN, with 5- or 10-way labels.

> Figure: A toy document of 128-token segments whose background is weakly negative, plus one 256-token span of positive evidence. Each window's logit is the mean of its segments. Default (4,096 tokens, evidence in the middle at tokens 1,920–2,176): head, tail and head+tail never see it (p = 0.370, 0.322, 0.351). Chunk-mean sees it but divides it among 8 chunks (p = 0.409, still wrong). Chunk-max is right but weak (p = 0.578) because the span straddles the chunk boundary at 2,048. Switch to stride 256: one window holds the whole span, and max rises to 0.735 at the price of 15 passes instead of 8. Move the evidence to 0% and head truncation wins (0.758). The cost panel shows full attention at 16.78M scores against 2.10M for chunks or a 512-token sliding window.

Worked example: a 3,000-token review and a 512-position encoder (attention scores per layer per head)

| Strategy | Tokens read | Encoder passes | Attention scores |
| --- | --- | --- | --- |
| Head truncation | 510 of 3,000 (17%) | 1 | $512^2=262{,}144$ |
| Head 128 + tail 382 | 510 of 3,000 (17%) | 1 | 262,144 |
| Chunk-and-pool | all 3,000 | $\lceil3000/510\rceil=6$ | $6\cdot512^2=1{,}572{,}864$ |
| Longformer, w = 512, [CLS] global | all 3,000 | 1 | $3000\cdot512+2\cdot3000\approx1.54$M |
| Full attention (hypothetical) | all 3,000 | 1 | $3000^2=9{,}000{,}000$ |

Full attention over 3,000 tokens would cost 5.7 times as much as six chunks, and 34 times one truncated pass. Chunking and the sliding window cost about the same. The difference is that Longformer's [CLS] token can combine evidence from every chunk inside the network, while chunk-and-pool combines it only at the pooling step.

```python
import torch
from transformers import AutoModelForSequenceClassification, AutoTokenizer

tok = AutoTokenizer.from_pretrained("bert-base-uncased")
model = AutoModelForSequenceClassification.from_pretrained("bert-base-uncased", num_labels=2)
text = open("review.txt").read()                     # e.g. 3,000 tokens

# 1) head 128 + tail 382 (+ [CLS], [SEP] = 512)
ids = tok(text, add_special_tokens=False)["input_ids"]
ids = ids[:128] + ids[-382:] if len(ids) > 510 else ids
batch = {"input_ids": torch.tensor([tok.build_inputs_with_special_tokens(ids)])}
p_headtail = model(**batch).logits.softmax(-1)

# 2) overlapping 512-token chunks (128 tokens shared between neighbours), then max-pool
enc = tok(text, truncation=True, max_length=512, stride=128,
          return_overflowing_tokens=True, padding=True, return_tensors="pt")
enc.pop("overflow_to_sample_mapping")
with torch.no_grad():
    chunk_logits = model(**enc).logits               # (m, 2)
p_chunks = chunk_logits.max(dim=0).values.softmax(-1)

# 3) a long encoder: 4,096 positions; the classification model puts global attention on <s> ([CLS]) by default
name = "allenai/longformer-base-4096"
ltok = AutoTokenizer.from_pretrained(name)
lmodel = AutoModelForSequenceClassification.from_pretrained(name, num_labels=2)
lbatch = ltok(text, truncation=True, max_length=4096, return_tensors="pt")
p_long = lmodel(**lbatch).logits.softmax(-1)          # fine-tune before trusting it
```

Head+tail truncation, overlapping chunks with max pooling, and Longformer (Hugging Face; not run here)

> **Silent truncation:** Tokenizers truncate without warning when `truncation=True`, and many pipelines set it by default. Check the token-length distribution of your corpus first. If half the documents exceed 512 tokens and the label depends on the conclusion, head truncation discards the evidence, and the model learns shortcuts from the opening paragraphs instead.

> **Max pooling and chunk boundaries:** Max pooling over non-overlapping chunks fails when the decisive passage is cut in half. Overlapping windows (stride smaller than the window) guarantee that any span shorter than the overlap lies entirely in some chunk, at roughly window/stride times the compute.

- **Legal and patent:** Contract clause types, court-case outcome and patent classification codes: long, structured documents, often multi-label.
- **Clinical coding:** Assigning diagnosis codes to discharge summaries: thousands of tokens, thousands of possible labels, evidence scattered across the note.
- **Reviews and news:** Long-review sentiment (see **Sentiment analysis**), news topic and hyperpartisan detection.

**Question:** A 6,000-token document is split into non-overlapping 510-token chunks for mean pooling. One chunk contains the only decisive passage. What weight does that chunk get in the document vector?

- A. 1/2
- B. 1/6
- C. 1/12
- D. 1, because the encoder sees all of it

**Answer:** C. 1/12 — $m=\lceil6000/510\rceil=12$ chunks, so mean pooling gives each $1/12$. The other 11 chunks can outvote it, which is why max or learned attention pooling is often better when evidence is local.

**Question:** Why did Sun et al. (2019) find head 128 + tail 382 better than head-only truncation on IMDb?

- A. The tail of a review often states the overall verdict, so keeping it preserves the evidence
- B. Tail tokens have better position embeddings
- C. Shorter inputs train faster
- D. It removes the [CLS] token

**Answer:** A. The tail of a review often states the overall verdict, so keeping it preserves the evidence — Both read 510 tokens. Reviewers often summarise their opinion in the last lines, so keeping the end recovers evidence that head-only truncation drops.

**Key points**

- BERT-style encoders stop at 512 positions, and full self-attention grows as n².
- Truncation: one pass, blind to the middle; head 128 + tail 382 is a strong default for reviews.
- Chunk-and-pool: $\lceil n/510\rceil$ passes; mean pooling dilutes local evidence by $1/m$, max needs the evidence inside one chunk (use overlap).
- HAN: bi-GRU + attention over words, then over sentences; no length limit, interpretable weights.
- Longformer/BigBird: sliding window + global [CLS], $O(n\cdot w)$, 4,096 tokens in one pass with cross-window context.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
