Definition
Document classification is Text classification where the input is a whole document (a review, article, patent, contract or medical record), often thousands of tokens long. The modelling question is how to turn a text far longer than the encoder's context window into one vector for the classifier head. The main answers are truncation, chunk-and-pool, hierarchical models such as the Hierarchical Attention Network (Yang et al., 2016), and long-context encoders such as Longformer (Beltagy, Peters and Cohan, 2020).
The label set and head are the same as for sentences: one vector , then , or per-class sigmoids for multi-label tasks such as legal or medical coding. What changes is the length.
BERT and RoBERTa have 512 learned position embeddings, so they cannot read token 513. Self-attention cost also grows as : a 4,096-token document needs 64 times the attention scores of one 512-token pass, per layer and per head. Before Transformers, bag-of-words and TF-IDF linear models handled any length, and they are still a strong baseline for topic labels.
Four strategies
Truncation is the default in most libraries (truncation=True, max_length=512). It is cheap and works when the label is decided early, as in news topics. Sun, Qiu, Xu and Huang (2019), in "How to Fine-Tune BERT for Text Classification?", compared head-only, tail-only and head+tail truncation on IMDb and Sogou News. Keeping the first 128 and the last 382 tokens worked best: reviews often state the verdict at the end.
Chunk-and-pool splits the document into windows (optionally overlapping), encodes each one, and combines the chunk vectors :
- [CLS] vector of chunk (510 content tokens + [CLS] + [SEP])
- document length in tokens; m is for non-overlapping chunks
Mean pooling weights every chunk equally, so a decisive paragraph in a long document is diluted by . Max pooling keeps the strongest signal, but only if one chunk contains the evidence. Learned pooling (attention, or a small LSTM or Transformer over chunk vectors, as in Pappagari et al.'s 2019 RoBERT and ToBERT) lets the model decide.
Hierarchical Attention Networks (Yang, Yang, Dyer, He, Smola and Hovy, 2016) build the hierarchy into the architecture. A bidirectional GRU reads the words of each sentence and attention pools them into a sentence vector . A second bidirectional GRU reads the sentence vectors and attention pools them into the document vector :
- bi-GRU state of word in sentence
- learned word-level context vector (“which words are informative?”)
- sentence vector; the same three equations with over sentence states give , and
HAN outperformed earlier models on six large review and question datasets (Yelp 2013–2015, IMDb, Yahoo Answers, Amazon), and its two attention layers show which sentences and words drove the decision. It has no length limit, but it is trained from scratch, without the pretraining that makes Transformers strong.
Long encoders remove the 512 limit. Longformer replaces full attention with a sliding window () plus global attention on chosen tokens. For classification the [CLS] token is global, so it attends to all 4,096 positions and they all attend to it. BigBird (Zaheer et al., 2020) adds random attention to the same pattern. Both cost , and stacked layers let information travel across windows, which chunking cannot do.
Long-document benchmarks
- IMDb
- 50,000 movie reviews, binary; many exceed 512 BERT tokens. Used by Sun et al. (2019) and Longformer.
- Hyperpartisan
- News articles labelled as hyperpartisan or not (SemEval-2019 Task 4); small and long, a standard Longformer and BigBird test.
- Yelp, Yahoo Answers, Amazon
- Large review and Q&A datasets used by HAN, with 5- or 10-way labels.
| Strategy | Tokens read | Encoder passes | Attention scores |
|---|---|---|---|
| Head truncation | 510 of 3,000 (17%) | 1 | |
| Head 128 + tail 382 | 510 of 3,000 (17%) | 1 | 262,144 |
| Chunk-and-pool | all 3,000 | ||
| Longformer, w = 512, [CLS] global | all 3,000 | 1 | M |
| Full attention (hypothetical) | all 3,000 | 1 |
Full attention over 3,000 tokens would cost 5.7 times as much as six chunks, and 34 times one truncated pass. Chunking and the sliding window cost about the same. The difference is that Longformer's [CLS] token can combine evidence from every chunk inside the network, while chunk-and-pool combines it only at the pooling step.
1import torch2from transformers import AutoModelForSequenceClassification, AutoTokenizer34tok = AutoTokenizer.from_pretrained("bert-base-uncased")5model = AutoModelForSequenceClassification.from_pretrained("bert-base-uncased", num_labels=2)6text = open("review.txt").read() # e.g. 3,000 tokens78# 1) head 128 + tail 382 (+ [CLS], [SEP] = 512)9ids = tok(text, add_special_tokens=False)["input_ids"]10ids = ids[:128] + ids[-382:] if len(ids) > 510 else ids11batch = {"input_ids": torch.tensor([tok.build_inputs_with_special_tokens(ids)])}12p_headtail = model(**batch).logits.softmax(-1)1314# 2) overlapping 512-token chunks (128 tokens shared between neighbours), then max-pool15enc = tok(text, truncation=True, max_length=512, stride=128,16 return_overflowing_tokens=True, padding=True, return_tensors="pt")17enc.pop("overflow_to_sample_mapping")18with torch.no_grad():19 chunk_logits = model(**enc).logits # (m, 2)20p_chunks = chunk_logits.max(dim=0).values.softmax(-1)2122# 3) a long encoder: 4,096 positions; the classification model puts global attention on <s> ([CLS]) by default23name = "allenai/longformer-base-4096"24ltok = AutoTokenizer.from_pretrained(name)25lmodel = AutoModelForSequenceClassification.from_pretrained(name, num_labels=2)26lbatch = ltok(text, truncation=True, max_length=4096, return_tensors="pt")27p_long = lmodel(**lbatch).logits.softmax(-1) # fine-tune before trusting itWhere it's used
Legal and patent
Contract clause types, court-case outcome and patent classification codes: long, structured documents, often multi-label.
Clinical coding
Assigning diagnosis codes to discharge summaries: thousands of tokens, thousands of possible labels, evidence scattered across the note.
Reviews and news
Long-review sentiment (see Sentiment analysis), news topic and hyperpartisan detection.
Check yourself
A 6,000-token document is split into non-overlapping 510-token chunks for mean pooling. One chunk contains the only decisive passage. What weight does that chunk get in the document vector?
Check yourself
Why did Sun et al. (2019) find head 128 + tail 382 better than head-only truncation on IMDb?
Remember
- ✓BERT-style encoders stop at 512 positions, and full self-attention grows as n².
- ✓Truncation: one pass, blind to the middle; head 128 + tail 382 is a strong default for reviews.
- ✓Chunk-and-pool: passes; mean pooling dilutes local evidence by , max needs the evidence inside one chunk (use overlap).
- ✓HAN: bi-GRU + attention over words, then over sentences; no length limit, interpretable weights.
- ✓Longformer/BigBird: sliding window + global [CLS], , 4,096 tokens in one pass with cross-window context.