Document classification

Classify texts longer than a 512-token encoder window: truncate (head, or head 128 + tail 382), chunk and pool, encode words then sentences (HAN), or read it all with a long encoder like Longformer.

8 min read708 wordscore0 viewsupdated 2026-10-15

Definition

Document classification is Text classification where the input is a whole document (a review, article, patent, contract or medical record), often thousands of tokens long. The modelling question is how to turn a text far longer than the encoder's context window into one vector for the classifier head. The main answers are truncation, chunk-and-pool, hierarchical models such as the Hierarchical Attention Network (Yang et al., 2016), and long-context encoders such as Longformer (Beltagy, Peters and Cohan, 2020).

The label set and head are the same as for sentences: one vector hh, then softmax(Wh+b)\mathrm{softmax}(Wh+b), or per-class sigmoids for multi-label tasks such as legal or medical coding. What changes is the length.

BERT and RoBERTa have 512 learned position embeddings, so they cannot read token 513. Self-attention cost also grows as n2n^2: a 4,096-token document needs 64 times the attention scores of one 512-token pass, per layer and per head. Before Transformers, bag-of-words and TF-IDF linear models handled any length, and they are still a strong baseline for topic labels.

Four strategies

a 3,000-token review, an encoder limited to 512 positions (BERT, RoBERTa)1 truncate(drop the rest)head 512head 128 + tail 382128 + 382 + [CLS] + [SEP] = 512 positions; head+tail never reads tokens 128–2,617encoder, 1 pass[CLS] → softmax2 chunk + pool(read everything)1enc2enc3enc4enc5enc6enc6 chunksshared weightsh₁ … h₆ ([CLS])pool: mean / max / attention / RNNdoc vector→ linear → softmax3 hierarchicalattention (HAN)Yang et al. 2016w1w2w3s1w1w2w3w4s2w1w2s3w1w2w3s4vsentences →words →v → softmaxno length limiteach level: bi-GRU, thenu = tanh(W h + b)α = softmax(uᵀ u_ctx)out = Σ α h (thick = high α)4 long encoderone pass, 4,096 tokensLongformer: sliding-window attention (512) on every token + global attention on [CLS];cost grows as n·w instead of n², and evidence anywhere in the 4,096 tokens reaches [CLS].1 cheapest, blind to the middle · 2, 3 read all, split context · 4 full context, more memory
fig.A 3,000-token review and a 512-token encoder. (1) Truncation keeps the first 512 tokens, or the first 128 plus the last 382, and never reads the rest. (2) Chunk-and-pool runs the encoder on six 512-token chunks with shared weights and pools the six [CLS] vectors. (3) A Hierarchical Attention Network encodes words into sentence vectors and sentences into a document vector, with attention at both levels (thicker lines: higher weight). (4) A long encoder such as Longformer reads 4,096 tokens in one pass.

Truncation is the default in most libraries (truncation=True, max_length=512). It is cheap and works when the label is decided early, as in news topics. Sun, Qiu, Xu and Huang (2019), in "How to Fine-Tune BERT for Text Classification?", compared head-only, tail-only and head+tail truncation on IMDb and Sogou News. Keeping the first 128 and the last 382 tokens worked best: reviews often state the verdict at the end.

Chunk-and-pool splits the document into mm windows (optionally overlapping), encodes each one, and combines the chunk vectors h1,…,hmh_1,\dots,h_m:

hdoc=1m∑j=1mhj    (mean)orhdoc=max⁡jhj    (elementwise max),m=⌈n510⌉h_{\text{doc}}=\frac1m\sum_{j=1}^{m}h_j\;\;\text{(mean)}\qquad\text{or}\qquad h_{\text{doc}}=\max_j h_j\;\;\text{(elementwise max)},\qquad m=\Big\lceil\frac{n}{510}\Big\rceil
hjh_j
[CLS] vector of chunk jj (510 content tokens + [CLS] + [SEP])
nn
document length in tokens; m is for non-overlapping chunks

Mean pooling weights every chunk equally, so a decisive paragraph in a long document is diluted by 1/m1/m. Max pooling keeps the strongest signal, but only if one chunk contains the evidence. Learned pooling (attention, or a small LSTM or Transformer over chunk vectors, as in Pappagari et al.'s 2019 RoBERT and ToBERT) lets the model decide.

Hierarchical Attention Networks (Yang, Yang, Dyer, He, Smola and Hovy, 2016) build the hierarchy into the architecture. A bidirectional GRU reads the words of each sentence and attention pools them into a sentence vector sis_i. A second bidirectional GRU reads the sentence vectors and attention pools them into the document vector vv:

uit=tanh⁡(Wwhit+bw),αit=exp⁡(uit⊤uw)∑t′exp⁡(uit′⊤uw),si=∑tαit hitu_{it}=\tanh(W_w h_{it}+b_w),\qquad \alpha_{it}=\frac{\exp(u_{it}^\top u_w)}{\sum_{t'}\exp(u_{it'}^\top u_w)},\qquad s_i=\sum_t\alpha_{it}\,h_{it}
hith_{it}
bi-GRU state of word tt in sentence ii
uwu_w
learned word-level context vector (“which words are informative?”)
sis_i
sentence vector; the same three equations with usu_s over sentence states give vv, and p=softmax(Wcv+bc)p=\mathrm{softmax}(W_c v+b_c)

HAN outperformed earlier models on six large review and question datasets (Yelp 2013–2015, IMDb, Yahoo Answers, Amazon), and its two attention layers show which sentences and words drove the decision. It has no length limit, but it is trained from scratch, without the pretraining that makes Transformers strong.

Long encoders remove the 512 limit. Longformer replaces full attention with a sliding window (w=512w=512) plus global attention on chosen tokens. For classification the [CLS] token is global, so it attends to all 4,096 positions and they all attend to it. BigBird (Zaheer et al., 2020) adds random attention to the same pattern. Both cost O(n⋅w)O(n\cdot w), and stacked layers let information travel across windows, which chunking cannot do.

Long-document benchmarks

IMDb
50,000 movie reviews, binary; many exceed 512 BERT tokens. Used by Sun et al. (2019) and Longformer.
Hyperpartisan
News articles labelled as hyperpartisan or not (SemEval-2019 Task 4); small and long, a standard Longformer and BigBird test.
Yelp, Yahoo Answers, Amazon
Large review and Q&A datasets used by HAN, with 5- or 10-way labels.
4,096 tokens = 32 segments of 128 · evidence (256 tokens, logit +3.0) at tokens 1,920–2,176gold label: positive · window logit = mean of its segment logits · background ≈ −0.6document04,096chunks (no overlap)logitp(pos)evidencehead 512−0.5330.370 ✗missedtail 512−0.7450.322 ✗missedhead 128 + tail 384−0.6140.351 ✗missedchunks → mean−0.3670.409 ✗1/8 weightchunks → max+0.3130.578 ✓best chunkattention scores per layer and head (query–key pairs)full attention over 4,09616.78M8 chunks × 512²2.10Msliding window w = 5122.10Mtruncate to 5120.26Mmean pooling dilutes the evidence by the chunk count; max keeps it if one chunk holds it all8 encoder passes of 512 tokens · full attention is 64× one 512-token passtoy model: segment logits are fixed numbers, not a trained encoder
chunk stride
fig.A toy document of 128-token segments whose background is weakly negative, plus one 256-token span of positive evidence. Each window's logit is the mean of its segments. Default (4,096 tokens, evidence in the middle at tokens 1,920–2,176): head, tail and head+tail never see it (p = 0.370, 0.322, 0.351). Chunk-mean sees it but divides it among 8 chunks (p = 0.409, still wrong). Chunk-max is right but weak (p = 0.578) because the span straddles the chunk boundary at 2,048. Switch to stride 256: one window holds the whole span, and max rises to 0.735 at the price of 15 passes instead of 8. Move the evidence to 0% and head truncation wins (0.758). The cost panel shows full attention at 16.78M scores against 2.10M for chunks or a 512-token sliding window.
StrategyTokens readEncoder passesAttention scores
Head truncation510 of 3,000 (17%)15122=262,144512^2=262{,}144
Head 128 + tail 382510 of 3,000 (17%)1262,144
Chunk-and-poolall 3,000⌈3000/510⌉=6\lceil3000/510\rceil=66⋅5122=1,572,8646\cdot512^2=1{,}572{,}864
Longformer, w = 512, [CLS] globalall 3,00013000⋅512+2⋅3000≈1.543000\cdot512+2\cdot3000\approx1.54M
Full attention (hypothetical)all 3,000130002=9,000,0003000^2=9{,}000{,}000
Worked example: a 3,000-token review and a 512-position encoder (attention scores per layer per head)

Full attention over 3,000 tokens would cost 5.7 times as much as six chunks, and 34 times one truncated pass. Chunking and the sliding window cost about the same. The difference is that Longformer's [CLS] token can combine evidence from every chunk inside the network, while chunk-and-pool combines it only at the pooling step.

python
1import torch
2from transformers import AutoModelForSequenceClassification, AutoTokenizer
3
4tok = AutoTokenizer.from_pretrained("bert-base-uncased")
5model = AutoModelForSequenceClassification.from_pretrained("bert-base-uncased", num_labels=2)
6text = open("review.txt").read() # e.g. 3,000 tokens
7
8# 1) head 128 + tail 382 (+ [CLS], [SEP] = 512)
9ids = tok(text, add_special_tokens=False)["input_ids"]
10ids = ids[:128] + ids[-382:] if len(ids) > 510 else ids
11batch = {"input_ids": torch.tensor([tok.build_inputs_with_special_tokens(ids)])}
12p_headtail = model(**batch).logits.softmax(-1)
13
14# 2) overlapping 512-token chunks (128 tokens shared between neighbours), then max-pool
15enc = tok(text, truncation=True, max_length=512, stride=128,
16 return_overflowing_tokens=True, padding=True, return_tensors="pt")
17enc.pop("overflow_to_sample_mapping")
18with torch.no_grad():
19 chunk_logits = model(**enc).logits # (m, 2)
20p_chunks = chunk_logits.max(dim=0).values.softmax(-1)
21
22# 3) a long encoder: 4,096 positions; the classification model puts global attention on <s> ([CLS]) by default
23name = "allenai/longformer-base-4096"
24ltok = AutoTokenizer.from_pretrained(name)
25lmodel = AutoModelForSequenceClassification.from_pretrained(name, num_labels=2)
26lbatch = ltok(text, truncation=True, max_length=4096, return_tensors="pt")
27p_long = lmodel(**lbatch).logits.softmax(-1) # fine-tune before trusting it
Head+tail truncation, overlapping chunks with max pooling, and Longformer (Hugging Face; not run here)

Where it's used

Legal and patent

Contract clause types, court-case outcome and patent classification codes: long, structured documents, often multi-label.

Clinical coding

Assigning diagnosis codes to discharge summaries: thousands of tokens, thousands of possible labels, evidence scattered across the note.

Reviews and news

Long-review sentiment (see Sentiment analysis), news topic and hyperpartisan detection.

Check yourself

A 6,000-token document is split into non-overlapping 510-token chunks for mean pooling. One chunk contains the only decisive passage. What weight does that chunk get in the document vector?

Check yourself

Why did Sun et al. (2019) find head 128 + tail 382 better than head-only truncation on IMDb?

Remember

  • ✓BERT-style encoders stop at 512 positions, and full self-attention grows as n².
  • ✓Truncation: one pass, blind to the middle; head 128 + tail 382 is a strong default for reviews.
  • ✓Chunk-and-pool: ⌈n/510⌉\lceil n/510\rceil passes; mean pooling dilutes local evidence by 1/m1/m, max needs the evidence inside one chunk (use overlap).
  • ✓HAN: bi-GRU + attention over words, then over sentences; no length limit, interpretable weights.
  • ✓Longformer/BigBird: sliding window + global [CLS], O(n⋅w)O(n\cdot w), 4,096 tokens in one pass with cross-window context.
← back to Deep Learning roadmap