Long-context NLP

Tasks over 10K–1M-token inputs: books, reports, codebases, many retrieved passages. Models use the middle of long inputs worst, effective length falls short of the window, and retrieval often matches long context at far lower cost.

8 min read803 wordsadvanced0 viewsupdated 2026-10-15

Definition

Long-context NLP is the set of tasks whose input is far longer than one passage, such as a book, a 100-page report, a code repository or dozens of retrieved documents, together with the methods and benchmarks for them. A model with a long window can accept such inputs. The task-level question is whether it uses all of the input. Liu et al. ("Lost in the Middle: How Language Models Use Long Contexts", 2023) showed that accuracy depends on where the relevant text sits. RULER (Hsieh et al., 2024) showed that a model's effective length is often well below its advertised window.

The architecture side is covered elsewhere. Sparse patterns such as Longformer and BigBird made 4K-token encoders affordable. Context window optimization covers RoPE rescaling, FlashAttention and ring attention, which take decoder LLMs to 128K tokens and beyond. This article is about the tasks, how to evaluate them, and when to use the long window rather than retrieve.

In the BERT era (2018–2020), long documents were truncated to 512 tokens or processed hierarchically. A model encoded each paragraph and then combined the paragraph vectors, or an extractive step chose sentences for a short reader. Both throw away interactions between distant parts of the text, and those interactions are what many long tasks require.

Tasks and benchmarks

TaskExample datasetsWhat it needs
Long-document summarizationarXiv / PubMed (Cohan et al., 2018), GovReport (Huang et al., 2021), BookSum (Kryściński et al., 2021)coverage of the whole input
Long-document QANarrativeQA (books and scripts, Kočiský et al., 2018), Qasper (NLP papers, Dasigi et al., 2021), QuALITY (Pang et al., 2022)finding and combining evidence
Multi-document QANaturalQuestions with 10–30 retrieved passages (Liu et al.)picking the one relevant passage
SuitesSCROLLS (Shaham et al., 2022), ZeroSCROLLS (2023), LongBench (Bai et al., 2023: 21 datasets, 6 categories, English and Chinese)realistic tasks, varied lengths
Synthetic probesneedle-in-a-haystack (Kamradt, 2023); RULER (13 tasks: retrieval, multi-hop tracing, aggregation, QA)controlled length and position
Long-input tasks and the benchmarks that collect them

Real tasks show whether a system is useful. Synthetic probes show why it fails, because length, position and the number of distractors can be set exactly. RULER reports an effective length: the longest input at which a model still scores above a fixed threshold. Many models that pass the plain needle test have effective lengths far below their windows.

one question about a 100,000-token document: three pipelinesanswer inthe middledocumentA · full long context100,000 tokens + questionone prompt, 128K windowlong-context LLMreads everythinganswerprefill 100K tokens · KV cache 12.2 GiB (Llama 3.1 8B, bf16)risk: the middle of the prompt is used least (lost in the middle)B · chunk and retrieve196 chunks× 512 tokensembed, scorekeep top 5reader prompt2,560 tokensanswerprefill 2.6K tokens per question · index built oncerisk: a chunk the retriever misses is gone; weak at aggregationC · map-reduce13 chunks× 8,192 tokenssummarise oranswer eachcombine the 13partial outputsanswer14 model calls; the map calls run in parallelrisk: details lost in partial outputs; cross-chunk links brokenA is simplest when it fits; B is cheapest per question; C suits whole-document summaries
fig.Three ways to answer a question about a 100,000-token document. A: send it all to a 128K-window model. That is the simplest route, but it prefills 100K tokens, needs a 12.2 GiB KV cache for Llama 3.1 8B, and depends on the model using the middle. B: split into 196 chunks of 512 tokens, retrieve the top 5 and read 2,560 tokens. That is cheap per question, but a missed chunk is lost and aggregation questions fail. C: map-reduce over 13 chunks of 8,192 tokens with 14 calls. It suits whole-document summaries, but links across chunks break.

Measuring position: the U-shaped curve

Acc(k)=1∣D∣∑(q, d⋆, y)∈D1[ y∈f(q;  d1,…,dk−1, d⋆, dk,…,dn−1)]\mathrm{Acc}(k)=\frac1{|\mathcal D|}\sum_{(q,\,d^\star,\,y)\in\mathcal D}\mathbb 1\Big[\,y\in f\big(q;\;d_1,\dots,d_{k-1},\,d^\star,\,d_{k},\dots,d_{n-1}\big)\Big]
d⋆d^\star
the gold document that contains the answer yy, placed at position kk among n−1n-1 distractors
f(q;⋅)f(q;\cdot)
the model's answer to question q given the documents in that order
y∈fy\in f
Liu et al. count an answer correct if it contains a gold answer string

Liu et al. ran this sweep with 10, 20 and 30 Wikipedia passages from NaturalQuestions-Open, and with a synthetic task that retrieves a value for a key from a JSON list of UUID pairs. Accuracy was highest with the gold passage first, slightly lower with it last, and lowest in the middle. This held for open and closed models and for their extended-context variants. In some settings GPT-3.5-Turbo answered worse with the gold passage in the middle than with no documents at all (closed-book). Putting the question both before and after the documents fixed the key–value task but barely changed multi-document QA.

πk=exp⁡ ⁣(relk+βpe−k/τ+βre−(n−1−k)/τ)∑jexp⁡ ⁣(relj+βpe−j/τ+βre−(n−1−j)/τ),P(correct)=πk⋆Aread+(1−πk⋆)Amisled\pi_k=\frac{\exp\!\big(\mathrm{rel}_k+\beta_p e^{-k/\tau}+\beta_r e^{-(n-1-k)/\tau}\big)}{\sum_{j}\exp\!\big(\mathrm{rel}_j+\beta_p e^{-j/\tau}+\beta_r e^{-(n-1-j)/\tau}\big)},\qquad P(\text{correct})=\pi_{k^\star}A_{\text{read}}+(1-\pi_{k^\star})A_{\text{misled}}
πk\pi_k
stylised probability that the reader relies on document kk (0-based): relevance plus a primacy bump βp\beta_p and a recency bump βr\beta_r
AmisledA_{\text{misled}}
accuracy when it relies on a distractor; set below closed-book accuracy, because a plausible wrong passage misleads

This toy reader is not fitted to any model. It shows the mechanism: when a position bias competes with relevance, a gold passage in the middle gets less weight, and more distractors dilute it further.

20 documents, 1 holds the answer · accuracy vs its position (stylised reader)0.30.40.50.60.70.811020position of the gold document in the promptaccuracyclosed-book 0.50retrieve top-5, rank order: 0.617the long prompt: shade = position bias, dark = gold documentquestion at the end · reader uses doc j with p ∝ exp(relevance + bias)gold at position 100.447 p(reads gold doc)0.176first position0.618lowest (pos 11)0.446last position0.544mean over positions0.482below closed-book15 of 20retrieve top-50.617 recall@50.850P(correct) = p(gold)·0.9 + (1 − p(gold))·0.35 · misled answers fall below closed-bookretrieval: score = relevance + N(0, σᵣ²), keep the top 5, put them first-to-last by scoreaveraged over 200 draws of distractor relevance; a stylised model, not a fitted LLM
documents
fig.The stylised reader with 20 documents (βp = 1.5, βr = 1.0, averaged over 200 draws of distractor relevance). Accuracy is 0.618 with the gold document first, falls to 0.446 at position 11 and recovers to 0.544 at position 20. Fifteen of the 20 positions score below the closed-book 0.50. Retrieving the top 5 and putting them in rank order scores 0.617 with recall@5 = 0.850. Even top-20, the same documents re-sorted by score, reaches 0.546 against a mean of 0.482 over positions. With 30 documents the dip deepens to 0.419. Set both biases to 0 and the curve is flat at 0.486.

The retrieval line shows a precision–recall trade-off in KK. At K=1K=1 recall is only 0.40 and accuracy is 0.570. K=3K=3 is best at 0.625 (recall 0.73). Beyond that, each added passage brings more distractors than recall gains. A noisier retriever (σr=2\sigma_r=2) drops recall@5 to 0.575 and accuracy to 0.519, barely above closed-book. This is the long-context side of Retrieval-augmented generation (RAG): the reader's position bias and the retriever's recall together set the accuracy.

Long context or retrieval?

prefill FLOPs≈2N T⏟weights+2 L d T2⏟causal attention,KV bytes=2 L nkv dh b⋅T\text{prefill FLOPs}\approx\underbrace{2N\,T}_{\text{weights}}+\underbrace{2\,L\,d\,T^2}_{\text{causal attention}},\qquad \text{KV bytes}=2\,L\,n_{kv}\,d_h\,b\cdot T
NN
parameters (about 8.0 × 10⁹ for Llama 3.1 8B)
L, dL,\ d
layers (32) and model width (4096); the T² term is QKᵀ plus AV under a causal mask
nkvdhn_{kv}d_h
8 KV heads × 128 dims with grouped-query attention; b=2b=2 bytes in bf16, so 128 KiB per token
RoutePrompt tokensPrefill FLOPsKV cache
Full context100,0001.6×1015+2.6×1015=4.2×10151.6\times10^{15}+2.6\times10^{15}=4.2\times10^{15}12.2 GiB
Top-5 of 512-token chunks2,5604.1×1013+1.7×1012=4.3×10134.1\times10^{13}+1.7\times10^{12}=4.3\times10^{13}0.31 GiB
Ratio39×≈ 99×39×
Worked example: one question over a 100,000-token document with Llama 3.1 8B (computed in Python)

At 100K tokens the attention term is larger than the weight term, so cost grows faster than the prompt. Caching the prefilled document helps when many questions are asked about it. Retrieval has its own costs: chunking, an embedding index, and a recall ceiling. Xu et al. ("Retrieval meets Long Context Large Language Models", 2023) found that a 4K-context model with retrieval matched a 16K-context model extended by position interpolation on long-document QA and summarization, and that retrieval also improved the long-context models.

Use when

  • +Long context: no index, no chunking errors; handles aggregation, cross-references and whole-document summaries
  • +Retrieval: about 40–100× less compute per question here; scales to corpora far beyond any window; sources are easy to cite
  • +Hybrid: retrieve generously, rerank, and let a long-context reader handle 10–30K tokens

Avoid when

  • −Long context: quadratic attention cost, large KV cache, and weakest use of the middle of the input
  • −Retrieval: a missed passage cannot be recovered; poor for questions about the whole document ("how many times…", "summarise")
  • −Both are hard to evaluate: answers may come from parametric memory, so compare with a closed-book run
python
1import random
2
3def build_prompt(question, gold, distractors, pos):
4 docs = distractors[:pos] + [gold] + distractors[pos:]
5 body = "\n\n".join(f"Document [{i + 1}] {d}" for i, d in enumerate(docs))
6 return f"Answer the question using the documents.\n\n{body}\n\nQuestion: {question}\nAnswer:"
7
8def position_sweep(ask, examples, n_docs=20, positions=(0, 4, 9, 14, 19), seed=0):
9 """ask(prompt) -> str. examples: dicts with question, answers, gold, distractors."""
10 rng = random.Random(seed)
11 acc = {}
12 for pos in positions:
13 hits = 0
14 for ex in examples:
15 dis = rng.sample(ex["distractors"], n_docs - 1)
16 out = ask(build_prompt(ex["question"], ex["gold"], dis, pos)).lower()
17 hits += any(a.lower() in out for a in ex["answers"]) # Liu et al. score: answer string contained
18 acc[pos + 1] = hits / len(examples)
19 return acc
20
21def toy_ask(prompt): # stand-in reader that only "sees" the first 3 and last 2 documents
22 docs = prompt.split("\n\n")[1:-1]
23 seen = docs[:3] + docs[-2:]
24 return "Paris" if any("capital of France is Paris" in d for d in seen) else "unknown"
25
26ex = [{"question": "What is the capital of France?", "answers": ["Paris"],
27 "gold": "The capital of France is Paris.",
28 "distractors": [f"Filler passage number {i} about something else." for i in range(40)]}]
29print(position_sweep(toy_ask, ex)) # {1: 1.0, 5: 0.0, 10: 0.0, 15: 0.0, 20: 1.0}
A lost-in-the-middle position sweep (run with a stand-in reader that only sees the first 3 and last 2 documents; replace toy_ask with a real model call)

Check yourself

In the worked example, why does the full-context route cost about 99× more prefill compute than top-5 retrieval, although its prompt is only 39× longer?

Check yourself

A RAG system retrieves 20 passages and passes them to the reader in random order. What change does the lost-in-the-middle result suggest first?

Remember

  • ✓Long-context tasks: long-document summarization and QA, multi-document QA, code; collected in SCROLLS, ZeroSCROLLS and LongBench, and probed by needle tests and RULER.
  • ✓Sweep the gold document's position kk: Acc(k)\mathrm{Acc}(k) is U-shaped, best first, worst in the middle, and the dip deepens with more documents.
  • ✓Effective length (RULER) is the longest input where accuracy stays above a threshold; it is often far below the advertised window.
  • ✓Prefill cost ≈2NT+2LdT2\approx2NT+2LdT^2: for Llama 3.1 8B at 100K tokens, attention exceeds the weight FLOPs, and the KV cache is 12.2 GiB.
  • ✓Chunk-and-retrieve is about 99× cheaper per question in the worked example, and retrieval helps even long-context models (Xu et al.). It fails on aggregation and on missed passages.
  • ✓Put the most relevant text at the edges, rerank, and always compare with a closed-book baseline.
← back to Deep Learning roadmap