Definition
Long-context NLP is the set of tasks whose input is far longer than one passage, such as a book, a 100-page report, a code repository or dozens of retrieved documents, together with the methods and benchmarks for them. A model with a long window can accept such inputs. The task-level question is whether it uses all of the input. Liu et al. ("Lost in the Middle: How Language Models Use Long Contexts", 2023) showed that accuracy depends on where the relevant text sits. RULER (Hsieh et al., 2024) showed that a model's effective length is often well below its advertised window.
The architecture side is covered elsewhere. Sparse patterns such as Longformer and BigBird made 4K-token encoders affordable. Context window optimization covers RoPE rescaling, FlashAttention and ring attention, which take decoder LLMs to 128K tokens and beyond. This article is about the tasks, how to evaluate them, and when to use the long window rather than retrieve.
In the BERT era (2018–2020), long documents were truncated to 512 tokens or processed hierarchically. A model encoded each paragraph and then combined the paragraph vectors, or an extractive step chose sentences for a short reader. Both throw away interactions between distant parts of the text, and those interactions are what many long tasks require.
Tasks and benchmarks
| Task | Example datasets | What it needs |
|---|---|---|
| Long-document summarization | arXiv / PubMed (Cohan et al., 2018), GovReport (Huang et al., 2021), BookSum (Kryściński et al., 2021) | coverage of the whole input |
| Long-document QA | NarrativeQA (books and scripts, Kočiský et al., 2018), Qasper (NLP papers, Dasigi et al., 2021), QuALITY (Pang et al., 2022) | finding and combining evidence |
| Multi-document QA | NaturalQuestions with 10–30 retrieved passages (Liu et al.) | picking the one relevant passage |
| Suites | SCROLLS (Shaham et al., 2022), ZeroSCROLLS (2023), LongBench (Bai et al., 2023: 21 datasets, 6 categories, English and Chinese) | realistic tasks, varied lengths |
| Synthetic probes | needle-in-a-haystack (Kamradt, 2023); RULER (13 tasks: retrieval, multi-hop tracing, aggregation, QA) | controlled length and position |
Real tasks show whether a system is useful. Synthetic probes show why it fails, because length, position and the number of distractors can be set exactly. RULER reports an effective length: the longest input at which a model still scores above a fixed threshold. Many models that pass the plain needle test have effective lengths far below their windows.
Measuring position: the U-shaped curve
- the gold document that contains the answer , placed at position among distractors
- the model's answer to question q given the documents in that order
- Liu et al. count an answer correct if it contains a gold answer string
Liu et al. ran this sweep with 10, 20 and 30 Wikipedia passages from NaturalQuestions-Open, and with a synthetic task that retrieves a value for a key from a JSON list of UUID pairs. Accuracy was highest with the gold passage first, slightly lower with it last, and lowest in the middle. This held for open and closed models and for their extended-context variants. In some settings GPT-3.5-Turbo answered worse with the gold passage in the middle than with no documents at all (closed-book). Putting the question both before and after the documents fixed the key–value task but barely changed multi-document QA.
- stylised probability that the reader relies on document (0-based): relevance plus a primacy bump and a recency bump
- accuracy when it relies on a distractor; set below closed-book accuracy, because a plausible wrong passage misleads
This toy reader is not fitted to any model. It shows the mechanism: when a position bias competes with relevance, a gold passage in the middle gets less weight, and more distractors dilute it further.
The retrieval line shows a precision–recall trade-off in . At recall is only 0.40 and accuracy is 0.570. is best at 0.625 (recall 0.73). Beyond that, each added passage brings more distractors than recall gains. A noisier retriever () drops recall@5 to 0.575 and accuracy to 0.519, barely above closed-book. This is the long-context side of Retrieval-augmented generation (RAG): the reader's position bias and the retriever's recall together set the accuracy.
Long context or retrieval?
- parameters (about 8.0 × 10⁹ for Llama 3.1 8B)
- layers (32) and model width (4096); the T² term is QKᵀ plus AV under a causal mask
- 8 KV heads × 128 dims with grouped-query attention; bytes in bf16, so 128 KiB per token
| Route | Prompt tokens | Prefill FLOPs | KV cache |
|---|---|---|---|
| Full context | 100,000 | 12.2 GiB | |
| Top-5 of 512-token chunks | 2,560 | 0.31 GiB | |
| Ratio | 39× | ≈ 99× | 39× |
At 100K tokens the attention term is larger than the weight term, so cost grows faster than the prompt. Caching the prefilled document helps when many questions are asked about it. Retrieval has its own costs: chunking, an embedding index, and a recall ceiling. Xu et al. ("Retrieval meets Long Context Large Language Models", 2023) found that a 4K-context model with retrieval matched a 16K-context model extended by position interpolation on long-document QA and summarization, and that retrieval also improved the long-context models.
Use when
- +Long context: no index, no chunking errors; handles aggregation, cross-references and whole-document summaries
- +Retrieval: about 40–100× less compute per question here; scales to corpora far beyond any window; sources are easy to cite
- +Hybrid: retrieve generously, rerank, and let a long-context reader handle 10–30K tokens
Avoid when
- −Long context: quadratic attention cost, large KV cache, and weakest use of the middle of the input
- −Retrieval: a missed passage cannot be recovered; poor for questions about the whole document ("how many times…", "summarise")
- −Both are hard to evaluate: answers may come from parametric memory, so compare with a closed-book run
1import random23def build_prompt(question, gold, distractors, pos):4 docs = distractors[:pos] + [gold] + distractors[pos:]5 body = "\n\n".join(f"Document [{i + 1}] {d}" for i, d in enumerate(docs))6 return f"Answer the question using the documents.\n\n{body}\n\nQuestion: {question}\nAnswer:"78def position_sweep(ask, examples, n_docs=20, positions=(0, 4, 9, 14, 19), seed=0):9 """ask(prompt) -> str. examples: dicts with question, answers, gold, distractors."""10 rng = random.Random(seed)11 acc = {}12 for pos in positions:13 hits = 014 for ex in examples:15 dis = rng.sample(ex["distractors"], n_docs - 1)16 out = ask(build_prompt(ex["question"], ex["gold"], dis, pos)).lower()17 hits += any(a.lower() in out for a in ex["answers"]) # Liu et al. score: answer string contained18 acc[pos + 1] = hits / len(examples)19 return acc2021def toy_ask(prompt): # stand-in reader that only "sees" the first 3 and last 2 documents22 docs = prompt.split("\n\n")[1:-1]23 seen = docs[:3] + docs[-2:]24 return "Paris" if any("capital of France is Paris" in d for d in seen) else "unknown"2526ex = [{"question": "What is the capital of France?", "answers": ["Paris"],27 "gold": "The capital of France is Paris.",28 "distractors": [f"Filler passage number {i} about something else." for i in range(40)]}]29print(position_sweep(toy_ask, ex)) # {1: 1.0, 5: 0.0, 10: 0.0, 15: 0.0, 20: 1.0}Check yourself
In the worked example, why does the full-context route cost about 99× more prefill compute than top-5 retrieval, although its prompt is only 39× longer?
Check yourself
A RAG system retrieves 20 passages and passes them to the reader in random order. What change does the lost-in-the-middle result suggest first?
Remember
- ✓Long-context tasks: long-document summarization and QA, multi-document QA, code; collected in SCROLLS, ZeroSCROLLS and LongBench, and probed by needle tests and RULER.
- ✓Sweep the gold document's position : is U-shaped, best first, worst in the middle, and the dip deepens with more documents.
- ✓Effective length (RULER) is the longest input where accuracy stays above a threshold; it is often far below the advertised window.
- ✓Prefill cost : for Llama 3.1 8B at 100K tokens, attention exceeds the weight FLOPs, and the KV cache is 12.2 GiB.
- ✓Chunk-and-retrieve is about 99× cheaper per question in the worked example, and retrieval helps even long-context models (Xu et al.). It fails on aggregation and on missed passages.
- ✓Put the most relevant text at the edges, rerank, and always compare with a closed-book baseline.