# Long-context NLP

> Tasks over 10K–1M-token inputs: books, reports, codebases, many retrieved passages. Models use the middle of long inputs worst, effective length falls short of the window, and retrieval often matches long context at far lower cost.

Source: https://www.mlfoundry.in/learn/long-context-nlp  
Section: Deep Learning › Deep Learning for NLP  
Level: advanced · 8 min read · updated 2026-10-15

**Long-context NLP:** is the set of tasks whose input is far longer than one passage, such as a book, a 100-page report, a code repository or dozens of retrieved documents, together with the methods and benchmarks for them. A model with a long window can accept such inputs. The task-level question is whether it *uses* all of the input. Liu et al. ("Lost in the Middle: How Language Models Use Long Contexts", 2023) showed that accuracy depends on *where* the relevant text sits. RULER (Hsieh et al., 2024) showed that a model's effective length is often well below its advertised window.

The architecture side is covered elsewhere. Sparse patterns such as **Longformer** and **BigBird** made 4K-token encoders affordable. **Context window optimization** covers RoPE rescaling, FlashAttention and ring attention, which take decoder LLMs to 128K tokens and beyond. This article is about the tasks, how to evaluate them, and when to use the long window rather than retrieve.

In the BERT era (2018–2020), long documents were truncated to 512 tokens or processed hierarchically. A model encoded each paragraph and then combined the paragraph vectors, or an extractive step chose sentences for a short reader. Both throw away interactions between distant parts of the text, and those interactions are what many long tasks require.

## Tasks and benchmarks

Long-input tasks and the benchmarks that collect them

| Task | Example datasets | What it needs |
| --- | --- | --- |
| Long-document summarization | arXiv / PubMed (Cohan et al., 2018), GovReport (Huang et al., 2021), BookSum (Kryściński et al., 2021) | coverage of the whole input |
| Long-document QA | NarrativeQA (books and scripts, Kočiský et al., 2018), Qasper (NLP papers, Dasigi et al., 2021), QuALITY (Pang et al., 2022) | finding and combining evidence |
| Multi-document QA | NaturalQuestions with 10–30 retrieved passages (Liu et al.) | picking the one relevant passage |
| Suites | SCROLLS (Shaham et al., 2022), ZeroSCROLLS (2023), LongBench (Bai et al., 2023: 21 datasets, 6 categories, English and Chinese) | realistic tasks, varied lengths |
| Synthetic probes | needle-in-a-haystack (Kamradt, 2023); RULER (13 tasks: retrieval, multi-hop tracing, aggregation, QA) | controlled length and position |

Real tasks show whether a system is useful. Synthetic probes show *why* it fails, because length, position and the number of distractors can be set exactly. RULER reports an **effective length**: the longest input at which a model still scores above a fixed threshold. Many models that pass the plain needle test have effective lengths far below their windows.

> Figure: Three ways to answer a question about a 100,000-token document. A: send it all to a 128K-window model. That is the simplest route, but it prefills 100K tokens, needs a 12.2 GiB KV cache for Llama 3.1 8B, and depends on the model using the middle. B: split into 196 chunks of 512 tokens, retrieve the top 5 and read 2,560 tokens. That is cheap per question, but a missed chunk is lost and aggregation questions fail. C: map-reduce over 13 chunks of 8,192 tokens with 14 calls. It suits whole-document summaries, but links across chunks break.

## Measuring position: the U-shaped curve

$$
\mathrm{Acc}(k)=\frac1{|\mathcal D|}\sum_{(q,\,d^\star,\,y)\in\mathcal D}\mathbb 1\Big[\,y\in f\big(q;\;d_1,\dots,d_{k-1},\,d^\star,\,d_{k},\dots,d_{n-1}\big)\Big]
$$

- $d^\star$: the gold document that contains the answer $y$, placed at position $k$ among $n-1$ distractors
- $f(q;\cdot)$: the model's answer to question q given the documents in that order
- $y\in f$: Liu et al. count an answer correct if it contains a gold answer string

Liu et al. ran this sweep with 10, 20 and 30 Wikipedia passages from NaturalQuestions-Open, and with a synthetic task that retrieves a value for a key from a JSON list of UUID pairs. Accuracy was highest with the gold passage first, slightly lower with it last, and lowest in the middle. This held for open and closed models and for their extended-context variants. In some settings GPT-3.5-Turbo answered worse with the gold passage in the middle than with no documents at all (closed-book). Putting the question both before and after the documents fixed the key–value task but barely changed multi-document QA.

$$
\pi_k=\frac{\exp\!\big(\mathrm{rel}_k+\beta_p e^{-k/\tau}+\beta_r e^{-(n-1-k)/\tau}\big)}{\sum_{j}\exp\!\big(\mathrm{rel}_j+\beta_p e^{-j/\tau}+\beta_r e^{-(n-1-j)/\tau}\big)},\qquad P(\text{correct})=\pi_{k^\star}A_{\text{read}}+(1-\pi_{k^\star})A_{\text{misled}}
$$

- $\pi_k$: stylised probability that the reader relies on document $k$ (0-based): relevance plus a primacy bump $\beta_p$ and a recency bump $\beta_r$
- $A_{\text{misled}}$: accuracy when it relies on a distractor; set below closed-book accuracy, because a plausible wrong passage misleads

This toy reader is not fitted to any model. It shows the mechanism: when a position bias competes with relevance, a gold passage in the middle gets less weight, and more distractors dilute it further.

> Figure: The stylised reader with 20 documents (βp = 1.5, βr = 1.0, averaged over 200 draws of distractor relevance). Accuracy is 0.618 with the gold document first, falls to 0.446 at position 11 and recovers to 0.544 at position 20. Fifteen of the 20 positions score below the closed-book 0.50. Retrieving the top 5 and putting them in rank order scores 0.617 with recall@5 = 0.850. Even top-20, the same documents re-sorted by score, reaches 0.546 against a mean of 0.482 over positions. With 30 documents the dip deepens to 0.419. Set both biases to 0 and the curve is flat at 0.486.

The retrieval line shows a precision–recall trade-off in $K$. At $K=1$ recall is only 0.40 and accuracy is 0.570. $K=3$ is best at 0.625 (recall 0.73). Beyond that, each added passage brings more distractors than recall gains. A noisier retriever ($\sigma_r=2$) drops recall@5 to 0.575 and accuracy to 0.519, barely above closed-book. This is the long-context side of **Retrieval-augmented generation (RAG)**: the reader's position bias and the retriever's recall together set the accuracy.

## Long context or retrieval?

$$
\text{prefill FLOPs}\approx\underbrace{2N\,T}_{\text{weights}}+\underbrace{2\,L\,d\,T^2}_{\text{causal attention}},\qquad \text{KV bytes}=2\,L\,n_{kv}\,d_h\,b\cdot T
$$

- $N$: parameters (about 8.0 × 10⁹ for Llama 3.1 8B)
- $L,\ d$: layers (32) and model width (4096); the T² term is QKᵀ plus AV under a causal mask
- $n_{kv}d_h$: 8 KV heads × 128 dims with grouped-query attention; $b=2$ bytes in bf16, so 128 KiB per token

Worked example: one question over a 100,000-token document with Llama 3.1 8B (computed in Python)

| Route | Prompt tokens | Prefill FLOPs | KV cache |
| --- | --- | --- | --- |
| Full context | 100,000 | $1.6\times10^{15}+2.6\times10^{15}=4.2\times10^{15}$ | 12.2 GiB |
| Top-5 of 512-token chunks | 2,560 | $4.1\times10^{13}+1.7\times10^{12}=4.3\times10^{13}$ | 0.31 GiB |
| Ratio | 39× | ≈ 99× | 39× |

At 100K tokens the attention term is larger than the weight term, so cost grows faster than the prompt. Caching the prefilled document helps when many questions are asked about it. Retrieval has its own costs: chunking, an embedding index, and a recall ceiling. Xu et al. ("Retrieval meets Long Context Large Language Models", 2023) found that a 4K-context model with retrieval matched a 16K-context model extended by position interpolation on long-document QA and summarization, and that retrieval also improved the long-context models.

**Strengths**

- Long context: no index, no chunking errors; handles aggregation, cross-references and whole-document summaries
- Retrieval: about 40–100× less compute per question here; scales to corpora far beyond any window; sources are easy to cite
- Hybrid: retrieve generously, rerank, and let a long-context reader handle 10–30K tokens

**Limitations**

- Long context: quadratic attention cost, large KV cache, and weakest use of the middle of the input
- Retrieval: a missed passage cannot be recovered; poor for questions about the whole document ("how many times…", "summarise")
- Both are hard to evaluate: answers may come from parametric memory, so compare with a closed-book run

```python
import random

def build_prompt(question, gold, distractors, pos):
    docs = distractors[:pos] + [gold] + distractors[pos:]
    body = "\n\n".join(f"Document [{i + 1}] {d}" for i, d in enumerate(docs))
    return f"Answer the question using the documents.\n\n{body}\n\nQuestion: {question}\nAnswer:"

def position_sweep(ask, examples, n_docs=20, positions=(0, 4, 9, 14, 19), seed=0):
    """ask(prompt) -> str. examples: dicts with question, answers, gold, distractors."""
    rng = random.Random(seed)
    acc = {}
    for pos in positions:
        hits = 0
        for ex in examples:
            dis = rng.sample(ex["distractors"], n_docs - 1)
            out = ask(build_prompt(ex["question"], ex["gold"], dis, pos)).lower()
            hits += any(a.lower() in out for a in ex["answers"])     # Liu et al. score: answer string contained
        acc[pos + 1] = hits / len(examples)
    return acc

def toy_ask(prompt):                     # stand-in reader that only "sees" the first 3 and last 2 documents
    docs = prompt.split("\n\n")[1:-1]
    seen = docs[:3] + docs[-2:]
    return "Paris" if any("capital of France is Paris" in d for d in seen) else "unknown"

ex = [{"question": "What is the capital of France?", "answers": ["Paris"],
       "gold": "The capital of France is Paris.",
       "distractors": [f"Filler passage number {i} about something else." for i in range(40)]}]
print(position_sweep(toy_ask, ex))      # {1: 1.0, 5: 0.0, 10: 0.0, 15: 0.0, 20: 1.0}
```

A lost-in-the-middle position sweep (run with a stand-in reader that only sees the first 3 and last 2 documents; replace toy_ask with a real model call)

> **The window is not the effective length:** A model that accepts 128K tokens and finds a single needle anywhere in them can still fail multi-hop tracing or counting at 32K. Report accuracy against length *and* position on tasks like your own. Include a closed-book baseline: on Wikipedia-based QA the model may know the answer already, which hides failures to read.

> **Order is a free parameter:** If position matters, put the most relevant material where the model looks: first, or last next to the question. Rerank retrieved passages instead of padding the prompt with more. In the figure, re-sorting the same 20 documents by retriever score raises accuracy from a mean of 0.482 to 0.546, and keeping only the top 3 reaches 0.625.

**Question:** In the worked example, why does the full-context route cost about 99× more prefill compute than top-5 retrieval, although its prompt is only 39× longer?

- A. The causal-attention term grows as T², and at 100K tokens it is larger than the weight term
- B. The KV cache must be recomputed for every layer twice
- C. Retrieval uses a smaller model
- D. Long prompts are tokenized less efficiently

**Answer:** A. The causal-attention term grows as T², and at 100K tokens it is larger than the weight term — Weight FLOPs scale as $2NT$ (39× more), but attention scales as $2LdT^2$. At $T=100{,}000$ the attention term is $2.6\times10^{15}$, larger than the $1.6\times10^{15}$ weight term. At $T=2{,}560$ it is only $1.7\times10^{12}$. The total ratio is $4.2\times10^{15}/4.3\times10^{13}\approx99$.

**Question:** A RAG system retrieves 20 passages and passes them to the reader in random order. What change does the lost-in-the-middle result suggest first?

- A. Sort passages by relevance so the best ones sit at the start (or next to the question), and pass fewer of them
- B. Retrieve 100 passages to raise recall
- C. Switch to a model with a larger window
- D. Shuffle the passages differently for each question

**Answer:** A. Sort passages by relevance so the best ones sit at the start (or next to the question), and pass fewer of them — Accuracy is highest when the relevant passage is at the edges and falls as distractors are added. Reranking places it where the model uses it, and fewer passages mean less dilution. Raising recall with more passages, or a larger window, pushes the answer further into the middle.

**Key points**

- Long-context tasks: long-document summarization and QA, multi-document QA, code; collected in SCROLLS, ZeroSCROLLS and LongBench, and probed by needle tests and RULER.
- Sweep the gold document's position $k$: $\mathrm{Acc}(k)$ is U-shaped, best first, worst in the middle, and the dip deepens with more documents.
- Effective length (RULER) is the longest input where accuracy stays above a threshold; it is often far below the advertised window.
- Prefill cost $\approx2NT+2LdT^2$: for Llama 3.1 8B at 100K tokens, attention exceeds the weight FLOPs, and the KV cache is 12.2 GiB.
- Chunk-and-retrieve is about 99× cheaper per question in the worked example, and retrieval helps even long-context models (Xu et al.). It fails on aggregation and on missed passages.
- Put the most relevant text at the edges, rerank, and always compare with a closed-book baseline.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
