Definition
Context window optimization is the set of methods that let a language model trained on sequences of length work on inputs of length . It covers three separate problems. Positions: the position encoding must stay inside what the model saw in training, as in position interpolation (Chen, Wong, Chen and Tian, 2023), NTK-aware scaling (2023), YaRN (Peng, Quesnelle, Fan and Shippole, 2023) and LongRoPE (Ding et al., 2024). Cost: attention must fit in memory and time, through FlashAttention, sliding-window or sparse attention, and ring attention across devices. Use: the model must be trained on long data and shown to retrieve and reason over it, which is measured with needle-in-a-haystack tests and "Lost in the Middle" (Liu et al., 2023).
GPT-3 had 2048 tokens of context and Llama 2 had 4096. Llama 3.1, Mistral's later models and most frontier models now accept 128K tokens or more. Almost none of that growth came from pre-training at full length from scratch, because attention cost grows as and most documents are short. Instead, models are pre-trained at 4K–8K tokens and then extended: the position encoding is adjusted, and a short phase of continued pre-training on long documents follows. The fixes below are therefore layered. Each solves one problem and leaves the others.
Why RoPE models break past their training length
- rotation per token of the -th 2-D pair of a head (see Rotary positional embeddings (RoPE)); in Llama 2 and in Llama 3
- wavelength: tokens per full turn, from for pair 0 to for pair 63 at ,
- turns completed inside the trained context . Pairs with have seen every angle. Pairs with have only seen the arc
High-frequency pairs wrap around many times within , so at any longer length they show only angles the model already knows. Low-frequency pairs never complete a turn. At position they reach angles up to , which the attention weights were never trained on. For Llama 2 (, , ) that is pairs 46–63, 18 of the 64. These slow pairs carry the coarse "how far away" signal, and when they leave the trained range, attention scores become erratic and perplexity blows up within a few hundred tokens past .
Rescaling the frequencies
- position interpolation: equivalent to feeding position . Nothing extrapolates, but neighbouring tokens become times closer in every pair
- NTK-aware scaling ("bloc97", Reddit, 2023): a larger base leaves pair 0 untouched and divides the slowest pair by exactly ; pairs in between are slowed geometrically
- YaRN's ramp on turns per context: fully interpolate pairs with , keep pairs with , blend linearly between
- YaRN also multiplies the attention logits by this factor, 1.459 at , to undo the flatter softmax at long range
Chen et al. proved that interpolated attention scores are far better behaved than extrapolated ones. They bounded the interpolated case about 600 times tighter, and extended LLaMA 7B–65B from 2048 to 32,768 tokens with about 1000 fine-tuning steps. Plain fine-tuning at the longer length adapted far more slowly. PI's cost is resolution. The fastest pair, which tells adjacent tokens apart, now turns only radian per token. NTK-aware scaling keeps the high frequencies and stretches mostly the low ones, and it works reasonably even with no fine-tuning, which made it popular in inference tools. Its "dynamic" variant raises only as the current sequence grows past . YaRN makes the split explicit by wavelength and adds the temperature. Peng et al. extended Llama 2 7B and 13B to 64K and 128K context while using about 10 times fewer tokens and 2.5 times fewer steps than earlier methods. Llama 3.1's released configuration uses a similar wavelength-based split with factor 8 from 8192 tokens.
Two other routes avoid the problem. Raising the base before long training, as Code Llama did (base , trained on 16K sequences, stable up to about 100K tokens), slows every pair so that fewer of them are out of range at the target length. Meta's Llama 2 Long (Xiong et al., 2023) did the same with base 500,000 during 400B tokens of continued pre-training. LongRoPE (Ding et al., 2024) searches for a separate rescale factor per dimension with an evolutionary search, extends in stages to 2048K tokens, and readjusts the factors so that short-context quality recovers. Phi-3's 128K models use it. ALiBi (Press, Smith and Lewis, ICLR 2022) has no rotation at all: it adds to each logit, so a longer input only adds larger penalties, and models trained on 1024 tokens ran at 2048 and beyond. Unlimited extrapolation does not mean the model uses distant tokens, though. ALiBi's penalties make far-away tokens nearly invisible.
Context extensions in published models
- Position interpolation
- LLaMA 7B–65B, 2048 → 32,768 tokens () with about 1000 fine-tuning steps (Chen et al., 2023)
- YaRN
- Llama 2 7B/13B to 64K () and 128K (); , ; logits
- Code Llama
- RoPE base raised from to , fine-tuned on 16,384-token sequences, usable to about 100K tokens
- Llama 2 Long
- base 500,000, continued pre-training on 400B tokens of long sequences (Xiong et al., 2023)
- Llama 3.1
- base 500,000; 8K → 128K in six stages of continued pre-training using about 800B tokens
- Mistral 7B
- sliding window in each of 32 layers, so information can travel about tokens through the stack
- Gemma 2
- alternates local sliding-window layers (4096) with global layers (8192)
- LongRoPE
- searched non-uniform per-dimension factors, extension to 2048K tokens; used for Phi-3's 128K variants
Making long attention affordable
- and , each ; a causal mask halves the useful work
- Llama 2 7B at : 2 GiB per head and 64 GiB per layer in bf16, which is why the matrix must never exist in full
- linear in , 16 GiB for one 32K sequence of Llama 2 7B (see KV cache)
FlashAttention (Dao et al., 2022) computes exact attention tile by tile in on-chip SRAM with an online softmax, so memory is and the score matrix is never written to HBM (see FlashAttention). It removes the memory wall but not the FLOPs. At 32K tokens attention is about 40% of Llama 2 7B's training FLOPs per token. Sliding-window attention caps each query at the last keys, giving work and a cache bounded by . Stacking layers lets information travel tokens. Mistral 7B uses , and Gemma 2 interleaves local and global layers so some layers still see everything. Fixed sparse patterns (Child et al., 2019; Longformer; BigBird) mix local windows, strides and global tokens (see Sparse attention). All of these are approximations that can miss a needle outside the pattern, which is why local/global designs keep some full-attention layers.
- devices in the ring; activation memory per device scales as T/P
- the online-softmax merge of FlashAttention, so partial results combine exactly
- each step passes numbers per layer to the next device, overlapped with the block computation
Ring attention (Liu, Zaharia and Abbeel, 2023) is sequence parallelism for attention. When one sequence is too long for one device's activations, split it into blocks. Each device keeps its query block and streams the key/value blocks around a ring, computing blockwise attention while the next block is in flight. The maximum context grows linearly with the number of devices. Training frameworks call this context parallelism (see Distributed LLM training). Causal masking makes the naive split unbalanced, because the device holding the first block has almost nothing to attend to. Common implementations therefore assign each device a block from the start and one from the end.
Data: what long-context training needs
Changing the frequencies only stops the model from breaking. To use a long context, the model needs training examples where distant tokens matter. Natural long documents are rare and skewed toward books, code repositories and papers. Fu et al. ("Data Engineering for Scaling Language Models to 128K Context", 2024) found that a few billion tokens of continued pre-training are enough. The important choice was to upsample long sequences within each domain and keep the domain mix the same. Upsampling long documents regardless of source shifts the mix toward books and code, and it hurt. Xiong et al. found that data quality mattered more than the sheer amount of long text. Llama 3.1 grew its context in six stages and moved to the next stage only after short-context scores had recovered and needle tests passed at the current length. Recipes also mix in synthetic long tasks, such as retrieval over concatenated documents or long-document QA, and keep a share of short data so that short-context quality does not regress.
Evaluation: does it use the context?
Needle in a haystack (Greg Kamradt, 2023) hides one out-of-place sentence ("The best thing to do in San Francisco is eat a sandwich and sit in Dolores Park on a sunny day.") at a chosen depth in filler text of a chosen length, then asks for it back. The result is a depth × length grid of pass/fail cells. It is cheap and catches positional breakage at once, but a single verbatim match is easy. RULER (Hsieh et al., 2024) adds multi-key and multi-value needles, variable tracking across hops, aggregation and QA. Nearly all models it tested were near-perfect on the plain needle, yet only about half of those claiming 32K or more held up at 32K on the harder tasks. The effective context is usually shorter than the advertised one.
Liu, Lin, Hewitt, Paranjape, Bevilacqua, Petroni and Liang ("Lost in the Middle: How Language Models Use Long Contexts", 2023) put the one relevant passage among 10, 20 or 30 retrieved documents and moved it from first to last. Accuracy formed a U shape: best when the answer was at the start or the end, worst in the middle. For GPT-3.5-Turbo, accuracy with the answer in the middle fell below its closed-book accuracy with no documents at all. Extended-context versions of the same models were not better at using their input. The practical lesson for retrieval systems is to put the most relevant material at the start or end of the prompt, and to rerank rather than pad.
Worked example: Llama 2 7B from 4K to 32K
(64 pairs), , , target , so . The table gives each method's for representative pairs (1 = unchanged, 0.125 = fully interpolated).
| pair i | wavelength (tokens) | turns in L | PI | NTK-aware | YaRN |
|---|---|---|---|---|---|
| 0 | 6.3 | 651.90 | 0.125 | 1.000 | 1.000 |
| 20 | 111.7 | 36.66 | 0.125 | 0.517 | 1.000 |
| 30 | 471.2 | 8.69 | 0.125 | 0.371 | 0.342 |
| 40 | 1,986.9 | 2.06 | 0.125 | 0.267 | 0.155 |
| 46 | 4,711.7 | 0.87 | 0.125 | 0.219 | 0.125 |
| 50 | 8,378.8 | 0.49 | 0.125 | 0.192 | 0.125 |
| 63 | 54,410.1 | 0.08 | 0.125 | 0.125 | 0.125 |
Reading it
- 1Pairs 46–63 have : 18 pairs that never finish a turn within 4096 tokens. Unmodified, pair 50 would reach its trained arc at 32K.
- 2PI multiplies every row by 0.125. No pair leaves its trained range, but pair 0 now moves 0.125 rad per token. Adjacent tokens differ by of the angle the model learned to read.
- 3NTK-aware uses base . Pair 0 keeps 1.000 and pair 63 gets 0.125. Pair 46 gets 0.219, so it reaches times its trained arc. 17 slow pairs overshoot a little, which is why NTK-aware scaling without fine-tuning degrades as grows.
- 4YaRN keeps pairs with more than 32 turns (pairs 0–20), sets every pair with fewer than one turn to exactly 0.125, and ramps the 25 in between. Logits are scaled by .
- 5Cost at 32K: a stored score matrix would be bytes = 2 GiB per head and 64 GiB per layer, so FlashAttention is required. The KV cache is 16 GiB per sequence, and attention is about 40% of training FLOPs per token.
1import numpy as np23def rope_freqs(d_head=128, base=10000.0):4 return base ** (-np.arange(0, d_head, 2) / d_head) # theta_i, i = 0..d/2-156def scaled_freqs(method, s, L, d_head=128, base=10000.0, alpha=1.0, beta=32.0):7 th = rope_freqs(d_head, base)8 if method == "pi": # Chen et al. 2023: positions m -> m/s9 return th / s10 if method == "ntk": # NTK-aware: raise the base11 return rope_freqs(d_head, base * s ** (d_head / (d_head - 2)))12 if method == "yarn": # Peng et al. 2023 ("NTK-by-parts")13 turns = L * th / (2 * np.pi) # rotations within the trained context14 gamma = np.clip((turns - alpha) / (beta - alpha), 0.0, 1.0)15 return (1 - gamma) * th / s + gamma * th16 return th1718L, s = 4096, 8 # Llama 2: 4K -> 32K19th = rope_freqs()20turns = L * th / (2 * np.pi)21print("pairs with < 1 turn in L:", int((turns < 1).sum()), "first:", int(np.argmax(turns < 1)))22for m in ("pi", "ntk", "yarn"):23 r = scaled_freqs(m, s, L) / th24 new_angles = (turns < 1) & (s * r > 1 + 1e-9) # reach angles never seen in training25 print(f"{m:4s} kept {int(np.isclose(r, 1).sum()):2d} divided by s {int(np.isclose(r, 1 / s).sum()):2d} "26 f"extrapolated {int(new_angles.sum()):2d} pair-0 step {scaled_freqs(m, s, L)[0]:.3f} rad")27print("NTK base:", round(10000 * s ** (128 / 126)), " YaRN logit scale:", round((0.1 * np.log(s) + 1) ** 2, 4))28# pairs with < 1 turn in L: 18 first: 4629# pi kept 0 divided by s 64 extrapolated 0 pair-0 step 0.125 rad30# ntk kept 1 divided by s 1 extrapolated 17 pair-0 step 1.000 rad31# yarn kept 21 divided by s 18 extrapolated 0 pair-0 step 1.000 rad32# NTK base: 82685 YaRN logit scale: 1.45913334# Needle-in-a-haystack grid: one fact hidden at a given depth of a filler context.35NEEDLE = "The secret passphrase is 'blue-harbor-42'."36QUESTION = "What is the secret passphrase? Answer with the passphrase only."3738def build_prompt(filler_sentences, n_tokens, depth, tokens_per_sentence=12):39 k = n_tokens // tokens_per_sentence # rough length control40 hay = (filler_sentences * (k // len(filler_sentences) + 1))[:k]41 hay.insert(int(depth * len(hay)), NEEDLE) # depth 0 = start, 1 = end42 return " ".join(hay) + "\n\n" + QUESTION4344def niah_grid(ask, filler, lengths, depths):45 """ask(prompt) -> model answer; returns a (len(lengths), len(depths)) 0/1 accuracy grid."""46 return np.array([[float("blue-harbor-42" in ask(build_prompt(filler, n, d))) for d in depths]47 for n in lengths])4849filler = ["The committee met on Tuesday to review the budget.", "Rain is expected later in the week."]50oracle = lambda p: "blue-harbor-42" if NEEDLE in p else "unknown" # stand-in for a real model call51print(niah_grid(oracle, filler, lengths=[1000, 8000, 32000], depths=[0.0, 0.5, 1.0]).mean()) # 1.0Use when
- +Extension is cheap: about 1000 steps (PI) or a few billion tokens of continued pre-training, against trillions for pre-training
- +YaRN-style by-wavelength rescaling keeps local resolution and leaves no frequency out of range
- +FlashAttention and ring attention make exact attention at 128K+ feasible in memory
- +Sliding windows and local/global mixes bound the KV cache and cost per token
Avoid when
- −Attention FLOPs still grow as T²: about 40% of Llama 2 7B's training compute at 32K
- −Windowed and sparse attention can miss information outside the pattern
- −Advertised context often exceeds effective context (RULER), and accuracy dips in the middle
- −Extension can erode short-context quality unless the data mix is controlled
Where it's used
Long documents
contracts, books, papers and transcripts in one prompt instead of chunked summaries
Repository-level code
Code Llama's 16K training with a raised base was aimed at whole-file and multi-file completion
Retrieval-augmented generation
more retrieved passages per query; lost-in-the-middle argues for reranking and placing the best passages first or last (see Retrieval-augmented transformers)
Agents and chat
long tool traces and multi-turn histories, usually combined with KV-cache reuse across turns
Check yourself
A RoPE model with d_head = 128 and base 10,000 is trained on 4096 tokens and extended to 32K with position interpolation. Which statement is true?
Check yourself
In 'Lost in the Middle', where did multi-document QA accuracy tend to be lowest?
Remember
- ✓RoPE pairs with wavelength never finish a turn in training; at they meet unseen angles. For Llama 2 that is pairs 46–63 of 64.
- ✓PI: for all pairs, safe but loses local resolution. NTK-aware: base , keeps fast pairs. YaRN: interpolate below 1 turn, keep above 32, plus logits .
- ✓Raising the base for long training (Code Llama 10⁶, Llama 2 Long 500,000), LongRoPE's searched factors and ALiBi's linear bias are the other position routes.
- ✓FlashAttention removes the T² memory (not FLOPs); sliding-window and sparse attention cut work to O(T·W); ring attention splits one sequence over P devices.
- ✓Long-context ability needs long, domain-balanced training data and real tests: needle-in-a-haystack grids, RULER, and the lost-in-the-middle U curve.