Context window optimization

Extending a context window from L to s·L needs three fixes: rescale RoPE so no frequency meets unseen angles (position interpolation, NTK-aware, YaRN), make attention affordable (FlashAttention, sliding windows, ring attention), and train and test on long data.

15 min read1821 wordsadvanced0 viewsupdated 2026-10-13

Definition

Context window optimization is the set of methods that let a language model trained on sequences of length LL work on inputs of length s⋅Ls\cdot L. It covers three separate problems. Positions: the position encoding must stay inside what the model saw in training, as in position interpolation (Chen, Wong, Chen and Tian, 2023), NTK-aware scaling (2023), YaRN (Peng, Quesnelle, Fan and Shippole, 2023) and LongRoPE (Ding et al., 2024). Cost: attention must fit in memory and time, through FlashAttention, sliding-window or sparse attention, and ring attention across devices. Use: the model must be trained on long data and shown to retrieve and reason over it, which is measured with needle-in-a-haystack tests and "Lost in the Middle" (Liu et al., 2023).

GPT-3 had 2048 tokens of context and Llama 2 had 4096. Llama 3.1, Mistral's later models and most frontier models now accept 128K tokens or more. Almost none of that growth came from pre-training at full length from scratch, because attention cost grows as T2T^2 and most documents are short. Instead, models are pre-trained at 4K–8K tokens and then extended: the position encoding is adjusted, and a short phase of continued pre-training on long documents follows. The fixes below are therefore layered. Each solves one problem and leaves the others.

extending a context window from L to s·L: three separate problems1 positionsRoPE anglesnever seenbeyond L0L2L3L4L0 … L after PIPI: m ↦ m/s maps 0…4L back into 0…LNTK-aware, YaRN, LongRoPE: per frequencyALiBi: linear distance bias, nothing to rotate2 computeand memory:T² scores,T·L·n_kv KVfull causal: T²/2FlashAttention: exact,T×T never storedsliding window W: T·Wreach grows W per layer(Mistral: 4096 × 32)Q0Q1Q2Q3ring attention, P = 4T/P queries per GPU,K/V blocks pass round3 dataand eval:can it usethe context?pretrain4K–8K tokenslong-context CPTlong docs upsampledevaluateneedle, RULERlost in the middle (schematic):accuracy vs answer position:best at start and endfirst … answer position … lastdepth × length
fig.The three problems of a longer context. Row 1: a RoPE model trained on L tokens has never seen positions L…4L. Position interpolation maps them back into 0…L, and NTK-aware scaling, YaRN and LongRoPE do the same per frequency. ALiBi has no rotation to go out of range. Row 2: full causal attention scores T²/2 query–key pairs. FlashAttention computes them exactly without storing the T×T matrix. A sliding window keeps T·W pairs. Ring attention splits the sequence over P devices and passes key/value blocks around the ring. Row 3: long-context continued pre-training, then evaluation. The U-shaped curve and the needle grid are schematic, not data. Hover a row to highlight it.

Why RoPE models break past their training length

θi=b−2i/d,λi=2πθi=2π b2i/d,ri=Lλi=L θi2π(i=0,…,d2−1)\theta_i=b^{-2i/d},\qquad \lambda_i=\frac{2\pi}{\theta_i}=2\pi\,b^{2i/d},\qquad r_i=\frac{L}{\lambda_i}=\frac{L\,\theta_i}{2\pi}\quad(i=0,\dots,\tfrac d2-1)
θi\theta_i
rotation per token of the ii-th 2-D pair of a head (see Rotary positional embeddings (RoPE)); b=10,000b=10{,}000 in Llama 2 and 500,000500{,}000 in Llama 3
λi\lambda_i
wavelength: tokens per full turn, from 2π≈6.32\pi\approx6.3 for pair 0 to 54,41054{,}410 for pair 63 at d=128d=128, b=104b=10^4
rir_i
turns completed inside the trained context LL. Pairs with ri≥1r_i\ge1 have seen every angle. Pairs with ri<1r_i<1 have only seen the arc [0,Lθi][0,L\theta_i]

High-frequency pairs wrap around many times within LL, so at any longer length they show only angles the model already knows. Low-frequency pairs never complete a turn. At position sLsL they reach angles up to sLθisL\theta_i, which the attention weights were never trained on. For Llama 2 (dh=128d_h=128, b=104b=10^4, L=4096L=4096) that is pairs 46–63, 18 of the 64. These slow pairs carry the coarse "how far away" signal, and when they leave the trained range, attention scores become erratic and perplexity blows up within a few hundred tokens past LL.

Rescaling the frequencies

PI: θi′=θisNTK-aware: θi′=(b sd/(d−2))−2i/dYaRN: θi′=(1−γ(ri))θis+γ(ri) θi,  γ(r)=clip(r−αβ−α,0,1)\text{PI: }\theta'_i=\frac{\theta_i}{s}\qquad \text{NTK-aware: }\theta'_i=\big(b\,s^{d/(d-2)}\big)^{-2i/d}\qquad \text{YaRN: }\theta'_i=\big(1-\gamma(r_i)\big)\frac{\theta_i}{s}+\gamma(r_i)\,\theta_i,\ \ \gamma(r)=\mathrm{clip}\Big(\frac{r-\alpha}{\beta-\alpha},0,1\Big)
PI\text{PI}
position interpolation: equivalent to feeding position m/sm/s. Nothing extrapolates, but neighbouring tokens become ss times closer in every pair
b sd/(d−2)b\,s^{d/(d-2)}
NTK-aware scaling ("bloc97", Reddit, 2023): a larger base leaves pair 0 untouched and divides the slowest pair by exactly ss; pairs in between are slowed geometrically
γ(ri)\gamma(r_i)
YaRN's ramp on turns per context: fully interpolate pairs with r<α=1r<\alpha=1, keep pairs with r>β=32r>\beta=32, blend linearly between
(0.1ln⁡s+1)2(0.1\ln s+1)^2
YaRN also multiplies the attention logits by this factor, 1.459 at s=8s=8, to undo the flatter softmax at long range

Chen et al. proved that interpolated attention scores are far better behaved than extrapolated ones. They bounded the interpolated case about 600 times tighter, and extended LLaMA 7B–65B from 2048 to 32,768 tokens with about 1000 fine-tuning steps. Plain fine-tuning at the longer length adapted far more slowly. PI's cost is resolution. The fastest pair, which tells adjacent tokens apart, now turns only 1/s1/s radian per token. NTK-aware scaling keeps the high frequencies and stretches mostly the low ones, and it works reasonably even with no fine-tuning, which made it popular in inference tools. Its "dynamic" variant raises ss only as the current sequence grows past LL. YaRN makes the split explicit by wavelength and adds the temperature. Peng et al. extended Llama 2 7B and 13B to 64K and 128K context while using about 10 times fewer tokens and 2.5 times fewer steps than earlier methods. Llama 3.1's released configuration uses a similar wavelength-based split with factor 8 from 8192 tokens.

Two other routes avoid the problem. Raising the base before long training, as Code Llama did (base 10610^6, trained on 16K sequences, stable up to about 100K tokens), slows every pair so that fewer of them are out of range at the target length. Meta's Llama 2 Long (Xiong et al., 2023) did the same with base 500,000 during 400B tokens of continued pre-training. LongRoPE (Ding et al., 2024) searches for a separate rescale factor per dimension with an evolutionary search, extends in stages to 2048K tokens, and readjusts the factors so that short-context quality recovers. Phi-3's 128K models use it. ALiBi (Press, Smith and Lewis, ICLR 2022) has no rotation at all: it adds −mh(i−j)-m_h(i-j) to each logit, so a longer input only adds larger penalties, and models trained on 1024 tokens ran at 2048 and beyond. Unlimited extrapolation does not mean the model uses distant tokens, though. ALiBi's penalties make far-away tokens nearly invisible.

Context extensions in published models

Position interpolation
LLaMA 7B–65B, 2048 → 32,768 tokens (s=16s=16) with about 1000 fine-tuning steps (Chen et al., 2023)
YaRN
Llama 2 7B/13B to 64K (s=16s=16) and 128K (s=32s=32); α=1\alpha=1, β=32\beta=32; logits ×(0.1ln⁡s+1)2\times(0.1\ln s+1)^2
Code Llama
RoPE base raised from 10410^4 to 10610^6, fine-tuned on 16,384-token sequences, usable to about 100K tokens
Llama 2 Long
base 500,000, continued pre-training on 400B tokens of long sequences (Xiong et al., 2023)
Llama 3.1
base 500,000; 8K → 128K in six stages of continued pre-training using about 800B tokens
Mistral 7B
sliding window W=4096W=4096 in each of 32 layers, so information can travel about 4096⋅32=131,0724096\cdot32=131{,}072 tokens through the stack
Gemma 2
alternates local sliding-window layers (4096) with global layers (8192)
LongRoPE
searched non-uniform per-dimension factors, extension to 2048K tokens; used for Phi-3's 128K variants
pair i turns m·θᵢ at position m, θᵢ = 10,000^(−2i/128) · trained L = 4,096 → s·L = 32,768θ′ᵢ/θᵢ per pair, log scale: ■ kept ■ partly slowed ■ divided by s ▲ new angles at s·L11/21/41/81/161/32▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲pair 0: λ = 6.3 tokens/turn■ under one turn in L: 18 pairspair 63: λ = 54,410pair 50: angles it reaches (from 0, counter-clockwise)0pair 50━ trained 0…L·θ━ beyond it at s·Lθ = 0.000750 rad/token, λ = 2π/θ = 8,379 tokens per turntraining reached 3.07 rad = 0.49 turnsat s·L: θ′ = 0.000750, reaches 24.57 rad (θ′/θ = 1.000)out of distribution: 8.00× the trained rangepairs with new angles at s·L: 18 of 64kept: 64 · divided by s: 0 · partly slowed: 0fastest pair moves 1.000 rad/token (trained 1.000)NTK base′ = base·s^(128/126) · YaRN: under α = 1 turn in L → θ/s, over β = 32 → θ · click a bar
method
RoPE base
fig.The 64 frequency pairs of one 128-dimensional RoPE head, trained on L = 4096 tokens with base 10,000 (Llama 2) and run at s·L. Bars show θ′/θ on a log axis, and the orange strip marks the 18 pairs (46–63) that never complete a turn within 4096 tokens. At the defaults (s = 8, no change) pair 50 has a wavelength of 8,379 tokens. Training took it to 3.07 rad, 0.49 of a turn, and at 32,768 tokens it reaches 24.57 rad, 8.00× the trained arc. All 18 slow pairs get new angles (red ▲). Position interpolation divides every pair by 8: no new angles, but the fastest pair now moves 0.125 rad per token instead of 1. NTK-aware scaling (base 82,685) keeps pair 0, divides pair 63 by 8 and slows the 62 in between. 17 slow pairs still overshoot: pair 50 reaches 4.72 rad (1.54×) and pair 46 1.75×. YaRN keeps the 21 pairs with more than 32 turns in L (pairs 0–20), divides the 18 slow pairs by exactly 8, ramps the 25 in between, and scales the logits by 1.459 (1.813 at s = 32). It leaves no new angles and keeps the fastest pair at 1 rad per token. Switch to Llama 3 (base 500,000, L = 8192) at s = 16: 29 pairs (35–63) never finish a turn, NTK-aware leaves 28 of them overshooting, and YaRN keeps 19 and interpolates 29. Click a bar to see that pair on the dial.

Making long attention affordable

FLOPsattn≈4 T2d per layer (forward, full),scores=H T2 b bytes if stored,KV=2 L nkv dh T b\text{FLOPs}_{\text{attn}}\approx 4\,T^2 d\ \text{per layer (forward, full)},\qquad \text{scores}=H\,T^2\,b\ \text{bytes if stored},\qquad \text{KV}=2\,L\,n_{kv}\,d_h\,T\,b
4T2d4T^2d
QK⊤QK^\top and AVAV, each 2T2d2T^2d; a causal mask halves the useful work
HT2bHT^2b
Llama 2 7B at T=32,768T=32{,}768: 2 GiB per head and 64 GiB per layer in bf16, which is why the matrix must never exist in full
KV\text{KV}
linear in TT, 16 GiB for one 32K sequence of Llama 2 7B (see KV cache)

FlashAttention (Dao et al., 2022) computes exact attention tile by tile in on-chip SRAM with an online softmax, so memory is O(T)O(T) and the score matrix is never written to HBM (see FlashAttention). It removes the memory wall but not the T2T^2 FLOPs. At 32K tokens attention is about 40% of Llama 2 7B's training FLOPs per token. Sliding-window attention caps each query at the last WW keys, giving O(TW)O(TW) work and a cache bounded by WW. Stacking ℓ\ell layers lets information travel ℓW\ell W tokens. Mistral 7B uses W=4096W=4096, and Gemma 2 interleaves local and global layers so some layers still see everything. Fixed sparse patterns (Child et al., 2019; Longformer; BigBird) mix local windows, strides and global tokens (see Sparse attention). All of these are approximations that can miss a needle outside the pattern, which is why local/global designs keep some full-attention layers.

device p holds Qp,Kp,Vp (T/P tokens);for j=0…P−1:  op⊕=attn(Qp,K(p−j) mod P,V(p−j) mod P), then send K,V to p+1\text{device } p \text{ holds } Q_p,K_p,V_p\ (T/P\text{ tokens});\quad \text{for } j=0\dots P-1:\ \ o_p \mathrel{\oplus}= \mathrm{attn}\big(Q_p,K_{(p-j)\bmod P},V_{(p-j)\bmod P}\big),\ \text{then send } K,V \text{ to } p+1
PP
devices in the ring; activation memory per device scales as T/P
⊕\oplus
the online-softmax merge of FlashAttention, so partial results combine exactly
send\text{send}
each step passes 2 (T/P) nkvdh2\,(T/P)\,n_{kv}d_h numbers per layer to the next device, overlapped with the block computation

Ring attention (Liu, Zaharia and Abbeel, 2023) is sequence parallelism for attention. When one sequence is too long for one device's activations, split it into PP blocks. Each device keeps its query block and streams the key/value blocks around a ring, computing blockwise attention while the next block is in flight. The maximum context grows linearly with the number of devices. Training frameworks call this context parallelism (see Distributed LLM training). Causal masking makes the naive split unbalanced, because the device holding the first block has almost nothing to attend to. Common implementations therefore assign each device a block from the start and one from the end.

Data: what long-context training needs

Changing the frequencies only stops the model from breaking. To use a long context, the model needs training examples where distant tokens matter. Natural long documents are rare and skewed toward books, code repositories and papers. Fu et al. ("Data Engineering for Scaling Language Models to 128K Context", 2024) found that a few billion tokens of continued pre-training are enough. The important choice was to upsample long sequences within each domain and keep the domain mix the same. Upsampling long documents regardless of source shifts the mix toward books and code, and it hurt. Xiong et al. found that data quality mattered more than the sheer amount of long text. Llama 3.1 grew its context in six stages and moved to the next stage only after short-context scores had recovered and needle tests passed at the current length. Recipes also mix in synthetic long tasks, such as retrieval over concatenated documents or long-document QA, and keep a share of short data so that short-context quality does not regress.

Evaluation: does it use the context?

Needle in a haystack (Greg Kamradt, 2023) hides one out-of-place sentence ("The best thing to do in San Francisco is eat a sandwich and sit in Dolores Park on a sunny day.") at a chosen depth in filler text of a chosen length, then asks for it back. The result is a depth × length grid of pass/fail cells. It is cheap and catches positional breakage at once, but a single verbatim match is easy. RULER (Hsieh et al., 2024) adds multi-key and multi-value needles, variable tracking across hops, aggregation and QA. Nearly all models it tested were near-perfect on the plain needle, yet only about half of those claiming 32K or more held up at 32K on the harder tasks. The effective context is usually shorter than the advertised one.

Liu, Lin, Hewitt, Paranjape, Bevilacqua, Petroni and Liang ("Lost in the Middle: How Language Models Use Long Contexts", 2023) put the one relevant passage among 10, 20 or 30 retrieved documents and moved it from first to last. Accuracy formed a U shape: best when the answer was at the start or the end, worst in the middle. For GPT-3.5-Turbo, accuracy with the answer in the middle fell below its closed-book accuracy with no documents at all. Extended-context versions of the same models were not better at using their input. The practical lesson for retrieval systems is to put the most relevant material at the start or end of the prompt, and to rerank rather than pad.

Worked example: Llama 2 7B from 4K to 32K

dh=128d_h=128 (64 pairs), b=10,000b=10{,}000, L=4096L=4096, target 32,76832{,}768, so s=8s=8. The table gives each method's θi′/θi\theta'_i/\theta_i for representative pairs (1 = unchanged, 0.125 = fully interpolated).

pair iwavelength (tokens)turns in LPINTK-awareYaRN
06.3651.900.1251.0001.000
20111.736.660.1250.5171.000
30471.28.690.1250.3710.342
401,986.92.060.1250.2670.155
464,711.70.870.1250.2190.125
508,378.80.490.1250.1920.125
6354,410.10.080.1250.1250.125
Per-pair rescaling at s = 8 (computed with the numpy code below)

Reading it

  1. 1Pairs 46–63 have ri<1r_i<1: 18 pairs that never finish a turn within 4096 tokens. Unmodified, pair 50 would reach 8×8\times its trained arc at 32K.
  2. 2PI multiplies every row by 0.125. No pair leaves its trained range, but pair 0 now moves 0.125 rad per token. Adjacent tokens differ by 1/81/8 of the angle the model learned to read.
  3. 3NTK-aware uses base 10,000⋅8128/126=82,68510{,}000\cdot8^{128/126}=82{,}685. Pair 0 keeps 1.000 and pair 63 gets 0.125. Pair 46 gets 0.219, so it reaches 8⋅0.219=1.758\cdot0.219=1.75 times its trained arc. 17 slow pairs overshoot a little, which is why NTK-aware scaling without fine-tuning degrades as ss grows.
  4. 4YaRN keeps pairs with more than 32 turns (pairs 0–20), sets every pair with fewer than one turn to exactly 0.125, and ramps the 25 in between. Logits are scaled by (0.1ln⁡8+1)2=1.4591(0.1\ln8+1)^2=1.4591.
  5. 5Cost at 32K: a stored score matrix would be 32,7682⋅232{,}768^2\cdot2 bytes = 2 GiB per head and 64 GiB per layer, so FlashAttention is required. The KV cache is 16 GiB per sequence, and attention is about 40% of training FLOPs per token.
python
1import numpy as np
2
3def rope_freqs(d_head=128, base=10000.0):
4 return base ** (-np.arange(0, d_head, 2) / d_head) # theta_i, i = 0..d/2-1
5
6def scaled_freqs(method, s, L, d_head=128, base=10000.0, alpha=1.0, beta=32.0):
7 th = rope_freqs(d_head, base)
8 if method == "pi": # Chen et al. 2023: positions m -> m/s
9 return th / s
10 if method == "ntk": # NTK-aware: raise the base
11 return rope_freqs(d_head, base * s ** (d_head / (d_head - 2)))
12 if method == "yarn": # Peng et al. 2023 ("NTK-by-parts")
13 turns = L * th / (2 * np.pi) # rotations within the trained context
14 gamma = np.clip((turns - alpha) / (beta - alpha), 0.0, 1.0)
15 return (1 - gamma) * th / s + gamma * th
16 return th
17
18L, s = 4096, 8 # Llama 2: 4K -> 32K
19th = rope_freqs()
20turns = L * th / (2 * np.pi)
21print("pairs with < 1 turn in L:", int((turns < 1).sum()), "first:", int(np.argmax(turns < 1)))
22for m in ("pi", "ntk", "yarn"):
23 r = scaled_freqs(m, s, L) / th
24 new_angles = (turns < 1) & (s * r > 1 + 1e-9) # reach angles never seen in training
25 print(f"{m:4s} kept {int(np.isclose(r, 1).sum()):2d} divided by s {int(np.isclose(r, 1 / s).sum()):2d} "
26 f"extrapolated {int(new_angles.sum()):2d} pair-0 step {scaled_freqs(m, s, L)[0]:.3f} rad")
27print("NTK base:", round(10000 * s ** (128 / 126)), " YaRN logit scale:", round((0.1 * np.log(s) + 1) ** 2, 4))
28# pairs with < 1 turn in L: 18 first: 46
29# pi kept 0 divided by s 64 extrapolated 0 pair-0 step 0.125 rad
30# ntk kept 1 divided by s 1 extrapolated 17 pair-0 step 1.000 rad
31# yarn kept 21 divided by s 18 extrapolated 0 pair-0 step 1.000 rad
32# NTK base: 82685 YaRN logit scale: 1.4591
33
34# Needle-in-a-haystack grid: one fact hidden at a given depth of a filler context.
35NEEDLE = "The secret passphrase is 'blue-harbor-42'."
36QUESTION = "What is the secret passphrase? Answer with the passphrase only."
37
38def build_prompt(filler_sentences, n_tokens, depth, tokens_per_sentence=12):
39 k = n_tokens // tokens_per_sentence # rough length control
40 hay = (filler_sentences * (k // len(filler_sentences) + 1))[:k]
41 hay.insert(int(depth * len(hay)), NEEDLE) # depth 0 = start, 1 = end
42 return " ".join(hay) + "\n\n" + QUESTION
43
44def niah_grid(ask, filler, lengths, depths):
45 """ask(prompt) -> model answer; returns a (len(lengths), len(depths)) 0/1 accuracy grid."""
46 return np.array([[float("blue-harbor-42" in ask(build_prompt(filler, n, d))) for d in depths]
47 for n in lengths])
48
49filler = ["The committee met on Tuesday to review the budget.", "Rain is expected later in the week."]
50oracle = lambda p: "blue-harbor-42" if NEEDLE in p else "unknown" # stand-in for a real model call
51print(niah_grid(oracle, filler, lengths=[1000, 8000, 32000], depths=[0.0, 0.5, 1.0]).mean()) # 1.0
RoPE frequency rescaling for PI, NTK-aware and YaRN with the per-pair classification, plus a needle-in-a-haystack grid harness (run with numpy; the model call is a stand-in)

Use when

  • +Extension is cheap: about 1000 steps (PI) or a few billion tokens of continued pre-training, against trillions for pre-training
  • +YaRN-style by-wavelength rescaling keeps local resolution and leaves no frequency out of range
  • +FlashAttention and ring attention make exact attention at 128K+ feasible in memory
  • +Sliding windows and local/global mixes bound the KV cache and cost per token

Avoid when

  • −Attention FLOPs still grow as T²: about 40% of Llama 2 7B's training compute at 32K
  • −Windowed and sparse attention can miss information outside the pattern
  • −Advertised context often exceeds effective context (RULER), and accuracy dips in the middle
  • −Extension can erode short-context quality unless the data mix is controlled

Where it's used

Long documents

contracts, books, papers and transcripts in one prompt instead of chunked summaries

Repository-level code

Code Llama's 16K training with a raised base was aimed at whole-file and multi-file completion

Retrieval-augmented generation

more retrieved passages per query; lost-in-the-middle argues for reranking and placing the best passages first or last (see Retrieval-augmented transformers)

Agents and chat

long tool traces and multi-turn histories, usually combined with KV-cache reuse across turns

Check yourself

A RoPE model with d_head = 128 and base 10,000 is trained on 4096 tokens and extended to 32K with position interpolation. Which statement is true?

Check yourself

In 'Lost in the Middle', where did multi-document QA accuracy tend to be lowest?

Remember

  • ✓RoPE pairs with wavelength λi=2πb2i/d>L\lambda_i=2\pi b^{2i/d}>L never finish a turn in training; at sLsL they meet unseen angles. For Llama 2 that is pairs 46–63 of 64.
  • ✓PI: θi/s\theta_i/s for all pairs, safe but loses local resolution. NTK-aware: base b sd/(d−2)b\,s^{d/(d-2)}, keeps fast pairs. YaRN: interpolate below 1 turn, keep above 32, plus logits ×(0.1ln⁡s+1)2\times(0.1\ln s+1)^2.
  • ✓Raising the base for long training (Code Llama 10⁶, Llama 2 Long 500,000), LongRoPE's searched factors and ALiBi's linear bias are the other position routes.
  • ✓FlashAttention removes the T² memory (not FLOPs); sliding-window and sparse attention cut work to O(T·W); ring attention splits one sequence over P devices.
  • ✓Long-context ability needs long, domain-balanced training data and real tests: needle-in-a-haystack grids, RULER, and the lost-in-the-middle U curve.
← back to Deep Learning roadmap