# Context window optimization

> Extending a context window from L to s·L needs three fixes: rescale RoPE so no frequency meets unseen angles (position interpolation, NTK-aware, YaRN), make attention affordable (FlashAttention, sliding windows, ring attention), and train and test on long data.

Source: https://www.mlfoundry.in/learn/context-window-optimization  
Section: Deep Learning › Large Language Models (LLMs)  
Level: advanced · 15 min read · updated 2026-10-13

**Context window optimization:** is the set of methods that let a language model trained on sequences of length $L$ work on inputs of length $s\cdot L$. It covers three separate problems. **Positions**: the position encoding must stay inside what the model saw in training, as in position interpolation (Chen, Wong, Chen and Tian, 2023), NTK-aware scaling (2023), YaRN (Peng, Quesnelle, Fan and Shippole, 2023) and LongRoPE (Ding et al., 2024). **Cost**: attention must fit in memory and time, through FlashAttention, sliding-window or sparse attention, and ring attention across devices. **Use**: the model must be trained on long data and shown to retrieve and reason over it, which is measured with needle-in-a-haystack tests and "Lost in the Middle" (Liu et al., 2023).

GPT-3 had 2048 tokens of context and Llama 2 had 4096. Llama 3.1, Mistral's later models and most frontier models now accept 128K tokens or more. Almost none of that growth came from pre-training at full length from scratch, because attention cost grows as $T^2$ and most documents are short. Instead, models are pre-trained at 4K–8K tokens and then *extended*: the position encoding is adjusted, and a short phase of continued pre-training on long documents follows. The fixes below are therefore layered. Each solves one problem and leaves the others.

> Figure: The three problems of a longer context. Row 1: a RoPE model trained on L tokens has never seen positions L…4L. Position interpolation maps them back into 0…L, and NTK-aware scaling, YaRN and LongRoPE do the same per frequency. ALiBi has no rotation to go out of range. Row 2: full causal attention scores T²/2 query–key pairs. FlashAttention computes them exactly without storing the T×T matrix. A sliding window keeps T·W pairs. Ring attention splits the sequence over P devices and passes key/value blocks around the ring. Row 3: long-context continued pre-training, then evaluation. The U-shaped curve and the needle grid are schematic, not data. Hover a row to highlight it.

## Why RoPE models break past their training length

$$
\theta_i=b^{-2i/d},\qquad \lambda_i=\frac{2\pi}{\theta_i}=2\pi\,b^{2i/d},\qquad r_i=\frac{L}{\lambda_i}=\frac{L\,\theta_i}{2\pi}\quad(i=0,\dots,\tfrac d2-1)
$$

- $\theta_i$: rotation per token of the $i$-th 2-D pair of a head (see **Rotary positional embeddings (RoPE)**); $b=10{,}000$ in Llama 2 and $500{,}000$ in Llama 3
- $\lambda_i$: wavelength: tokens per full turn, from $2\pi\approx6.3$ for pair 0 to $54{,}410$ for pair 63 at $d=128$, $b=10^4$
- $r_i$: turns completed inside the trained context $L$. Pairs with $r_i\ge1$ have seen every angle. Pairs with $r_i<1$ have only seen the arc $[0,L\theta_i]$

High-frequency pairs wrap around many times within $L$, so at any longer length they show only angles the model already knows. Low-frequency pairs never complete a turn. At position $sL$ they reach angles up to $sL\theta_i$, which the attention weights were never trained on. For Llama 2 ($d_h=128$, $b=10^4$, $L=4096$) that is pairs 46–63, 18 of the 64. These slow pairs carry the coarse "how far away" signal, and when they leave the trained range, attention scores become erratic and perplexity blows up within a few hundred tokens past $L$.

## Rescaling the frequencies

$$
\text{PI: }\theta'_i=\frac{\theta_i}{s}\qquad \text{NTK-aware: }\theta'_i=\big(b\,s^{d/(d-2)}\big)^{-2i/d}\qquad \text{YaRN: }\theta'_i=\big(1-\gamma(r_i)\big)\frac{\theta_i}{s}+\gamma(r_i)\,\theta_i,\ \ \gamma(r)=\mathrm{clip}\Big(\frac{r-\alpha}{\beta-\alpha},0,1\Big)
$$

- $\text{PI}$: position interpolation: equivalent to feeding position $m/s$. Nothing extrapolates, but neighbouring tokens become $s$ times closer in every pair
- $b\,s^{d/(d-2)}$: NTK-aware scaling ("bloc97", Reddit, 2023): a larger base leaves pair 0 untouched and divides the slowest pair by exactly $s$; pairs in between are slowed geometrically
- $\gamma(r_i)$: YaRN's ramp on turns per context: fully interpolate pairs with $r<\alpha=1$, keep pairs with $r>\beta=32$, blend linearly between
- $(0.1\ln s+1)^2$: YaRN also multiplies the attention logits by this factor, 1.459 at $s=8$, to undo the flatter softmax at long range

Chen et al. proved that interpolated attention scores are far better behaved than extrapolated ones. They bounded the interpolated case about 600 times tighter, and extended LLaMA 7B–65B from 2048 to 32,768 tokens with about 1000 fine-tuning steps. Plain fine-tuning at the longer length adapted far more slowly. PI's cost is resolution. The fastest pair, which tells adjacent tokens apart, now turns only $1/s$ radian per token. NTK-aware scaling keeps the high frequencies and stretches mostly the low ones, and it works reasonably even with no fine-tuning, which made it popular in inference tools. Its "dynamic" variant raises $s$ only as the current sequence grows past $L$. YaRN makes the split explicit by wavelength and adds the temperature. Peng et al. extended Llama 2 7B and 13B to 64K and 128K context while using about 10 times fewer tokens and 2.5 times fewer steps than earlier methods. Llama 3.1's released configuration uses a similar wavelength-based split with factor 8 from 8192 tokens.

Two other routes avoid the problem. **Raising the base before long training**, as Code Llama did (base $10^6$, trained on 16K sequences, stable up to about 100K tokens), slows every pair so that fewer of them are out of range at the target length. Meta's Llama 2 Long (Xiong et al., 2023) did the same with base 500,000 during 400B tokens of continued pre-training. **LongRoPE** (Ding et al., 2024) searches for a separate rescale factor per dimension with an evolutionary search, extends in stages to 2048K tokens, and readjusts the factors so that short-context quality recovers. Phi-3's 128K models use it. **ALiBi** (Press, Smith and Lewis, ICLR 2022) has no rotation at all: it adds $-m_h(i-j)$ to each logit, so a longer input only adds larger penalties, and models trained on 1024 tokens ran at 2048 and beyond. Unlimited extrapolation does not mean the model uses distant tokens, though. ALiBi's penalties make far-away tokens nearly invisible.

**Context extensions in published models**

- **Position interpolation:** LLaMA 7B–65B, 2048 → 32,768 tokens ($s=16$) with about 1000 fine-tuning steps (Chen et al., 2023)
- **YaRN:** Llama 2 7B/13B to 64K ($s=16$) and 128K ($s=32$); $\alpha=1$, $\beta=32$; logits $\times(0.1\ln s+1)^2$
- **Code Llama:** RoPE base raised from $10^4$ to $10^6$, fine-tuned on 16,384-token sequences, usable to about 100K tokens
- **Llama 2 Long:** base 500,000, continued pre-training on 400B tokens of long sequences (Xiong et al., 2023)
- **Llama 3.1:** base 500,000; 8K → 128K in six stages of continued pre-training using about 800B tokens
- **Mistral 7B:** sliding window $W=4096$ in each of 32 layers, so information can travel about $4096\cdot32=131{,}072$ tokens through the stack
- **Gemma 2:** alternates local sliding-window layers (4096) with global layers (8192)
- **LongRoPE:** searched non-uniform per-dimension factors, extension to 2048K tokens; used for Phi-3's 128K variants

> Figure: The 64 frequency pairs of one 128-dimensional RoPE head, trained on L = 4096 tokens with base 10,000 (Llama 2) and run at s·L. Bars show θ′/θ on a log axis, and the orange strip marks the 18 pairs (46–63) that never complete a turn within 4096 tokens. At the defaults (s = 8, no change) pair 50 has a wavelength of 8,379 tokens. Training took it to 3.07 rad, 0.49 of a turn, and at 32,768 tokens it reaches 24.57 rad, 8.00× the trained arc. All 18 slow pairs get new angles (red ▲). Position interpolation divides every pair by 8: no new angles, but the fastest pair now moves 0.125 rad per token instead of 1. NTK-aware scaling (base 82,685) keeps pair 0, divides pair 63 by 8 and slows the 62 in between. 17 slow pairs still overshoot: pair 50 reaches 4.72 rad (1.54×) and pair 46 1.75×. YaRN keeps the 21 pairs with more than 32 turns in L (pairs 0–20), divides the 18 slow pairs by exactly 8, ramps the 25 in between, and scales the logits by 1.459 (1.813 at s = 32). It leaves no new angles and keeps the fastest pair at 1 rad per token. Switch to Llama 3 (base 500,000, L = 8192) at s = 16: 29 pairs (35–63) never finish a turn, NTK-aware leaves 28 of them overshooting, and YaRN keeps 19 and interpolates 29. Click a bar to see that pair on the dial.

## Making long attention affordable

$$
\text{FLOPs}_{\text{attn}}\approx 4\,T^2 d\ \text{per layer (forward, full)},\qquad \text{scores}=H\,T^2\,b\ \text{bytes if stored},\qquad \text{KV}=2\,L\,n_{kv}\,d_h\,T\,b
$$

- $4T^2d$: $QK^\top$ and $AV$, each $2T^2d$; a causal mask halves the useful work
- $HT^2b$: Llama 2 7B at $T=32{,}768$: 2 GiB per head and 64 GiB per layer in bf16, which is why the matrix must never exist in full
- $\text{KV}$: linear in $T$, 16 GiB for one 32K sequence of Llama 2 7B (see **KV cache**)

**FlashAttention** (Dao et al., 2022) computes exact attention tile by tile in on-chip SRAM with an online softmax, so memory is $O(T)$ and the score matrix is never written to HBM (see **FlashAttention**). It removes the memory wall but not the $T^2$ FLOPs. At 32K tokens attention is about 40% of Llama 2 7B's training FLOPs per token. **Sliding-window attention** caps each query at the last $W$ keys, giving $O(TW)$ work and a cache bounded by $W$. Stacking $\ell$ layers lets information travel $\ell W$ tokens. Mistral 7B uses $W=4096$, and Gemma 2 interleaves local and global layers so some layers still see everything. Fixed **sparse patterns** (Child et al., 2019; Longformer; BigBird) mix local windows, strides and global tokens (see **Sparse attention**). All of these are approximations that can miss a needle outside the pattern, which is why local/global designs keep some full-attention layers.

$$
\text{device } p \text{ holds } Q_p,K_p,V_p\ (T/P\text{ tokens});\quad \text{for } j=0\dots P-1:\ \ o_p \mathrel{\oplus}= \mathrm{attn}\big(Q_p,K_{(p-j)\bmod P},V_{(p-j)\bmod P}\big),\ \text{then send } K,V \text{ to } p+1
$$

- $P$: devices in the ring; activation memory per device scales as T/P
- $\oplus$: the online-softmax merge of FlashAttention, so partial results combine exactly
- $\text{send}$: each step passes $2\,(T/P)\,n_{kv}d_h$ numbers per layer to the next device, overlapped with the block computation

**Ring attention** (Liu, Zaharia and Abbeel, 2023) is sequence parallelism for attention. When one sequence is too long for one device's activations, split it into $P$ blocks. Each device keeps its query block and streams the key/value blocks around a ring, computing blockwise attention while the next block is in flight. The maximum context grows linearly with the number of devices. Training frameworks call this **context parallelism** (see **Distributed LLM training**). Causal masking makes the naive split unbalanced, because the device holding the first block has almost nothing to attend to. Common implementations therefore assign each device a block from the start and one from the end.

## Data: what long-context training needs

Changing the frequencies only stops the model from breaking. To *use* a long context, the model needs training examples where distant tokens matter. Natural long documents are rare and skewed toward books, code repositories and papers. Fu et al. ("Data Engineering for Scaling Language Models to 128K Context", 2024) found that a few billion tokens of continued pre-training are enough. The important choice was to upsample long sequences *within* each domain and keep the domain mix the same. Upsampling long documents regardless of source shifts the mix toward books and code, and it hurt. Xiong et al. found that data quality mattered more than the sheer amount of long text. Llama 3.1 grew its context in six stages and moved to the next stage only after short-context scores had recovered and needle tests passed at the current length. Recipes also mix in synthetic long tasks, such as retrieval over concatenated documents or long-document QA, and keep a share of short data so that short-context quality does not regress.

## Evaluation: does it use the context?

**Needle in a haystack** (Greg Kamradt, 2023) hides one out-of-place sentence ("The best thing to do in San Francisco is eat a sandwich and sit in Dolores Park on a sunny day.") at a chosen depth in filler text of a chosen length, then asks for it back. The result is a depth × length grid of pass/fail cells. It is cheap and catches positional breakage at once, but a single verbatim match is easy. **RULER** (Hsieh et al., 2024) adds multi-key and multi-value needles, variable tracking across hops, aggregation and QA. Nearly all models it tested were near-perfect on the plain needle, yet only about half of those claiming 32K or more held up at 32K on the harder tasks. The *effective* context is usually shorter than the advertised one.

Liu, Lin, Hewitt, Paranjape, Bevilacqua, Petroni and Liang ("Lost in the Middle: How Language Models Use Long Contexts", 2023) put the one relevant passage among 10, 20 or 30 retrieved documents and moved it from first to last. Accuracy formed a **U shape**: best when the answer was at the start or the end, worst in the middle. For GPT-3.5-Turbo, accuracy with the answer in the middle fell below its closed-book accuracy with no documents at all. Extended-context versions of the same models were not better at using their input. The practical lesson for retrieval systems is to put the most relevant material at the start or end of the prompt, and to rerank rather than pad.

## Worked example: Llama 2 7B from 4K to 32K

$d_h=128$ (64 pairs), $b=10{,}000$, $L=4096$, target $32{,}768$, so $s=8$. The table gives each method's $\theta'_i/\theta_i$ for representative pairs (1 = unchanged, 0.125 = fully interpolated).

Per-pair rescaling at s = 8 (computed with the numpy code below)

| pair i | wavelength (tokens) | turns in L | PI | NTK-aware | YaRN |
| --- | --- | --- | --- | --- | --- |
| 0 | 6.3 | 651.90 | 0.125 | 1.000 | 1.000 |
| 20 | 111.7 | 36.66 | 0.125 | 0.517 | 1.000 |
| 30 | 471.2 | 8.69 | 0.125 | 0.371 | 0.342 |
| 40 | 1,986.9 | 2.06 | 0.125 | 0.267 | 0.155 |
| 46 | 4,711.7 | 0.87 | 0.125 | 0.219 | 0.125 |
| 50 | 8,378.8 | 0.49 | 0.125 | 0.192 | 0.125 |
| 63 | 54,410.1 | 0.08 | 0.125 | 0.125 | 0.125 |

**Reading it**

1. Pairs 46–63 have $r_i<1$: 18 pairs that never finish a turn within 4096 tokens. Unmodified, pair 50 would reach $8\times$ its trained arc at 32K.
2. PI multiplies every row by 0.125. No pair leaves its trained range, but pair 0 now moves 0.125 rad per token. Adjacent tokens differ by $1/8$ of the angle the model learned to read.
3. NTK-aware uses base $10{,}000\cdot8^{128/126}=82{,}685$. Pair 0 keeps 1.000 and pair 63 gets 0.125. Pair 46 gets 0.219, so it reaches $8\cdot0.219=1.75$ times its trained arc. 17 slow pairs overshoot a little, which is why NTK-aware scaling without fine-tuning degrades as $s$ grows.
4. YaRN keeps pairs with more than 32 turns (pairs 0–20), sets every pair with fewer than one turn to exactly 0.125, and ramps the 25 in between. Logits are scaled by $(0.1\ln8+1)^2=1.4591$.
5. Cost at 32K: a stored score matrix would be $32{,}768^2\cdot2$ bytes = 2 GiB per head and 64 GiB per layer, so FlashAttention is required. The KV cache is 16 GiB per sequence, and attention is about 40% of training FLOPs per token.

```python
import numpy as np

def rope_freqs(d_head=128, base=10000.0):
    return base ** (-np.arange(0, d_head, 2) / d_head)            # theta_i, i = 0..d/2-1

def scaled_freqs(method, s, L, d_head=128, base=10000.0, alpha=1.0, beta=32.0):
    th = rope_freqs(d_head, base)
    if method == "pi":                                            # Chen et al. 2023: positions m -> m/s
        return th / s
    if method == "ntk":                                           # NTK-aware: raise the base
        return rope_freqs(d_head, base * s ** (d_head / (d_head - 2)))
    if method == "yarn":                                          # Peng et al. 2023 ("NTK-by-parts")
        turns = L * th / (2 * np.pi)                              # rotations within the trained context
        gamma = np.clip((turns - alpha) / (beta - alpha), 0.0, 1.0)
        return (1 - gamma) * th / s + gamma * th
    return th

L, s = 4096, 8                                                    # Llama 2: 4K -> 32K
th = rope_freqs()
turns = L * th / (2 * np.pi)
print("pairs with < 1 turn in L:", int((turns < 1).sum()), "first:", int(np.argmax(turns < 1)))
for m in ("pi", "ntk", "yarn"):
    r = scaled_freqs(m, s, L) / th
    new_angles = (turns < 1) & (s * r > 1 + 1e-9)                 # reach angles never seen in training
    print(f"{m:4s} kept {int(np.isclose(r, 1).sum()):2d}  divided by s {int(np.isclose(r, 1 / s).sum()):2d}  "
          f"extrapolated {int(new_angles.sum()):2d}  pair-0 step {scaled_freqs(m, s, L)[0]:.3f} rad")
print("NTK base:", round(10000 * s ** (128 / 126)), " YaRN logit scale:", round((0.1 * np.log(s) + 1) ** 2, 4))
# pairs with < 1 turn in L: 18 first: 46
# pi   kept  0  divided by s 64  extrapolated  0  pair-0 step 0.125 rad
# ntk  kept  1  divided by s  1  extrapolated 17  pair-0 step 1.000 rad
# yarn kept 21  divided by s 18  extrapolated  0  pair-0 step 1.000 rad
# NTK base: 82685  YaRN logit scale: 1.4591

# Needle-in-a-haystack grid: one fact hidden at a given depth of a filler context.
NEEDLE = "The secret passphrase is 'blue-harbor-42'."
QUESTION = "What is the secret passphrase? Answer with the passphrase only."

def build_prompt(filler_sentences, n_tokens, depth, tokens_per_sentence=12):
    k = n_tokens // tokens_per_sentence                           # rough length control
    hay = (filler_sentences * (k // len(filler_sentences) + 1))[:k]
    hay.insert(int(depth * len(hay)), NEEDLE)                     # depth 0 = start, 1 = end
    return " ".join(hay) + "\n\n" + QUESTION

def niah_grid(ask, filler, lengths, depths):
    """ask(prompt) -> model answer; returns a (len(lengths), len(depths)) 0/1 accuracy grid."""
    return np.array([[float("blue-harbor-42" in ask(build_prompt(filler, n, d))) for d in depths]
                     for n in lengths])

filler = ["The committee met on Tuesday to review the budget.", "Rain is expected later in the week."]
oracle = lambda p: "blue-harbor-42" if NEEDLE in p else "unknown"    # stand-in for a real model call
print(niah_grid(oracle, filler, lengths=[1000, 8000, 32000], depths=[0.0, 0.5, 1.0]).mean())   # 1.0
```

RoPE frequency rescaling for PI, NTK-aware and YaRN with the per-pair classification, plus a needle-in-a-haystack grid harness (run with numpy; the model call is a stand-in)

> **Passing the needle test is not long-context ability:** A single verbatim needle can be found by attending to one surprising sentence. It says little about tracking a variable across 50 pages or aggregating facts spread through a document. Models that pass the plain needle at 128K often fail RULER's multi-hop and aggregation tasks well before that. Report the effective length on harder tasks, and check short-context benchmarks after extension, since a careless recipe trades them away.

> **Frequencies, not positions, are what go out of range:** A RoPE model does not see the integer position. It sees angles. High-frequency pairs wrap within a few tokens and are never out of range at any length. Only the pairs whose wavelength exceeds $L$ are. That is why the good methods treat pairs differently: NTK-aware scaling by geometry, YaRN by an explicit wavelength threshold, LongRoPE by search. Uniform PI over-corrects the fast pairs, which is the local resolution it loses.

> **Train with the scaling you will serve:** The rescaled frequencies are part of the model. A checkpoint fine-tuned with YaRN at $s=16$ must be served with the same factor, temperature and original length. In Hugging Face this is the `rope_scaling` field of the config (types such as `linear` for PI, `dynamic` for dynamic NTK and `yarn`). Serving with a different setting, or with dynamic scaling the model never saw, silently degrades quality without any error.

**Strengths**

- Extension is cheap: about 1000 steps (PI) or a few billion tokens of continued pre-training, against trillions for pre-training
- YaRN-style by-wavelength rescaling keeps local resolution and leaves no frequency out of range
- FlashAttention and ring attention make exact attention at 128K+ feasible in memory
- Sliding windows and local/global mixes bound the KV cache and cost per token

**Limitations**

- Attention FLOPs still grow as T²: about 40% of Llama 2 7B's training compute at 32K
- Windowed and sparse attention can miss information outside the pattern
- Advertised context often exceeds effective context (RULER), and accuracy dips in the middle
- Extension can erode short-context quality unless the data mix is controlled

- **Long documents:** contracts, books, papers and transcripts in one prompt instead of chunked summaries
- **Repository-level code:** Code Llama's 16K training with a raised base was aimed at whole-file and multi-file completion
- **Retrieval-augmented generation:** more retrieved passages per query; lost-in-the-middle argues for reranking and placing the best passages first or last (see **Retrieval-augmented transformers**)
- **Agents and chat:** long tool traces and multi-turn histories, usually combined with KV-cache reuse across turns

**Question:** A RoPE model with d_head = 128 and base 10,000 is trained on 4096 tokens and extended to 32K with position interpolation. Which statement is true?

- A. Only the 18 slowest pairs are rescaled
- B. Every pair is divided by 8, so no angle leaves the trained range, but adjacent tokens are 8× harder to tell apart
- C. The fastest pair is unchanged and the slowest is divided by 8
- D. Interpolation needs no fine-tuning because no angle is new

**Answer:** B. Every pair is divided by 8, so no angle leaves the trained range, but adjacent tokens are 8× harder to tell apart — PI sets $\theta'_i=\theta_i/s$ for every pair, the same as using position $m/8$. Nothing extrapolates, but the fastest pair turns 0.125 rad per token instead of 1. Option C describes NTK-aware scaling. Chen et al. still fine-tuned for about 1000 steps so the model could adapt to the compressed positions.

**Question:** In 'Lost in the Middle', where did multi-document QA accuracy tend to be lowest?

- A. When the relevant document came first
- B. When the relevant document came last
- C. When the relevant document was in the middle of the context
- D. Accuracy was flat across positions for long-context models

**Answer:** C. When the relevant document was in the middle of the context — Accuracy was U-shaped: highest with the answer at the start (primacy) or the end (recency), lowest in the middle. For GPT-3.5-Turbo, middle-position accuracy dropped below its closed-book score. Extended-context variants were not better at using their input.

**Key points**

- RoPE pairs with wavelength $\lambda_i=2\pi b^{2i/d}>L$ never finish a turn in training; at $sL$ they meet unseen angles. For Llama 2 that is pairs 46–63 of 64.
- PI: $\theta_i/s$ for all pairs, safe but loses local resolution. NTK-aware: base $b\,s^{d/(d-2)}$, keeps fast pairs. YaRN: interpolate below 1 turn, keep above 32, plus logits $\times(0.1\ln s+1)^2$.
- Raising the base for long training (Code Llama 10⁶, Llama 2 Long 500,000), LongRoPE's searched factors and ALiBi's linear bias are the other position routes.
- FlashAttention removes the T² memory (not FLOPs); sliding-window and sparse attention cut work to O(T·W); ring attention splits one sequence over P devices.
- Long-context ability needs long, domain-balanced training data and real tests: needle-in-a-haystack grids, RULER, and the lost-in-the-middle U curve.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
