Long-Context LLMs

For NAO + anyone who keeps hearing "1 million token context" and wonders what that actually buys you. A context window is just how many tokens the model can attend to at once. Making it bigger sounds trivial — "let it read more" — but it runs into two hard walls: a math wall (the model was trained on positions it has never seen) and a compute wall (attention cost grows with the square of the sequence length). This article is about the tricks that knock both walls down — and the uncomfortable evidence that even when you do, the model often doesn't actually use the middle of that giant window.

What it is (intuition first)

A transformer reads a sequence of tokens and, at every layer, lets every token "look at" every other token via [[attention-and-transformers|attention]]. The context window is the maximum number of tokens it can hold in that view at once. GPT-2 (2019) had 1,024. Llama-2 (2023) had 4,096. By 2026, Gemini ships a 1M–2M token window, GPT-5.x and Anthropic's Claude family offer 1M+ token windows (check current docs for exact per-model limits).

Why isn't a long window free? Two reasons:

1. The model never saw those positions during training. A model trained on sequences of length 4,096 has learned what "position 1," "position 100," "position 4,000" feel like. Ask it about token 8,000 and the positional signal is out-of-distribution — like a ruler that only goes to 4,096 cm suddenly being asked to measure 8 metres. Outputs degenerate. Fixing this without retraining from scratch is context extension (PI / NTK / YaRN — they remap positions the model has seen onto a longer sequence).

1. Attention is O(N²) in sequence length. Doubling the context quadruples the attention compute and the size of the attention matrix. For a million tokens, the naive N×N score matrix is astronomically large and won't fit on any single accelerator. Fixing this is a systems problem — solved by [[flash-attention|FlashAttention]]-style tiling and, across many devices, by Ring Attention.

1. And the cruel twist: even after you fix (1) and (2), the model frequently ignores information buried in the middle of the window. This is lost-in-the-middle, and "I have a 1M window" tells you almost nothing about whether the model can use it — which is what benchmarks like RULER were built to expose.

Hold those three threads — math, systems, does-it-actually-work — they are the whole article.

Why it matters

• Whole documents, codebases, conversations in one shot. Long context is the cheap alternative to retrieval: instead of chunking + embedding + a vector DB ([[embeddings]], [[graphrag]]), you stuff the whole thing in. For OpenAlice / Alice this is the difference between "Alice remembers this chat" and "Alice re-derives memory from a RAG store" (see [[agent-memory-systems]]).
• Agents live or die on it. A coding agent reading a repo, a research agent reading 30 papers, Alice tracking a long Telegram thread — all consume context fast. The effective window is the agent's working memory.
• It reshapes the RAG-vs-context tradeoff. A genuinely reliable 1M window would kill a lot of retrieval engineering. The RULER results below are exactly why retrieval is not dead.
• Cost scales with context. You pay per input token, and attention compute is super-linear. A 200K-token prompt is not 50× a 4K prompt in price — and the KV cache memory it pins on the GPU is a real serving constraint.

How it works (real mechanics)

Part 0 — The thing we're stretching: RoPE

Almost all modern context extension operates on Rotary Position Embedding (RoPE) — the dominant positional scheme since Llama. (Full treatment in [[positional-encoding]]; the short version follows so this article stands alone.)

RoPE splits each head's dimension |D| into pairs and rotates each pair by an angle proportional to the token's absolute position m. Each dimension-pair d rotates at its own frequency:

θ_d = b^(-2d/|D|)        with base b = 10000

Low d → high frequency (rotates fast, encodes fine/local position). High d → low frequency (rotates slowly, encodes coarse/global position). The beautiful property: the dot product between a query at position m and a key at position n depends only on the relative offset m − n, with a natural long-range decay. The wavelength of dimension d — how many tokens before that pair completes a full 2π rotation — is:

λ_d = 2π / θ_d = 2π · b^(2d/|D|)

This λ_d is the load-bearing quantity for everything below. Some dimensions complete many rotations inside the training window (high frequency, short λ); some never even finish one rotation (low frequency, long λ). They must be treated differently.

Part 1 — Context extension (the math wall)

The goal: take a model trained to length L and make it work at L' > L, ideally with little or no fine-tuning. Define the scale factor:

s = L' / L

(a) Position Interpolation (PI) — Chen et al., 2023. The simplest idea: don't extrapolate to unseen positions, interpolate into seen ones. Squash every position by s so the longest new position L' maps back to the familiar L:

g(m) = m / s          h(θ_d) = θ_d   (frequencies unchanged)

A token at position 8,000 in an s=2 extension is rotated as if it were at position 4,000 — a position the model knows. Works, but it crushes the high-frequency dimensions: fine-grained local position resolution gets blurred by the same factor s, hurting tasks that need precise nearby-token ordering. PI needs fine-tuning (~1000 steps) to recover.

(b) NTK-aware interpolation. The insight from the Neural Tangent Kernel literature: don't scale all dimensions equally. Instead of interpolating positions, change the base so that high frequencies are barely touched and low frequencies are interpolated hard. YaRN writes the NTK-aware base as:

b' = b · s^(|D|/(|D|-2))

This spreads the "interpolation pressure" non-uniformly across dimensions: high-frequency (local) dimensions stay near-extrapolated (preserving local resolution), low-frequency (global) dimensions get interpolated. It can extend context without fine-tuning for modest s — but at large s it over-extrapolates the very highest frequencies and starts to fail.

(c) NTK-by-parts — the key refinement, and the heart of YaRN. Decide per dimension whether to interpolate, based on how many times that dimension's wavelength fits inside the original window. Define the ratio:

r(d) = L / λ_d

• If r(d) > β (wavelength is short — many rotations fit in L): the model has seen the full periodic behaviour of this dimension already. Don't interpolate — leave it alone.
• If r(d) < α (wavelength is long — less than one rotation fits in L): the model only ever saw a fraction of a turn. Interpolate fully (PI-style), because extrapolating an unfinished rotation is exactly what blows up.
• In between: ramp smoothly with a linear blend:

γ(r) = 0            if r < α
γ(r) = 1            if r > β
γ(r) = (r − α)/(β − α)   otherwise

YaRN's recommended thresholds for Llama-family models: α = 1, β = 32. The effective per-dimension frequency is then a γ-weighted mix of the interpolated and non-interpolated frequency. This is what makes YaRN work across many dimensions cleanly where naive NTK fails.

(d) Attention temperature (the YaRN finishing move). Extending context spreads attention probability mass thinner over more tokens, flattening the softmax distribution and raising its entropy. YaRN counteracts this by scaling the pre-softmax logits by a temperature 1/t — and the slick part is they fold it into the existing RoPE rotation (so it costs zero extra code or inference overhead). The empirically fitted factor:

√(1/t) = 0.1 · ln(s) + 1

i.e. for a 16× extension you scale logits by a small, log-growing factor that sharpens attention back toward where it was at training length.

Why YaRN is the one to remember: combining NTK-by-parts + temperature, YaRN extended Llama-style models to 64K and 128K context while requiring ~10× fewer tokens and ~2.5× fewer training steps than PI to reach comparable quality. It became the default open-weights extension recipe.

One-line mental model of all four: PI squashes everyone equally (blurs local detail); NTK-aware squashes the global dims more than local ones via the base; NTK-by-parts decides per-dimension using r(d) = L/λ_d; YaRN adds a temperature so the softmax doesn't go flat. Same RoPE knobs, increasingly surgical use of them.

Part 2 — Ring Attention (the compute wall)

Context extension makes the positions legal; it does nothing about the O(N²) attention cost and the O(N²) score matrix that won't fit in one device's memory. For million-token contexts you must shard the sequence across devices — without ever materialising the full matrix.

[[flash-attention|FlashAttention]] already solved the single-device version: process attention in blocks using an online (streaming) softmax, so you never store the full N×N matrix, only running statistics. Ring Attention (Liu et al., 2023) extends that to many devices arranged in a logical ring:

Setup:   split the N-token sequence into N blocks across N hosts.
         Host i permanently owns query block Q_i.
         Host i starts holding KV block (K_i, V_i).

Inner loop (repeated N times, once per KV block in the sequence):
  for step in 1..N:
    # 1. compute: blockwise attention of MY query Q_i against the
    #    KV block currently in my hands, accumulating with online softmax
    accumulate_blockwise_attention(Q_i, K_cur, V_cur)
    # 2. communicate: send my KV block to the NEXT host in the ring,
    #    receive a new KV block from the PREVIOUS host
    K_cur, V_cur = ring_rotate(K_cur, V_cur)   # overlapped with step 1

After N steps every query block has seen every KV block — full (exact, not approximate) attention — but no host ever held more than one KV block at a time. The genius is overlap: while a host computes attention on the block it holds, it is simultaneously shipping that block onward and pulling in the next. The overlap is free as long as block compute time ≥ block transfer time, which gives the condition on block size c ≥ F/B (FLOPs/Bandwidth ratio) — in practice ~1K tokens/block on A100 NVLink or TPUv4, needing roughly 6–10K tokens per device to fully hide communication.

The payoffs reported:

• Memory per device becomes independent of total sequence length (≈ 6bch bytes/layer for batch b, block c, hidden h) instead of O(N²).
• Context scales linearly with device count. 32× A100 GPUs → ~4M-token context; a TPUv4-1024 pod → ~8M tokens for a 7B model — without approximation, at MFU comparable to standard attention.

So: Ring Attention is to multi-device what FlashAttention is to single-device — the same blockwise online-softmax trick, with KV blocks rotating around a ring so each device's memory footprint stays flat.

Part 3 — Does it actually work? Lost-in-the-middle & the evals

Now the honest part. Suppose you've extended positions (Part 1) and you can physically run the long context (Part 2). Can the model use it?

Lost in the Middle (Liu et al., 2023) is the foundational negative result. Setup: multi-document QA with 10 / 20 / 30 documents where exactly one holds the answer, plus a synthetic key-value retrieval task over 75 / 140 / 300 UUID pairs (no linguistic confounds). They slide the position of the relevant document/key through the context and measure accuracy. The finding is a U-shaped curve: accuracy is highest when the relevant info is at the very start or very end, and drops sharply in the middle.

Concrete numbers, GPT-3.5-Turbo, 20 documents:

That's an ~18.6 percentage-point swing purely from where the answer sits. The most damning detail: middle-of-context accuracy (53.8%) fell below the closed-book baseline (56.1%) — i.e., the model did worse with the relevant document present-but-buried than with no documents at all. And crucially: extended-context models were "not necessarily better at using input context" — when a 10–20 doc context fit in both a base and an extended-window version, their curves were nearly superimposed. A bigger window did not fix the middle.

Needle-in-a-Haystack (NIAH) is the popular sanity check: hide one fact (the "needle") at a random depth in a long filler text (the "haystack") and ask the model to retrieve it, sweeping depth × length to produce a heatmap. It's a great smoke test — but it's too easy: a single verbatim string in semantically unrelated filler is the best case. Passing NIAH says little about real use.

RULER (Hsieh et al., 2024) is the grown-up benchmark and the honest yardstick. It extends NIAH into four task families — retrieval (single / multi-key / multi-value / multi-query needles), multi-hop tracing, aggregation, and QA — all configurable from 4K to 128K. Its headline:

Despite near-perfect scores on vanilla NIAH, almost all models drop sharply as context grows. Of models claiming ≥32K context, only about half maintain satisfactory performance at 32K. A model's advertised window is not its usable window.

This gave the field the term effective context length: the longest length at which a model still clears a strong baseline (RULER anchors to Llama2-7B's quality at 4K). The gap between claimed and effective is routinely 2–8×. "200K token model" frequently means "reliable to maybe 32–64K."

Key ideas & tradeoffs

Cross-cutting tradeoffs worth internalising:

• Extension is not free fidelity. Every extension method trades positional precision for reach. YaRN minimises the trade but doesn't abolish it.
• Exact long attention vs. approximate cheap attention. Ring Attention is exact and pays in hardware (devices, bandwidth). Sparse/linear attention and [[state-space-models|state-space models (Mamba)]] are cheap and pay in expressivity — SSMs swap quadratic attention for a recurrent O(N) scan but must compress all history into a fixed state, which is its own lost-in-the-middle.
• Long context vs. retrieval. The RULER gap is the empirical case for retrieval. In practice the winning pattern is a hybrid: retrieve to shrink the haystack, then let a long window read the survivors — which is much of what [[graphrag]] formalises.
• KV-cache memory is the real serving cost. At inference, every prior token's K and V stay pinned in GPU memory. Long contexts make the KV cache, not FLOPs, the binding constraint — hence the cottage industry of KV-cache eviction / compression (and why [[quantization]] of the cache matters).

Honest caveats & open questions

• Claimed ≫ effective, still, in 2026. Frontier vendors now advertise 1M–2M windows, but independent RULER-style evaluation of that full length is sparse and usually behind the marketing. Treat any "N-token context" claim as an upper bound on capacity, not a guarantee of usable reasoning. Believe benchmarks, not spec sheets.
• Lost-in-the-middle is mitigated, not solved. Newer models flatten the U-curve compared to GPT-3.5, but a recency/primacy bias persists. Practical consequence that survives to today: put the most important content at the top or bottom of a long prompt, never buried in the middle.
• The numbers above are from the original 2023/2024 papers. They're the canonical mechanism references; absolute accuracies on current frontier models are higher. The shape of the problem (U-curve, claimed≫effective) has proven durable; the magnitudes move.
• YaRN's constants are empirical. α=1, β=32, and √(1/t)=0.1·ln(s)+1 were fitted for Llama-family models. They are heuristics, not derived from first principles, and transfer imperfectly to other architectures/tokenizers.
• Ring Attention's overlap assumption can break. If block compute doesn't exceed transfer time (small blocks, slow interconnect, tiny models), the communication stops being free and you eat latency. It shines on big models + fat interconnects, less so on commodity setups.
• Position-extension methods only legalise positions; they don't teach long-range reasoning. A model can attend to token 500K and still be unable to integrate a fact there with one at token 5K. Retrieval depth ≠ reasoning depth — multi-hop tracing and aggregation (RULER's harder tracks) degrade well before single-needle retrieval does.
• Open question — is the future long windows or smart memory? Long context vs. external [[agent-memory-systems|memory]] / retrieval is an unsettled architectural bet. SSMs and hybrids gamble that you don't need to keep everything if you compress well; the long-context camp gambles that attention over everything wins. Both are alive in 2026.

How it connects to OpenAlice

• Alice's working memory is a context-window problem. A long Telegram or streaming thread, a coding task spanning many files — these are exactly the multi-document setups where lost-in-the-middle bites. The practical takeaway is wired into how prompts are composed: the soul/identity and the live task go at the edges of the prompt, not the middle (consistent with the prompt-composition discipline in Alice's prompt studio). This is positional hygiene, not personality change.
• RAG-vs-context is a live design axis here. Atlas itself is the retrieval layer for the ecosystem: instead of pasting 60 repos into a million-token window and praying, agents query Atlas (atlas.code.search, semantic search, callers/callees) to retrieve the relevant slice first, then reason over it. That is the hybrid the RULER results recommend — and it is precisely the [[graphrag]] thesis: structure + retrieval beats brute-force context.
• Agent memory ≠ context window. [[agent-memory-systems]] persists across sessions; the context window is per-call working memory. Long-context tricks expand the working set; they don't replace durable memory. Alice needs both.
• Cost discipline. Long prompts are expensive in tokens and KV-cache memory. For the cost-ladder research rig and for Alice serving, "just use a bigger window" is rarely the cheapest correct answer — retrieve, then read.
• It rests on the same primitives we already document. Context extension is pure [[positional-encoding]] surgery (RoPE/PI/NTK/YaRN). The compute side is [[flash-attention]] generalised to a ring. The "is it real?" side is benchmark honesty. Understanding long context = understanding those three articles together.

See also

[[positional-encoding]] · [[attention-and-transformers]] · [[flash-attention]] · [[graphrag]] · [[embeddings]] · [[agent-memory-systems]] · [[state-space-models]] · [[quantization]] · [[test-time-compute-reasoning]]