Attention Sinks & KV-Cache Compression

The eviction/sinks deep-dive. This is one lane of the broader picture in [[kv-cache-optimization]] — that umbrella covers all three axes of shrinking the cache (architecture GQA/MQA/MLA · eviction · quantization KIVI). Here we go deep on just the eviction & attention-sink lane; for the architectural and quantization levers, see the umbrella.

For NAO + anyone who keeps a long chat with Alice running for hours and wonders why it doesn't fall apart — or why "200K context" costs what it costs. This is the inference-time, memory side of long context. [[long-context]] handles two walls — legalising unseen positions (RoPE scaling / YaRN) and the exact-attention compute wall (Ring Attention). This article is about the third, quieter wall that bites at serving time: the KV cache — the pile of per-token Key/Value vectors that grows with every token and pins precious GPU memory. We cover the one weird mechanistic discovery — attention sinks — that lets a fixed-window model stream forever, and the family of eviction / compression tricks (H2O, SnapKV) that shrink the cache 5–20× with little quality loss.

What it is (intuition first)

When a transformer generates text autoregressively, it does not re-read the whole prompt for every new token. That would be O(N²) per step. Instead it caches, for every token it has already processed, that token's Key and Value vectors at every layer — the KV cache. Generating token N+1 then costs only one new query attending over the cached N keys/values. This is the single most important inference optimisation in all of LLM serving (see [[llm-inference-internals]]).

The catch: the KV cache grows linearly with sequence length and batch size, and it lives in GPU memory the entire time. For a long conversation or a 1M-token document, the cache — not the model weights, not the FLOPs — becomes the binding constraint. Two distinct problems fall out of this:

Streaming / unbounded generation. What if you want Alice to chat for days without ever resetting? You can't keep growing the cache forever. The obvious fix — a sliding window that just drops the oldest tokens — catastrophically fails: perplexity explodes the moment the first few tokens scroll out. The surprising reason why, and the dead-simple fix, is attention sinks (StreamingLLM).

Cache too big even for a bounded context. Even at a fixed 32K or 128K prompt, the KV cache can be many gigabytes. Can we keep only the tokens that matter and throw the rest away? Yes — that's KV-cache eviction / compression (H2O, SnapKV, and friends), built on the empirical fact that attention is extremely sparse: a few tokens hoard most of the attention mass.

Hold those two threads — stream forever and shrink the cache — they are the whole article. Both exploit the same underlying truth: attention does not spread evenly; it concentrates.

Why it matters

It's what makes long-running agents physically possible. A coding agent, a multi-day Telegram thread with Alice, a streaming session — all generate unbounded token streams. Without a streaming strategy, you either reset (lose context) or OOM. StreamingLLM-style attention sinks are the mechanism that lets a finite-window model serve an infinite stream.
KV-cache memory, not compute, is the real serving cost at long context. Once your prompt is long, the GPU memory pinned by the cache caps how many requests you can batch — which directly sets your $/token. Halving the cache can double throughput. This is why every serving stack (vLLM, SGLang, TensorRT-LLM) has cache-management machinery.
It exposes a deep, surprising fact about transformers. Attention sinks were a genuine discovery — nobody designed them; they emerged from training and were reverse-engineered. They connect serving efficiency to [[mechanistic-interpretability]] (massive activations, the softmax "no-op") and even changed how new models are pretrained (a dedicated sink token).
It's the cheap complement to the expensive long-context methods. Ring Attention buys exact million-token attention with a rack of GPUs. Attention sinks + eviction buy most of the practical benefit on one GPU by being approximate but smart. Most real deployments live here, not on the TPU pod.

How it works (the real mechanics)

Part 1 — The naive baselines, and why they fail

You have a model trained with attention window L (say 4K) and you want to generate token after token past L. Two obvious approaches:

Dense / re-cache (window recomputation). Keep the most recent L tokens and recompute their KV from scratch each time the window slides. Quality is fine, but it's slow — you redo work constantly. StreamingLLM reports their approach is up to 22.2× faster than this sliding-window-with-recomputation baseline.
Sliding-window KV cache (evict oldest). Keep a rolling cache of the last L KV pairs; when a new token arrives, drop the oldest. Cheap and obvious. And it breaks spectacularly: the instant the very first tokens of the sequence get evicted, perplexity shoots up and generation degenerates — even though those first tokens are usually semantically meaningless (a <bos>, "The", a newline). That paradox is the whole clue.

Part 2 — Attention sinks: the discovery (StreamingLLM)

Xiao et al. (2023) visualised the attention maps and found the smoking gun: in a trained LLM, a large fraction of attention mass is dumped onto the first few tokens of the sequence, regardless of whether those tokens are relevant. They named these positions attention sinks.

Why do sinks exist? (the honest, mechanistic answer.) It comes down to softmax. Attention weights are produced by a softmax, which forces them to sum to exactly 1 — there is no "attend to nothing" option. The MIT Han Lab write-up calls this a "deafening democracy where abstention is disallowed." When a head, at some layer, has no token it actually wants to look at, it still has a full unit of attention probability it is required to spend. So the model learns a trick: park that leftover, "no-op" attention on a few fixed, always-available positions. The first tokens are the natural choice because, in a causal/autoregressive model, they are visible to *every* later token — they're the one set of positions reliably present in every context. A faint initial bias gets amplified through training into a dedicated dumping ground.

This is the same family of phenomena as massive activations seen in interpretability work ([[mechanistic-interpretability]]): a few hidden-state dimensions / token positions carry enormous magnitude and serve a structural, not semantic, role.

The fix is almost insultingly simple. If the first tokens are load-bearing as sinks, then never evict them. StreamingLLM keeps a tiny number of initial sink tokens permanently pinned in the cache, plus a normal sliding window of recent tokens for everything else:

KV cache layout  =  [ 4 sink tokens (frozen) ] + [ ~1020 most-recent tokens (rolling) ]
                     ^ never evicted             ^ FIFO: drop oldest as new arrive

A standard config is a 1024-token cache: 4 permanent sinks + 1020 recent. With this, a model trained at finite length streams stably over 4 million+ tokens with no fine-tuning — perplexity stays flat where the plain sliding window blew up. Crucially, the sinks are about position-zero presence, not content: keeping the first 4 tokens' KV works; keeping 4 arbitrary recent extra tokens does not.

The one-paragraph mental model: softmax can't abstain, so heads dump spare attention on the ever-present first tokens. Evict those and every head's softmax suddenly has to re-normalise its "no-op" mass onto real tokens → garbage. Pin the sinks, slide the rest, stream forever.

Part 3 — Designing sinks in: the dedicated sink token

StreamingLLM also asked: if vanilla models improvise sinks out of content tokens, what if we give them a purpose-built one? They added a single learnable [SINK] placeholder token to every training sample during pretraining. Result: a model trained this way needs only one dedicated sink at inference, versus four content tokens for a vanilla model — and it stops "hijacking" semantically meaningful initial tokens as sinks. This is why several frontier models now ship a dedicated attention-sink token (or a learned per-head sink bias) by design — the discovery fed back into architecture. (Treat the specific "which 2026 models ship it" claims as vendor-stated unless you've seen the weights/config; the mechanism is well-established, the per-model adoption details are not always public.)

Part 4 — KV-cache eviction: keep only the heavy hitters (H2O)

StreamingLLM keeps a positional skeleton (sinks + recent). A different question: within a bounded context, can we keep the content-important tokens and drop the rest? Zhang et al. (2023), H2O (Heavy-Hitter Oracle), start from a second empirical fact: attention is sparse — a small set of tokens, the "heavy hitters" (H₂), account for most of the attention mass, and they correlate with frequently co-occurring tokens.

H2O frames cache eviction as keeping a balance of (a) recent tokens and (b) heavy hitters, scored greedily by accumulated attention each token has received so far. Under memory pressure, evict the lowest-scoring non-recent tokens. They prove the greedy policy is a (near-optimal) dynamic submodular problem, so it has theoretical footing rather than being pure heuristic.

Reported results (from the paper — treat as the paper's claims on its hardware):

With roughly 20% of the KV cache retained, accuracy is largely preserved across OPT, LLaMA, and GPT-NeoX on diverse tasks.
Up to 29× throughput vs. DeepSpeed Zero-Inference and HuggingFace Accelerate, and 3× vs. FlexGen, on OPT-6.7B / OPT-30B.
Up to 1.9× latency reduction at matched batch size.

Part 5 — Prompt-time compression: SnapKV

H2O evicts during generation. SnapKV (Li et al., 2024) attacks the other big cost — a huge prompt whose KV you must store before you even start generating. The observation: each attention head consistently focuses on the same specific prompt positions during generation, and you can predict which ones from a small "observation window" at the end of the prompt. SnapKV uses that window to vote, per head, on which earlier KV positions matter, clusters and pools them (so it keeps coherent spans, not scattered tokens), and keeps only those — fine-tuning-free.

Reported results:

3.6× faster generation and 8.2× better memory efficiency at 16K-token inputs, with comparable accuracy across 16 long-sequence datasets.
Enables processing up to 380K tokens on a single A100-80GB GPU with "negligible accuracy drop," validated via Needle-in-a-Haystack.

Part 6 — Where this sits relative to the other long-context tricks

Method	Problem solved	Core trick	Exact or approx?
RoPE scaling / YaRN ([[positional-encoding]], [[long-context]])	unseen positions	remap/interpolate RoPE frequencies	exact attention, new positions
Ring Attention ([[long-context]])	O(N²) compute across devices	rotate KV blocks around a ring	exact
[[flash-attention]]	O(N²) memory on one device	blockwise online softmax	exact
StreamingLLM (sinks)	unbounded streaming	pin sink tokens + sliding window	approx (drops middle)
H2O	cache too big (decode)	evict by accumulated-attention	approx (drops light tokens)
SnapKV	cache too big (prompt)	per-head pooled selection via obs. window	approx
MLA ([[deepseek-architecture]])	cache footprint per token	low-rank latent KV projection	exact-ish (architectural)

The clean split: the first three are exact and pay in math/hardware; the sink/eviction/compression family is approximate and pays in the risk of dropping something you needed. They compose — you can run YaRN-extended positions, FlashAttention kernels, and H2O eviction together.

Key ideas & tradeoffs

Sinks are a *positional* skeleton; eviction is a *content* filter. Both exploit attention sparsity, but differently: StreamingLLM keeps tokens for where they are (position 0–3), H2O/SnapKV keep tokens for how much attention they got. The state of the art often combines: keep sinks and heavy hitters and the recent window.
Approximate by construction — you *will* drop the middle. A pure sliding window + sinks model has, by design, no access to tokens that scrolled out. StreamingLLM is explicit about this: it gives you stable streaming, not long-range recall. If the answer is in token 1M and it's been evicted, it's gone. This is a different failure mode than lost-in-the-middle ([[long-context]]) — here the middle is literally absent, not merely under-attended.
Eviction is not the only axis. Two orthogonal levers shrink the same cache and compose with everything here — architectural reduction (GQA/MQA/MLA, baked at pretraining) and quantization (store K/V in fewer bits, KIVI). Both are covered in the [[kv-cache-optimization]] umbrella; this article stays in the eviction/sinks lane.
Sink count is small and saturating. ~4 sinks for vanilla models, ~1 for a model pretrained with a [SINK] token. More sinks don't keep helping; it's a structural slot, not a knob you scale.

Honest caveats & open questions

Streaming ≠ recall, and this is constantly mis-sold. "StreamingLLM gives infinite context" is the dangerous shorthand. It gives infinite *generation with a **bounded effective memory** (the window). It does not* let the model recall arbitrary earlier tokens — those were evicted. For genuine long-range recall you still need a longer real window, retrieval ([[graphrag]]), or external [[agent-memory-systems|memory]].
Eviction can drop the token you needed. H2O/SnapKV bet that accumulated attention (past) predicts future importance. Usually true; not always. Adversarial or "delayed-relevance" inputs — a token nobody attended to early that becomes critical late — are exactly where greedy eviction can fail. The benchmarks (LongBench, NIAH, the 16 datasets SnapKV cites) say "negligible drop" on those distributions; your workload may differ.
The numbers are paper-reported, on the authors' hardware/models. The 22.2×, 29×, 3.6×, 8.2× figures above come from the respective papers (mostly 2023–2024, OPT/LLaMA-2-era models). Treat them as order-of-magnitude evidence the method works, not as guarantees for gpt-5.x-class models or your serving stack. I have not independently reproduced them.
Why sinks exist is well-supported but not fully closed. The softmax-can't- abstain / no-op-dump story (amplified by the "first tokens are always visible" argument) is the consensus mechanistic explanation and is backed by ablations (zeroing sinks, the [SINK]-token experiment). But the precise interaction with massive activations, LayerNorm, and per-head specialisation is still active [[mechanistic-interpretability]] territory.
"Frontier model X ships a sink token" is often vendor/inference-detail. The idea is published and adopted; which specific 2026 models ship a dedicated sink, and how, is not always disclosed. Flagged as such — don't state it as fact without the config.
Interaction with RoPE position IDs. When you pin sinks + slide a window, you must decide what position the kept tokens carry (StreamingLLM assigns positions within the cache, not their original absolute positions). Get this wrong and you re-introduce the position-extrapolation failure ([[positional-encoding]]). It's a real implementation gotcha, not a free lunch.

How it connects to OpenAlice

Alice's long-running chats are exactly the streaming case. A multi-day Telegram thread or a streaming session is an unbounded token stream — the StreamingLLM scenario. Whether Alice's provider uses sinks under the hood is the provider's concern, but the consequence shapes our design: a model that streams stably is not the same as one that recalls everything it streamed. That distinction is why Alice has durable [[agent-memory-systems|memory]] + retrieval, not just a big window — the window can be streaming-bounded even when it's advertised as huge.
It's why "just use a bigger window" is rarely the cheapest correct answer. KV-cache memory caps batch size caps $/token. For the cost-ladder rig and for serving Alice, retrieving the relevant slice (via Atlas: atlas.code.search, semantic search) and reading a short context is cheaper and often better than streaming a giant one — the same retrieve-then-read thesis as [[graphrag]] and [[long-context]], now justified from the memory side rather than the accuracy side.
Composable with everything we route to. When [[model-routing]] picks an open model, its KV strategy (MLA vs. vanilla, native sinks vs. none) is part of why its long-context cost/quality differs — useful when choosing which model eats a 100K-token transcript vs. a short chat.
A foundation node for the inference track. This article is the memory half of inference; [[llm-inference-internals]] is the serving-systems half (PagedAttention, RadixAttention prefix-sharing, continuous batching) and [[speculative-decoding]] is the latency half. Together they're "how an LLM is actually served," downstream of the architecture nodes ([[attention-and-transformers]], [[flash-attention]]).

Attention Sinks & KV-Cache Compression (StreamingLLM, H2O, SnapKV)