KV-Cache Optimization (compression, eviction, quantization)

One-line summary. During decoding, every prior token's Key and Value vectors stay pinned in GPU memory so the model doesn't recompute them. That KV cache grows linearly with sequence length × batch size and, on long contexts or many concurrent users, becomes the binding constraint — not FLOPs. KV-cache optimization is the family of tricks that shrinks it: architectural (fewer KV heads — GQA/MQA — or low-rank latents — MLA), eviction/selection (keep only the tokens that matter — attention sinks, H2O, SnapKV), and quantization (store K/V in 2–4 bits — KIVI). The prize is more concurrent sequences and longer contexts on the same hardware; the risk is silently dropping the one token the answer needed.

What it is (intuition first)

When an autoregressive LLM generates token N, attention makes it look back over tokens 1…N−1. Recomputing the Keys and Values for all those past tokens on every step would be quadratic and brutal, so the model caches them: each token's K and V, at every layer and head, are computed once and kept. Token N then only computes its own new K/V and attends against the stored rest. This is the KV cache, and it is what makes decode O(N) per step instead of O(N²). (The K/Q/V mechanics live in [[attention-and-transformers]]; the serving systems that hold the cache live in [[llm-inference-internals]].)

The catch is memory. The cache size is roughly:

KV_bytes ≈ 2 (K and V) · L_layers · H_kv (heads) · d_head · L_seq · batch · bytes_per_elem

The numbers get large fast. From the vLLM/Anyscale measurements quoted in [[llm-inference-internals]], a single LLaMA-13B sequence can pin ~1.7 GB of KV cache, growing ~1 MB per token. Decode is memory-bandwidth-bound: each step the GPU mostly loads bytes (weights + KV) rather than doing math, so throughput is set by how big a batch you can fit, which is set by how much KV fits. Therefore shrinking the KV cache is almost directly shrinking your serving cost and unlocking longer context. That is the entire motivation.

Think of it as a working memory you must carry the whole conversation. You can: (1) carry fewer notebooks (architectural — GQA/MLA reduce how much you write per token), (2) throw away pages you'll never re-read (eviction — but you can't know the future, so you guess via attention), or (3) write smaller (quantization — fewer bits per number). The three are largely orthogonal and stack.

Why it matters

Decode is memory-bound, and KV is the memory. Unlike training (compute-bound) or prefill (compute-bound), token-by-token decoding is gated by bandwidth and capacity. KV cache is the part of that footprint that grows with your workload (long chats, big batches), so it's the most actionable lever after weight quantization.
It's what makes long context affordable. A 128K-token context with a vanilla multi-head model would pin tens of GB of KV per sequence. Every frontier long-context model ([[long-context]]) ships some KV reduction — it's not optional at that scale.
It's the lever for many concurrent users. Because batch size (and thus throughput, [[llm-inference-internals]]) is capped by KV footprint, halving the cache can roughly double the batch — a near-linear throughput win on memory-bound decode.
It composes with the rest of the stack. Architectural KV reduction is baked at pretraining; eviction and KV-quantization are inference-time and stack on top of PagedAttention, [[flash-attention]], weight [[quantization]], and even [[speculative-decoding]]. None of them conflict.

How it works (the real mechanics)

There are three families. The survey by Li et al. (2024) organizes them as token-level (selection/eviction, quantization, merging), model-level (architecture), and system-level (paging/offload — covered in [[llm-inference-internals]]). We focus on the first two — the algorithmic frontier.

Family 1 — Architectural: fewer KV heads (MQA → GQA → MLA)

The cheapest KV is the KV you never store separately. Standard Multi-Head Attention (MHA) gives every query head its own K and V head. Two ideas exploit redundancy across heads:

Multi-Query Attention (MQA) — all query heads share a single K/V head. This shrinks the KV cache by the number of heads (e.g. 32× for a 32-head model) and speeds decode dramatically, but loses quality.
Grouped-Query Attention (GQA) (Ainslie et al. 2023) — the now-standard middle ground. Query heads are partitioned into G groups; heads within a group share one K/V head. So KV scales with G (e.g. 8) instead of the full head count. The paper shows you can "uptrain" an existing MHA checkpoint into GQA using ~5% of the original pretraining compute, reaching quality close to MHA at speed close to MQA. GQA is why Llama-2/3-class models serve long context cheaply.

Multi-head Latent Attention (MLA) (DeepSeek-V2, 2024) goes further: instead of storing per-head K/V, it stores a low-rank joint latent vector per token and reconstructs K/V on the fly via learned up-projection matrices. The cache holds the small latent, not the full K/V. DeepSeek reports MLA both reduces KV cache significantly and matches/beats MHA quality — a rare both-ways win — and it's what lets DeepSeek-V2 (236B total / 21B active MoE) serve 128K context economically. See [[deepseek-architecture]] for the full design. (Caveat: MLA interacts subtly with [[positional-encoding|RoPE]]; DeepSeek splits a "decoupled RoPE" dimension to make the two compatible — a real engineering wrinkle, not free.)

The tradeoff for this whole family: it must be decided at pretraining time (or via the 5%-compute uptrain). You can't bolt GQA/MLA onto a deployed MHA model at inference.

Family 2 — Eviction & selection: keep only the tokens that matter

If you can't change the architecture, drop KV entries you won't need. The core problem: you can't see the future, so you predict importance from attention patterns — exploiting the fact that attention is sparse (a few tokens hoard most of the mass). The landmark methods are attention sinks / StreamingLLM (pin the first "sink" tokens + slide a recent window), H2O (keep the high-attention "heavy hitters" + recent), and SnapKV (compress the prompt KV before generation via an observation window). The unifying caveat: eviction is a bet on attention as an importance proxy that can quietly drop a token a later reasoning step needed.

Deep-dive deferred. The mechanics of all three — why sinks exist (the softmax "no-op" dump), H2O's heavy-hitter scoring, SnapKV's per-head pooled selection, the reported speedups, and the exact-vs-approximate map against Ring Attention / YaRN — live in [[attention-sinks-and-kv-cache]]. This umbrella stays at the level of "eviction is one of the three axes"; go there for the full treatment.

Family 3 — Quantization: store K/V in fewer bits

Orthogonal to which tokens you keep is how precisely you store each one. KV quantization shrinks bytes-per-element from FP16 (16 bits) toward INT4/INT2.

KIVI (Liu et al. 2024) is the canonical tuning-free 2-bit method. Its key empirical finding is an asymmetry: the Key cache should be quantized per-channel (group elements along the channel/feature dimension), while the Value cache should be quantized per-token. This matches where outliers live in each tensor. Tuning-free (no retraining), KIVI reports ~2.6× less peak memory (including weights), enabling up to 4× larger batch and 2.35×–3.47× throughput on real workloads while keeping quality nearly intact. See [[quantization]] for the general theory of per-channel vs per-token scaling and outliers.

The tradeoff: KV quantization is the most general (works on any model, stacks with everything) but pushing below ~4 bits, or onto very long contexts, can degrade quality — and the accuracy/throughput frontier at 2-bit is, honestly, still being mapped. The 2024 numbers above are vendor/author-reported on specific models; re-benchmark on yours.

How they compose

These stack multiplicatively. A realistic modern long-context serving config is: GQA or MLA (architectural, baked in) + PagedAttention (no fragmentation, [[llm-inference-internals]]) + optional eviction for extreme lengths + optional KV quantization for capacity. Each attacks a different axis of KV_bytes: heads, fragmentation, sequence length, and bits-per-element.

Key ideas & tradeoffs

The KV cache is the workload-dependent part of memory. Weights are fixed; KV grows with your context length and concurrency. That's why it's the highest-leverage inference optimization for chat/agent/RAG workloads with long, variable prompts.
Three orthogonal axes, one budget. Heads (architecture), tokens (eviction), bits (quantization). You can spend savings on longer context or more batch — your call, same memory budget.
Architectural reduction is "free" at serve time but expensive to adopt. GQA/MLA win with no inference-time quality decision, but they're a pretraining (or 5%-uptrain) commitment. Eviction/quantization are bolt-on but introduce a quality knob you must tune.
Eviction is a bet on attention; quantization is a bet on precision. Both can be lossless-looking on average and lossy on the tail (a specific needle, a specific reasoning hop). The failure mode is silent.
Long context and KV optimization are inseparable. You cannot serve 128K+ context economically without some member of this family. See [[long-context]].

Honest caveats & open questions

"Lossless" claims are metric-dependent — and the tail is where it bites. Many methods report negligible perplexity loss but are evaluated thinly on reasoning. Recent work (e.g. "Hold Onto That Thought," 2512; "The Pitfalls of KV Cache Compression," 2510) argues compression that's fine for retrieval can hurt multi-step reasoning and long chain-of-thought — exactly the workloads ([[test-time-compute-reasoning]]) that are growing. Treat "lossless compression" as task-conditional, not absolute.
Reported speedups are author/vendor numbers on specific models. KIVI's 2.35–3.47× (and the eviction-side speedups in [[attention-sinks-and-kv-cache]]) are real but configuration-specific (model, context length, hardware, batch). They are not portable constants — re-measure.
Eviction caveats live next door. The fact that query-aware eviction can break prefix-cache reuse, and that greedy eviction can drop a token a later step needed, are treated in [[attention-sinks-and-kv-cache]]; they're inference-time quality knobs that stack on top of (and don't conflict with) the architectural + quantization levers here.
2-bit KV is not a settled default. Sub-4-bit quantization can degrade on long contexts; the per-channel/per-token asymmetry helps but isn't a complete fix. The accuracy/throughput frontier below 4 bits is actively contested.
MLA isn't a drop-in. Its low-rank latent interacts with RoPE and with existing MHA checkpoints; "MHA→MLA" conversion is its own research line (arXiv:2502.14837) and not yet routine.
The field churns fast. Dozens of eviction/compression papers ship per quarter (OBCache, ChunkKV, AttentionPredictor, R-KV…). The principles here (attention sinks, heavy-hitters, asymmetric quantization, low-rank latents) are durable; any specific method's ranking is not.

How it connects to OpenAlice

OpenAlice orchestrates frontier models rather than serving its own weights, so KV-cache optimization shows up as provider economics + local-model feasibility rather than something we re-implement:

Prompt-prefix stability is the cheap win we already get. Alice's prompts carry large, stable preambles (Soul/Identity blocks, system prompt, few-shot scaffolding). On any backend with PagedAttention/RadixAttention prefix caching, keeping that preamble byte-stable across turns is a direct KV-reuse win. The existing "byte-identical prompt-composer" discipline pays off twice: personality fidelity and KV-cache hits ([[llm-inference-internals]], [[context-engineering]]).
KV + weight quantization is what makes no-GPU/local helpers possible. The blal.de server has no GPU; any locally-served small helper ([[small-language-models]]) is feasible only because KV quantization + weight [[quantization]] shrink both the weight-load and the cache footprint enough to run on CPU/RAM.
Long-context cost is a routing input. Because KV (not FLOPs) dominates long-context serving cost, "how long is this context" is a real signal for [[model-routing]] — a short query and a 100K-token RAG dump have very different cost profiles even on the same model.
Educationally, this sits between attention and serving. For OpenAlice Academy it's the bridge node: it presupposes [[attention-and-transformers]], explains a constraint that [[long-context]] and [[deepseek-architecture]] must each solve, and stacks beneath the systems view in [[llm-inference-internals]] and the latency trick in [[speculative-decoding]].

KV-Cache Optimization (compression, eviction, quantization)

KV-Cache Optimization (compression, eviction, quantization)

What it is (intuition first)

Why it matters

How it works (the real mechanics)

Family 1 — Architectural: fewer KV heads (MQA → GQA → MLA)

Family 2 — Eviction & selection: keep only the tokens that matter

Family 3 — Quantization: store K/V in fewer bits

How they compose

Key ideas & tradeoffs

Honest caveats & open questions

How it connects to OpenAlice

See also