LLM Inference Internals (vLLM / serving)

Beginner ramp: Training is where a model learns. Inference (also "serving") is where a trained model answers — the part NAO's chat, Alice's replies, and every API call actually hit. It looks simple ("send prompt, get tokens back"), but doing it fast and cheap for many users at once is a hard systems problem. This article is about that systems problem: where the time and the GPU memory go, and the handful of tricks (paged attention, continuous batching, prefill/decode separation) that make modern serving 10–24× faster than the naive version.

What it is — intuition first

When you generate from an autoregressive LLM, the model produces one token at a time. To make token N, the model attends back over tokens 1…N−1. Naively, that would mean re-reading the whole prompt and all prior output on every single step — quadratic, brutal.

The standard fix is the KV cache: for each token, the model computes a Key and a Value vector at every attention layer, and caches them. Token N only needs to compute its own new K/V, then attend against the cached K/V of everything before it. This turns each decode step into roughly O(N) attention work instead of re-running the prompt. (The mechanics of K, Q, V and attention live in [[attention-and-transformers]] and [[flash-attention]].)

So inference splits cleanly into two phases:

Prefill — process the whole prompt in one big parallel pass, fill the KV cache, emit the first token. This is one forward pass over many tokens at once. Compute-bound (lots of matmul, the GPU's tensor cores are busy).
Decode — generate tokens one at a time, each reading the entire KV cache + model weights, doing tiny per-step compute. Memory-bandwidth-bound (the GPU spends its time loading bytes, not multiplying).

That asymmetry — prefill is a compute job, decode is a memory job — is the single most important fact in serving, and almost every optimization below is downstream of it.

Why it matters

Inference, not training, is where most production cost and latency live. Three numbers from real systems frame the stakes:

KV cache is enormous. A single sequence on LLaMA-13B can need up to 1.7 GB of KV cache, and it grows by ~1 MB per token per sequence on a 13B model. On a 40 GB A100 after weights, that caps you at ~28 concurrent sequences at 512-token length — before any intermediate activations (Anyscale).
Naive memory management wastes 60–80% of that cache to fragmentation and over-reservation, so you serve far fewer users than the hardware could (vLLM paper).
Naive batching wastes the GPU in time. With variable-length chat outputs, static batching can collapse to 81 tokens/sec, where a well-batched system hits 1,862 tokens/sec on the same hardware — a 23× gap (Anyscale).

The job of an inference engine (vLLM, TGI, SGLang, TensorRT-LLM) is to attack both axes — wasted memory and wasted time — so a fixed GPU serves the most useful tokens per second within your latency budget.

How it works — real mechanics

1. The roofline: why decode is memory-bound

Per decode step, the GPU must load all model weights (and the KV cache) from HBM to compute, but does only a tiny amount of math per loaded byte — low arithmetic intensity. So decode throughput is set by how big a batch you can fit, because batching amortizes the weight-load over many sequences:

arithmetic_intensity ≈ FLOPs_done / bytes_moved
decode  → very low  (load all weights, do 1-token-wide matmul)  → memory-bound
prefill → high      (load weights once, do N-token-wide matmul) → compute-bound

Consequence: batch size is the throughput lever for decode, and it is capped by how much KV cache fits in memory. Hence the obsession with KV-cache memory efficiency — every byte you don't waste is another sequence you can batch.

2. PagedAttention — KV cache as virtual memory (vLLM)

The problem PagedAttention solves: classic systems store each sequence's KV cache in one contiguous buffer sized to the max possible length. Three wastes follow:

Internal fragmentation — you reserve max-length, but most sequences are short → the unused tail is dead.
Reservation waste — slots reserved for tokens not generated yet.
External fragmentation — variable-sized buffers leave unusable gaps between them.

Measured cost: 60–80% of KV memory wasted; only 20–40% actually holds tokens (vLLM blog).

The fix (the OS-paging analogy): partition the KV cache into fixed-size blocks, each holding the K/V for a fixed number of tokens (e.g. 16). A sequence's logical blocks need not be physically contiguous — a per-sequence block table maps logical block → physical block, exactly like an OS page table maps virtual pages to physical frames. Physical blocks are allocated on demand as the sequence grows.

Sequence "The cat sat ..."        Block table              Physical KV blocks (HBM, non-contiguous)
  logical block 0  ─────────────►  [0 -> phys 7]  ───────► phys 7  : The cat sat on
  logical block 1  ─────────────►  [1 -> phys 2]  ───────► phys 2  : the mat and ...
  logical block 2 (partial)  ───►  [2 -> phys 9]  ───────► phys 9  : purred <pad><pad>

Now waste only ever occurs in the last, partially-filled block of each sequence → under 4% waste (vs 60–80%). That recovered memory directly raises the batchable concurrency, which (per the roofline) raises decode throughput.

Sharing via copy-on-write. Because blocks are indirected through a table with reference counts, multiple sequences can point at the same physical block. Two big wins:

Parallel sampling / beam search from one prompt: all samples share the prompt's KV blocks read-only. When a sample needs to diverge (write into a shared block), the engine does copy-on-write — clones just that block, decrements the refcount. vLLM reports up to 55% memory saved on parallel-sampling workloads.
Shared prefixes (system prompts, few-shot preambles) shared across different requests.

# Sketch of the block manager's copy-on-write on a shared KV block
def append_token_kv(seq, k, v):
    blk = seq.logical_blocks[-1]
    phys = block_table[seq][blk]
    if refcount[phys] > 1:            # block is shared → must not overwrite others
        new_phys = allocate_block()
        copy(new_phys, phys)          # copy-on-write
        refcount[phys] -= 1
        block_table[seq][blk] = new_phys
        phys = new_phys
    write_kv(phys, offset_in_block(seq), k, v)

Headline numbers: 2–4× throughput over the prior state of the art (FasterTransformer, Orca); up to 24× over vanilla HuggingFace Transformers, 2.2–3.5× over an early HuggingFace TGI; LMSYS saw up to 30× over their initial HF backend.

3. Continuous batching — iteration-level scheduling

Memory efficiency lets you fit a big batch; continuous batching keeps that batch full over time.

Static batching holds a fixed batch until every sequence finishes. But chat outputs have wildly different lengths — when sequence 3 stops after 2 tokens and sequence 1 runs to 1500, the slot from sequence 3 sits idle until the whole batch drains. Massive temporal waste.

Continuous batching (a.k.a. iteration-level scheduling, from the Orca paper, OSDI'22) reschedules the batch every forward pass:

for each iteration (one forward step):
    finished = [s for s in batch if s.emitted_eos or s.at_max_len]
    for s in finished:
        return s; free its KV blocks; remove from batch
    while waiting_queue and kv_memory_available():   # inject new work into freed slots
        batch.add(next_request())                    # its prefill runs this step
    run_model_step(batch)                            # one token for every active seq

A finished sequence is evicted the instant it emits EOS and its KV blocks are freed, and a waiting request is pulled in to fill the slot — no waiting for the batch to drain. This is why it improves both throughput and latency at all percentiles: new requests start almost immediately instead of queueing behind a stuck batch. Reported gains over naive static batching: ~23× (vLLM), ~8× (TGI), ~4× (FasterTransformer's optimized static batching) — Anyscale benchmark.

A subtlety: a newly injected request needs its prefill computed, which can momentarily stall ongoing decodes (a "prefill bubble"). Engines mitigate with chunked prefill (split a long prompt's prefill across several steps so it interleaves with decode) or by piggybacking small prefills onto decode steps.

4. RadixAttention — automatic cross-request KV reuse (SGLang)

PagedAttention's copy-on-write shares prefixes if you explicitly fork them. RadixAttention (SGLang) makes prefix sharing automatic and persistent across unrelated requests. It keeps the KV cache of completed/in-flight requests in a radix tree (compressed prefix trie) keyed on token sequences. A new request walks the tree; any matching prefix (shared system prompt, common few-shot block, multi-turn history) is a cache hit — its KV is reused, not recomputed. An LRU eviction policy reclaims tree nodes under memory pressure, and a cache-aware scheduler orders requests to maximize hits. SGLang also compiles structured-output constraints into a compressed finite-state machine for fast JSON/grammar decoding. Reported: up to 6.4× throughput over prior systems on agentic, RAG, few-shot, and multi-turn-chat workloads where prefixes repeat heavily.

5. Prefill/decode disaggregation — DistServe

Continuous batching mixes prefill and decode in the same batch on the same GPU. But prefill is compute-bound and decode is memory-bound, so they interfere: a chunky prefill spikes the TTFT (time-to-first-token) of others, while decode wants steady TPOT (time-per-output-token). DistServe disaggregates them onto separate GPU pools, each scaled and parallelized for its own profile, then streams the KV cache from the prefill instance to the decode instance.

Definitions. TTFT = prefill latency (first token). TPOT = average per-token decode latency. Goodput = max request rate that still meets the SLO (e.g. 90% of requests under their TTFT/TPOT targets) per provisioned GPU — the metric that actually matters, vs raw throughput which can hide SLO violations.
KV transfer is cheap on fast interconnect: over NVLink (≈600 GB/s) the cross-instance KV copy is <0.1% of total latency; >95% of requests see <30 ms transfer delay.
Gains: up to 4.48× more requests, or a 10.2× tighter SLO, on summarization (OPT-66B); 2.0–3.41× higher request rate on chatbot workloads vs vLLM-style colocation (DistServe, OSDI'24).

Key ideas & tradeoffs

Throughput vs latency is the master tradeoff. Bigger batches → higher tokens/sec (throughput) but each user waits longer per step (worse TPOT), and a long prefill in the batch worsens everyone's TTFT. Continuous batching narrows the conflict; disaggregation lets you tune each phase independently. There is no free lunch — you're allocating a fixed memory-bandwidth budget across users.
Memory efficiency *is* throughput. Because decode is memory-bound and batch size is capped by KV-cache footprint, paging (≤4% waste) and prefix reuse translate almost linearly into more concurrent sequences → more throughput.
Block size is a tuning knob. Smaller KV blocks → less internal fragmentation but more block-table overhead and more kernel indirection; larger blocks → cheaper indexing but more tail waste. Engines default around 16 tokens.
Goodput > throughput for real SLAs. A system can post huge aggregate throughput while violating latency SLOs for most users; DistServe's framing (goodput) is the honest metric.
The engines overlap heavily. vLLM (PagedAttention + continuous batching), TGI (HuggingFace's production server; continuous batching, since adopted paged KV and flash kernels), SGLang (RadixAttention + structured decoding), TensorRT-LLM (NVIDIA, fused kernels + in-flight batching). Most ideas here have cross-pollinated; "which is fastest" depends on workload (prefix-heavy → SGLang shines; tight-SLO mixed traffic → disaggregation).
All of this composes with kernel- and weight-level tricks. [[flash-attention]] makes the attention kernel itself memory-efficient and fast (tiling, no materialized N×N matrix) — orthogonal to and stacked with PagedAttention. [[quantization]] (INT8/FP8/INT4 weights and increasingly KV-cache quantization) shrinks both the weight-load (helping decode's memory-bound bottleneck) and the KV footprint (more batchable sequences). Speculative decoding (a small draft model proposes, the big model verifies in one pass) attacks decode's sequential latency directly.

Honest caveats & open questions

Benchmark numbers are workload-specific and partly marketing. "23×" and "24×" come from adversarial variance setups (wide output-length spread) vs naive static batching — a strong but cherry-favorable baseline. Against a competent baseline the gap shrinks. Treat any single multiplier as "up to," not typical.
The field moves fast; some baselines are dated. The vLLM-vs-TGI numbers are from 2023; TGI later adopted paged KV and flash-attention, and vLLM has since shipped chunked prefill, FP8 KV cache, prefix caching, and speculative decoding. Always re-benchmark on your model/hardware/traffic.
Disaggregation isn't a universal win. It adds an extra KV transfer, needs fast interconnect (NVLink-class), and only pays off at scale with mixed SLO-sensitive traffic. For small single-GPU deployments, colocated continuous batching is simpler and often better. ("Beyond the Buzz: A Pragmatic Take on Inference Disaggregation," arXiv:2506.05508, argues exactly this.)
Prefix-cache reuse can leak across tenants if KV blocks are shared between users without isolation — a real security/privacy consideration for multi-tenant serving. Sharing read-only prompt prefixes is safe; sharing anything user-specific is not.
KV-cache quantization is still maturing. It boosts capacity but can degrade quality on long contexts; the accuracy/throughput frontier is unsettled.
Numbers I did not independently verify: the original Orca continuous-batching paper is OSDI'22 (Yu et al.), not on arXiv — I could not fetch its PDF and relied on the vLLM/Anyscale secondary descriptions of iteration-level scheduling. The "≤4% waste," "60–80% waste," and "55% sampling savings" figures come from the vLLM paper/blog and were not cross-checked against an independent re-implementation. The Anyscale continuous-batching blog returned HTTP 403 on direct fetch; its figures here are from a cached/secondary read and should be re-confirmed before quoting in a customer-facing context.

How it connects to OpenAlice

OpenAlice is mostly a client of inference rather than a host of it — Alice's brain runs on Codex OAuth (gpt-5.5 on mvp, gpt-5.4 in prod) and OpenRouter providers, so the lab does not operate its own vLLM/SGLang fleet today. That said, this material is directly load-bearing in several places:

Latency budgets and streaming. Alice's chat experience is dominated by TTFT (how fast the first token streams) and TPOT (smoothness of the stream) — exactly the metrics in this article. The memory note project-codex-stream-truncation-long-reply-2026-06-03 (Codex truncating long replies at ~120 s / ~35 K chars) is a decode-side latency/cap failure of the upstream serving stack; understanding prefill-vs-decode and TTFT/TPOT is how we reason about salvaging partials instead of discarding + retrying.
Prompt-prefix reuse. Alice's prompts carry large, stable preambles (Soul/Identity blocks, system prompt, few-shot scaffolding — see the prompt-inventory and personality-preservation notes). On any serving backend with RadixAttention-style or PagedAttention prefix caching, keeping that preamble byte-stable across turns is a measurable cost/latency win (cache hits on the shared prefix). The existing "byte-identical prompt-composer" discipline pays off twice: personality fidelity and KV-cache reuse.
Throughput vs latency is a product decision. If the lab ever self-hosts (the openalice-runner vision, or local models for the benchmark rig at bench.blal.pro), the batch-size / SLO tradeoff here is the knob between "cheap, high-throughput batch jobs" and "snappy interactive Alice." Note the server is CPU-only (no GPU) today — so any local serving is llama.cpp-class CPU inference where these GPU-roofline mechanics still qualitatively apply (decode stays memory-bound) but the absolute numbers differ wildly.
Quantization is the lever that fits a model on no-GPU hardware — see [[quantization]]; KV-cache and weight quantization are what make CPU/edge serving of the local helper models feasible at all.

LLM Inference Internals (vLLM / serving)

LLM Inference Internals (vLLM / serving)

What it is — intuition first

Why it matters

How it works — real mechanics

1. The roofline: why decode is memory-bound

2. PagedAttention — KV cache as virtual memory (vLLM)

3. Continuous batching — iteration-level scheduling

4. RadixAttention — automatic cross-request KV reuse (SGLang)

5. Prefill/decode disaggregation — DistServe

Key ideas & tradeoffs

Honest caveats & open questions

How it connects to OpenAlice

See also