LLM Serving Economics — why tokens cost what they cost

For NAO + anyone staring at an API bill or a `$/1M tokens` price list and wondering where the number comes from. This is the economics article, not the systems article. [[llm-inference-internals]] explains how a serving engine works (paged attention, continuous batching, disaggregation); this one asks the dollar question on top of it: why does a token cost what it costs, and what actually moves that cost up or down? The one-sentence version: serving cost ≈ (price of renting a GPU per second) ÷ (useful tokens that GPU emits per second), and that denominator is governed by a single brutal fact — decode is bound by memory bandwidth, not math — which is why batching is the master lever and why every cost trick is, underneath, a trick to put more useful tokens through the same memory bus. Intuition first, then the real mechanics and the honest caveats. Costs and throughput move fast and are hardware-specific, so this article gives relationships, not invented price tags; any concrete number is labelled illustrative.

This is the cost companion to [[llm-inference-internals]] (the systems deep-dive it sits on top of). It pulls together the levers documented elsewhere — [[kv-cache-optimization]], [[quantization]], [[speculative-decoding]], [[flash-attention]], [[distillation]], [[mixture-of-experts]] — and asks of each: which way does it bend the cost curve, and at what price?

What it is (intuition first)

Forget GPUs for a second. The cost of serving an LLM is almost a rental-economics problem:

cost_per_token  ≈  (cost of renting the hardware per second)
                   ──────────────────────────────────────────
                   (useful tokens that hardware produces per second)

The numerator — the rent — is set by the market for accelerators and barely under your control on a given day. Everything an inference engineer does is an attack on the denominator: getting more useful tokens per second out of the same rented silicon. That's it. "Make tokens cheaper" and "raise throughput per GPU at an acceptable latency" are the same sentence.

So the whole game reduces to: how many useful tokens can one GPU emit per second? And to answer that you have to know the one asymmetry that dominates everything (covered in depth in [[llm-inference-internals]], summarised here because the economics ride on it):

• Prefill — reading the prompt. One big parallel pass over all prompt tokens at once, filling the KV cache and producing the first output token. Compute-bound: the GPU's matmul units are the bottleneck. You pay for it once per request, and it scales with prompt length.
• Decode — writing the answer. One token at a time, each step re-reading the entire model weights (and the growing KV cache) from memory to do a tiny sliver of math. Memory-bandwidth-bound: the GPU spends its time moving bytes, not multiplying. You pay for it once per output token.

The economic punchline of that asymmetry: in decode, a single sequence leaves the GPU's compute units almost idle — you've loaded all the weights just to generate one token's worth of math. The fix is to make the same weight-load serve many sequences at once: batching. A batch of 32 sequences reads the weights once and does 32 tokens of math per load — ~32× the useful work for ~1× the memory traffic. Batch size is the throughput dial, and therefore the cost dial. Almost every other lever exists to let you crank that dial further (fit a bigger batch) or to skip work entirely.

A useful mental image: the GPU is a freight train. The locomotive (loading model weights from HBM) costs the same whether you pull one boxcar or fifty. Running near-empty trains — small batches, one user at a time — is what makes tokens expensive. Filling the train is what makes them cheap.

Why it matters

• Inference, not training, is where the lifetime cost lives. A model is trained once; it is served billions of times. For any product with real traffic, the cumulative inference bill dwarfs the one-time training cost, so a 2× serving-efficiency win is a 2× margin win, repeated forever.
• The denominator swings by an order of magnitude on the *same* hardware. Anyscale's benchmark shows naive static batching collapsing to ~81 tokens/sec where continuous batching hits ~1,862 tokens/sec on identical hardware — a ~23× gap (a deliberately adversarial, wide-output-variance setup; treat it as "up to," see caveats) (Anyscale↗). Same rent, ~23× the tokens, ~1/23 the cost per token. The systems layer, not the model, moved the price.
• Price is a *latency-throughput* point, not a single number. You can always make tokens cheaper by batching harder — but each user then waits longer per token. A provider's $/1M tokens is implicitly a promise about latency (how fast first token arrives, how smoothly the stream flows) at some batch size. Two providers quoting the same price can deliver very different experiences; a cheaper price often means a fuller, slower train.
• It's why input tokens are cheaper than output tokens. On most public pricing, output (decode) tokens cost several times more than input (prefill) tokens. That isn't arbitrary: prefill is parallel and compute-bound (cheap per token, you process the whole prompt in one pass), while decode is sequential and memory-bound (expensive per token, one weight-load per token). The price list is the roofline, made visible.
• It reframes "is this feature worth it?" Whether you can afford long context windows, multi-sample [[self-consistency-and-sampling]], agentic tool-loops, or long [[test-time-compute-reasoning]] chains is a serving-economics question: every one of those multiplies decode tokens, the expensive kind.

How it works (the real mechanics)

1. The roofline: why decode cost is set by memory bandwidth

The governing quantity is arithmetic intensity — FLOPs done per byte moved from memory:

arithmetic_intensity ≈ FLOPs_done / bytes_moved
prefill → high   (load weights once, do an N-token-wide matmul)  → compute-bound
decode  → very low (load all weights, do a 1-token-wide matmul)  → memory-bound

A modern accelerator can do hundreds of FLOPs in the time it takes to move one byte from HBM. So in decode, where you move all the weight bytes to do almost no math, the compute units sit idle waiting on memory — the GPU's expensive tensor cores run at a tiny fraction of their peak (low "MFU", model-FLOPs utilization; see Pope et al.↗ for the inference-roofline framing). You are paying to rent compute you cannot use until you raise arithmetic intensity, and the only way to raise it in decode is to batch: amortise each weight-load over more sequences.

This is the load-bearing economic fact:

Decode throughput-per-GPU — and therefore cost-per-token — is set by how big a batch you can keep on the GPU, which is set by how much KV cache fits in memory after the weights.

Hence the chain: more free memory → bigger batch → higher arithmetic intensity → more useful tokens/sec → lower $/token. Every memory-saving trick is a cost-saving trick.

2. KV cache: the bottleneck that caps the batch (and the bill)

The thing that fills up memory and caps the batch is the KV cache — the stored Keys and Values for every token of every in-flight sequence (see [[kv-cache-optimization]] and [[attention-sinks-and-kv-cache]] for the mechanics). It is large and it grows with every generated token. Anyscale's figures: a single sequence on a 13B model can need on the order of ~1 GB+ of KV cache, growing ~1 MB per token, so a 40 GB GPU after weights might hold only on the order of a few dozen concurrent sequences (Anyscale↗ — model/length-specific, illustrative).

So the KV cache is the direct governor of batch size, hence of throughput, hence of cost. Two consequences run the economics:

• Wasted KV memory is wasted money. Classic contiguous allocation wasted 60–80% of KV memory to fragmentation and over-reservation; PagedAttention (vLLM) cut that to <4% by paging the cache into fixed blocks. That recovered memory becomes batch size becomes throughput becomes lower cost — vLLM reports 2–4× throughput over the prior state of the art (vLLM, arXiv:2309.06180↗). This is why the serving engine, not just the model, is a cost decision.
• Reusing KV is free tokens. Shared prefixes (a stable system prompt, a few-shot preamble, multi-turn history) can be computed once and reused across requests instead of re-prefilled each time — prefix caching / RadixAttention. For workloads with heavy prefix repetition this skips real prefill work, which is a direct cost cut. (This is also why keeping prompt preambles byte-stable has a cost payoff — see the OpenAlice section.)

3. Continuous batching: keeping the expensive train full

Fitting a big batch (memory) is only half of it; you have to keep it full over time. Static batching holds a fixed batch until every sequence finishes — but chat outputs vary wildly in length, so when a 2-token reply shares a batch with a 1,500-token one, that slot sits idle (and billed) until the whole batch drains. Continuous batching (iteration-level scheduling, from the Orca paper, OSDI'22) reschedules every forward pass: the instant a sequence emits EOS, it's evicted, its KV freed, and a waiting request is pulled into the slot (Yu et al., OSDI'22↗). No idle billed slots. This is the lever behind the ~8–23× figures above and is the single biggest "same hardware, far cheaper tokens" win in the stack.

The catch — and it's the recurring theme of serving economics — is the throughput↔latency tradeoff. Bigger, fuller batches raise tokens/sec (cheaper) but make each user wait longer per token (worse TPOT), and a chunky prefill injected into the batch spikes everyone's time-to-first-token. You are allocating one fixed memory-bandwidth budget across users; cost and snappiness pull against each other. There is no setting that maximises both.

4. Disaggregation: stop prefill and decode from fighting

Because prefill is compute-bound and decode is memory-bound, mixing them in one batch on one GPU makes them interfere — a big prefill stalls ongoing decodes and wrecks their latency. Phase splitting / disaggregation (Microsoft's Splitwise, arXiv:2311.18677↗; and DistServe, OSDI'24, arXiv:2401.09670↗) puts prefill on one GPU pool and decode on another, each sized and parallelised for its own profile, streaming the KV cache between them over fast interconnect.

The economic framing here is goodput, not raw throughput: the maximum request rate that still meets the latency SLO per provisioned GPU. A colocated system can post huge aggregate throughput while quietly violating latency targets for most users; disaggregation lets you provision the cheap-but-different prefill and decode hardware independently and hit the SLO at lower total cost. Splitwise and DistServe both report serving substantially more SLO-respecting traffic per dollar than colocated baselines (gains are workload- and SLO-specific; see sources). The honest caveat: disaggregation only pays off at scale with fast interconnect and mixed SLO-sensitive traffic — for a small single-GPU deployment, plain continuous batching is simpler and usually cheaper.

5. The cost-curve levers, and which way each one bends it

Every optimization in the library maps onto one of two cost moves: fit a bigger batch (raise the denominator) or skip work (shrink the numerator's workload). Here's the map:

Two of these deserve a cost-intuition note. Quantization is the strongest single cost lever in decode precisely because decode is memory-bound: halving the bytes you move per token (FP16→INT8) roughly halves the decode bottleneck and frees memory for a bigger batch — it bends both terms of the equation. Speculative decoding cuts *latency*, not necessarily cost — it spends extra compute (the draft + the verify) to shorten the sequential critical path, which is exactly the right trade when compute is idle (small batches) but a worse one when the batch is already large and compute-bound. The lever you reach for depends on whether you're latency-starved or throughput-starved.

6. An illustrative cost sketch (do not quote these numbers)

To make the relationships concrete — all numbers below are made-up order-of-magnitude placeholders to show the *shape*, not real prices:

Suppose a GPU rents for some $R/hour and, well-batched, emits ~T output tokens/sec. Then output cost ≈ $R / (3600 · T) per token. If a systems change (paging + continuous batching) triples T, the cost per token falls ~3×, with zero change to $R or the model. If quantization then doubles the batch you can fit, T rises again and cost falls further — until you hit the latency SLO, at which point you stop cranking the batch and the price stabilises at that latency-throughput point.

The point of the sketch is the structure: cost is rent over useful tokens/sec, the systems layer multiplies the denominator several-fold before the model is even touched, and you stop when latency — not arithmetic — says stop.

Key ideas & tradeoffs

• The denominator is the whole game. cost ≈ rent / useful-tokens-per-sec. You rarely control the rent; you control how full the train runs. Most of the 10×-class wins in serving are systems wins (batching + paging), not model wins.
• Decode is the expensive phase, and it's memory-bound. Output tokens cost more than input tokens because decode loads all the weights per token and does almost no math. This single fact explains the price list, the value of batching, and why quantization (fewer bytes moved) is such a strong lever.
• Batch size is the master dial, capped by KV-cache footprint. More free memory → bigger batch → higher arithmetic intensity → cheaper tokens. That's why "save KV memory" and "lower cost" are synonyms.
• Throughput vs latency is an inescapable tension. Every price is secretly a latency promise at some batch size. You cannot maximise tokens/sec and minimise per-user latency at once; disaggregation and chunked prefill narrow the conflict but never erase it. Goodput (SLO-respecting throughput per GPU) is the honest cost metric, not raw throughput.
• The cheapest token is the one you never generate. Prefix-cache reuse, [[semantic-caching-for-llms]], [[model-routing]] to a smaller model, and shorter outputs all beat any per-token optimization, because they remove the work entirely.
• Levers compose, but with diminishing and interacting returns. Quantization + paging + continuous batching stack multiplicatively up to a point; but speculative decoding's win shrinks as the batch grows (compute gets scarce), and disaggregation only pays off at scale. There is no universal recipe — the right stack depends on your traffic shape and SLO.

Honest caveats & open questions

• No real prices here, on purpose. GPU rents, provider margins, and throughput numbers move month-to-month and are wildly hardware- and workload-specific. Every concrete figure in this article is either cited to a primary source (and labelled "up to" / illustrative) or an explicit placeholder. Do not quote any number here as a current price — re-measure on your own model, hardware, and traffic.
• The headline multipliers are adversarial baselines. "23×" (continuous vs static batching) and "2–4× / up to 24×" (PagedAttention) come from setups chosen to flatter the new method — wide output-length variance, naive baselines. Against a competent baseline the gap shrinks. Treat all such numbers as ceilings, not typical.
• Some primaries are secondary here. The Orca continuous-batching paper (OSDI'22) and the Anyscale continuous-batching blog were not independently re-fetched in this writing; the batching figures lean on vLLM/Anyscale descriptions and should be re-confirmed before customer-facing use. Splitwise and DistServe goodput gains are reproduced from the papers' own claims, not an independent re-run.
• "Cost per token" hides real complexity. Real bills fold in idle/over-provisioned capacity, autoscaling lag, traffic burstiness, tail-latency SLOs, multi-tenancy, and reserved-vs-spot pricing. The clean rent / tokens-per-sec model is the right intuition but a floor, not an invoice.
• Prefix-cache sharing is a privacy boundary, not just a cost trick. Sharing read-only prompt prefixes across requests is safe and cheap; sharing anything user-specific can leak KV state across tenants. The cost win and the security risk live in the same mechanism.
• The frontier moves under you. FP8/INT4 KV cache, chunked prefill, disaggregation, and ever-better speculative methods keep shifting where the bottleneck is — and MoE / long-context / heavy reasoning shift the workload the other way. Any economics conclusion here is a snapshot of a moving roofline.

How it connects to OpenAlice

OpenAlice is mostly a client of inference, not a host of it — Alice's brain runs on Codex OAuth (gpt-5.5 on mvp, gpt-5.4 in prod) and OpenRouter providers, so the lab doesn't operate its own vLLM/SGLang fleet today. But the economics are load-bearing on both sides of that boundary:

• Every product decision that multiplies output tokens is a cost decision. Long [[test-time-compute-reasoning]] chains, multi-sample [[self-consistency-and-sampling]], agentic tool-loops, and big context windows all multiply decode tokens — the expensive, memory-bound kind. The Worker vs Companion per-chat mode axis (Worker = "for real work like a diploma," which runs the expensive thinker) is, economically, a coarse "how much decode should this conversation be allowed to spend" dial — the same intuition as [[model-routing]]: pay for the big model only when the turn earns it.
• Byte-stable prompt preambles pay off twice. Alice's large, stable Soul / Identity / system-prompt blocks are exactly the shared prefix that prefix-caching (RadixAttention / PagedAttention) reuses for free. The existing "byte-identical prompt-composer" discipline — kept for personality fidelity — also maximises KV-cache hits on any prefix-caching backend, which is a direct serving-cost win. Personality preservation and cost efficiency want the same thing here.
• Streaming UX is the latency half of the throughput↔latency trade. Alice's chat feel is dominated by TTFT (first token) and TPOT (stream smoothness) — the exact knobs a provider tunes against batch size. The known Codex long-reply truncation (~120 s / ~35 K chars, project-codex-stream-truncation-long-reply-2026-06-03) is a decode-side latency/cap failure of the upstream serving stack; understanding the prefill/decode economics is how we reason about salvaging partials instead of discarding-and-retrying.
• No-GPU server means quantization is the enabling lever, not a nice-to-have. The blal.de server is CPU-only. If the lab ever self-hosts a local helper model (the openalice-runner vision, or local models behind bench.blal.pro), the roofline still applies qualitatively — decode stays memory-bound — but on CPU the absolute numbers differ wildly, and [[quantization]] / [[bitnet-1bit-llms]] are what make any local serving feasible at all. The cost equation doesn't change; only the rent and the achievable tokens/sec do.

Sources

• Kwon et al., *PagedAttention / vLLM* (arXiv:2309.06180)↗ — the KV-memory-waste → batch-size → throughput chain that underpins the cost argument.
• Yu et al., *Orca: continuous / iteration-level batching* (OSDI'22)↗ — the keep-the-batch-full lever; (not independently re-fetched here; figures via vLLM/Anyscale secondary descriptions).
• Patel et al., *Splitwise: phase splitting* (arXiv:2311.18677)↗ — prefill/decode disaggregation for cost/goodput.
• Zhong et al., *DistServe* (OSDI'24, arXiv:2401.09670)↗ — goodput-optimised disaggregation; the "goodput > throughput" framing.
• Anyscale, *How continuous batching enables 23x throughput*↗ — the same-hardware throughput-gap figures (adversarial baseline; "up to," returned HTTP 403 on direct fetch in prior sessions — re-confirm before quoting).
• Pope et al., *Efficiently Scaling Transformer Inference* (arXiv:2211.05102)↗ — the inference roofline / MFU framing behind "decode is memory-bound."

See also

[[llm-inference-internals]] (the systems deep-dive this sits on) · [[kv-cache-optimization]] · [[attention-sinks-and-kv-cache]] · [[quantization]] · [[speculative-decoding]] · [[flash-attention]] · [[mixture-of-experts]] · [[distillation]] · [[small-language-models]] · [[model-routing]] · [[semantic-caching-for-llms]] · [[test-time-compute-reasoning]] · [[scaling-laws]] · [[deepseek-architecture]] (MLA compresses the KV cache — a structural attack on the same memory bottleneck)