Model Routing & Cost-Optimal Serving

For NAO + anyone building a multi-model system. Not every query needs your most expensive model. This article is the engineering counterpart to [[fusion-and-llm-councils]]: fusion asks "can many cheap models together beat one frontier model?" — routing asks "for **this** query, which **one** model is enough?" Both are ways to spend less for the same quality. We ground this in three real techniques that operate at three different layers — routing (pick a model before generating), cascading (escalate models after generating), and speculative decoding (use a small model to make a big one faster) — and connect all three to OpenAlice's cost ladder.

The one-sentence idea

Match each request to the cheapest resource that can serve it well — choose the model per-query (routing), retry only the hard ones on a stronger model (cascading), or let a small model do the easy token-by-token work and have the big model only check it (speculative decoding).

The economic premise is empirical and robust: query difficulty is heavily skewed. A large fraction of real traffic is easy — chit-chat, formatting, lookups, simple rewrites — and a cheap model answers it indistinguishably from a frontier model. The expensive model only earns its price on the hard tail. A system that always calls the frontier model pays the tail price for every token. Routing/cascading let you pay it only on the tail.

Three layers, three mechanisms

These are commonly lumped together as "cost optimization" but they are orthogonal and compose. Keep them straight:

The survey Dynamic Model Routing and Cascading (arXiv:2603.04445) draws the same line: routing is "selects the most appropriate model based on query characteristics" (one shot, no generation needed to decide); cascading "sequentially queries a pool of LLMs until a reliable response is obtained" (you must generate, then judge, then maybe redo). Routing is cheaper to run (no wasted generations) but blind — it bets before seeing any output. Cascading is more expensive on hard queries (you pay the weak model and the strong one) but is self-correcting because it sees the actual answer before escalating.

How it works — Routing (RouteLLM)

RouteLLM (Ong et al., UC Berkeley / LMSYS, arXiv:2406.18665) is the cleanest study of the binary routing case: one strong model M_strong (e.g. GPT-4), one weak model M_weak (e.g. Mixtral-8x7B), and a learned router that decides per-query.

The router is a win-probability model

The core object is a scoring function s(M, q) — how well model M answers query q. The best-performing router uses matrix factorization borrowed from recommender systems (think Netflix: users × movies → here, models × queries). It embeds each model as a vector v_m and each query as v_q, and scores via a bilinear form (in words: s = w₂ᵀ · (v_m ⊙ (W₁ᵀ v_q + b)), where ⊙ is element-wise product). The probability that the strong model "wins" a head-to-head on this query follows a Bradley-Terry sigmoid:

P(win_strong | q) = σ( s(M_strong, q) − s(M_weak, q) )

The routing rule is then a single threshold α ∈ [0,1]:

route to strong if P(win_strong | q) ≥ α, else route to weak.

α is the cost dial. Low α → almost everything goes to the cheap model (maximize savings); high α → almost everything to the frontier model (maximize quality). You calibrate α on a validation set to hit a target like "I'm willing to send 30% of traffic to GPT-4."

How you train it

The training signal is human preference data: triplets of (prompt, response A, response B, label∈{A wins, B wins, tie}), mined from Chatbot Arena. The router learns which kinds of prompts the strong model actually wins on — not via hand-coded difficulty rules, but from millions of crowd votes. Three other router families were tried: a similarity-weighted (SW) Elo ranker (training-free, solved at inference — best on raw sparse Arena data), a BERT classifier, and a causal-LLM classifier (a fine-tuned Llama-3-8B — strongest on math/GSM8K but needs 8×A100 to train). Matrix factorization won overall.

Data augmentation matters more than the architecture

Raw Arena data is sparse — most model pairs were never compared on most prompt types. Two augmentation tricks closed the gap and were the single biggest lever:

• Golden-label data: take MMLU's validation split (~1,500 samples, <2% of training data), derive preference labels by checking responses against the known correct answer.
• LLM-judge labels: have GPT-4 judge 120K response pairs (GPT-4 vs Mixtral on the Nectar dataset) — cost ~$700, doubled the matrix-factorization router's efficiency on MT Bench.

The metrics (worth internalizing — they generalize to any router)

• PGR — Performance Gap Recovered: (router − weak) / (strong − weak). Normalizes the router between the weak model (0.0) and the strong model (1.0). PGR = 0.5 means "halfway to frontier quality."
• CPT(x%) — Call-Performance Threshold: the minimum fraction of strong-model calls needed to reach x% PGR. This is the headline cost number. On MT Bench, CPT(50%) ≈ 37% — half the quality gap is recovered while sending only 37% of traffic to GPT-4 (vs 49% for random routing).
• APGR — Average PGR: area under the PGR-vs-cost curve across 10 α settings. A single number for "how good is this router across the whole cost spectrum."

The numbers

Cost reduction vs always-call-the-strong-model, at matched quality: MT Bench ≈ 3.66× cheaper at 95% quality (the easy, conversational distribution — routing shines); MMLU ≈ 1.41×; GSM8K ≈ 1.49× (math is genuinely hard across the board, less easy traffic to offload). Crucially, routers trained on the GPT-4/Mixtral pair transferred to an unseen Claude-3-Opus / Llama-3-8B pair without retraining (APGR 0.703) — evidence the router learns query difficulty as an abstraction, not just "this specific model's quirks."

How it works — Cascading (FrugalGPT)

FrugalGPT (Chen, Zaharia, Zou, Stanford, arXiv:2305.05176) is the canonical cascade. Where routing decides up front, a cascade generates first, then decides whether to keep going.

The mechanism:

1. Send the query to the cheapest model in an ordered list.
2. A learned quality/scoring function (FrugalGPT uses a DistilBERT regressor) scores the answer's reliability.
3. A stop judge compares the score to a per-model threshold. If it clears the bar → return that answer, done. If not → escalate to the next, more expensive model and repeat.

FrugalGPT names three cost strategies — prompt adaptation (shorter/cheaper prompts), LLM approximation (caching, distillation), and the LLM cascade (the headline). The cascade's order and thresholds are jointly optimized under a cost budget to maximize accuracy. Reported headline: match GPT-4's performance at up to 98% lower cost, or +4% accuracy at the same cost — on favourable benchmarks (more on the caveat below).

The hard, unsolved part of cascading is the confidence/deferral estimate — knowing when the cheap answer is wrong without an oracle. The survey catalogs the families: external LLM-as-a-judge; self-verification (AutoMix's few-shot self-check framed as a POMDP; Self-REF's fine-tuned confidence tokens); conformal prediction on logits (CP-Router); and hidden-state probes — the survey notes trained hidden-state probes give "the most reliable confidence estimates," while verbalized self-confidence ("I'm 90% sure") shows "low alignment between reported uncertainty and prediction correctness." That calibration gap is the central risk: a cascade is only as good as its ability to notice its own mistakes.

How it works — Speculative decoding (a different kind of "routing")

Speculative decoding (Leviathan, Kalman, Matias, Google, ICML 2023 Oral, arXiv:2211.17192) routes at the token level and is the one technique here with zero quality cost — a free lunch that's now standard in vLLM/TGI.

The trick exploits that LLM inference is memory-bandwidth-bound: verifying K tokens in one forward pass costs barely more than generating one. So:

1. A small draft model q cheaply guesses the next γ tokens (autoregressively, fast).
2. The big target model p scores all γ guesses in a single parallel forward pass.
3. A modified rejection sampling scheme accepts each draft token x with probability min(1, p(x)/q(x)); on the first rejection it resamples that one token from an adjusted distribution and discards the rest.

The deep result: this scheme makes the output distribution provably identical to sampling from the target model alone — same temperature, same everything. You are not approximating the big model; you are reproducing it exactly, just faster. The speedup is governed by the acceptance rate α (how often the draft agrees with the target): more agreement → more tokens accepted per target pass → fewer expensive forward passes. Reported 2×–3× wall-clock speedup with identical outputs on T5-XXL. The tradeoff is alignment vs draft cost: a tinier draft model is faster to run but diverges more from the target (lower α, fewer accepted tokens). The sweet spot is a draft model in the same family, ~10–20× smaller.

Key ideas & tradeoffs

• Difficulty is skewed, and that skew is the whole business case. Routing wins big on conversational traffic (MT Bench 3.66×) and only modestly on uniformly-hard traffic (MMLU 1.41×). Measure your traffic's difficulty distribution before assuming the headline savings.
• One dial: cost ↔ quality. RouteLLM's α, a cascade's thresholds — they all collapse to a single knob you slide along a Pareto frontier. The engineering job is calibrating that knob on a representative validation set, not picking it once.
• Routing is cheaper but blind; cascading is self-correcting but pays twice. Routing never wastes a generation but bets without seeing output. Cascading sees the answer but, on a hard query, you pay the weak model and the strong one and the scorer — net more expensive than going straight to strong. They compose: route first to skip the obvious frontier-only queries, cascade the ambiguous middle.
• Speculative decoding is the only free lunch. Routing/cascading trade quality for cost (and you choose where on the curve). Speculative decoding trades nothing — it's exact — it only needs GPU memory for two models. If you serve your own weights, do this unconditionally.
• The router/judge is itself a model with error. A routing mistake sends a hard query to the weak model and ships a bad answer silently. A cascade mistake is a mis-calibrated confidence score. The quality of the meta-model bounds the whole system.
• Preference data > hand rules. RouteLLM's lift came from learning difficulty from Arena votes and cheap augmentation — not from clever architecture. Data is the lever.
• The survey's frontier is composition + generalization. Most published work does single-stage routing; production wants routing + cascading + bandit adaptation together, and routers that don't need retraining when you swap a model (ICL-Router, UniRoute, clustering methods are early attempts).

Honest caveats & limitations

• Headline savings are best-case and benchmark-dependent. "98% cheaper" / "3.66×" come from favourable distributions (chat, MT Bench). On uniformly hard or out-of-distribution traffic the gains shrink toward 1× — and a mis-calibrated router can be worse than always-strong because it ships wrong answers at the weak price.
• Confidence estimation is genuinely unsolved. Cascades live or die on knowing when the cheap answer is wrong. Verbalized confidence is poorly calibrated; probes need training data and degrade out-of-domain. There is no general, reliable, cheap "is this answer good?" oracle yet — that's an open research problem, not a config setting.
• Routers drift and must be retrained. A router is trained against specific models. Swap the weak model, change a system prompt, or let the underlying API update silently, and the router's win-probability estimates rot. RouteLLM's cross-pair transfer is encouraging but not a guarantee; treat the router as a model with a maintenance burden, not a static rule.
• Cascades add latency and a double-spend tail. Every escalation is a full extra generation. P99 latency on hard queries can be worse than always-frontier, and the cost on those queries is strictly higher.
• Speculative decoding needs your own weights + the memory for two models. It does nothing for closed APIs (you can't insert a draft model into OpenAI's serving stack), and its speedup collapses if draft/target distributions diverge (off-distribution inputs, aggressive sampling, a too-small draft).
• Non-determinism complicates evaluation. Sampling temperature means the same router/cascade can pass on one run and fail on the next; you need fixed-seed/golden-trace harnesses to measure routers honestly — exactly the problem OpenAlice's trace-replay corpus solves.
• "Same quality" hides distribution shift. Aggregate quality can match while the failure modes change — a router that offloads to a weak model may be fine on average but systematically worse on a sub-population (e.g. a non-English cohort) that the aggregate metric hides.

How it connects to OpenAlice

OpenAlice already practices this philosophy under a different name: the cost ladder (see agent memory project-ecosystem-trio-unified-2026-06-02). NAO's operating rule — "push every improvement as far DOWN the ladder as possible before spending tokens" — is cost-optimal serving, generalized beyond model choice to the whole improvement loop:

1. Token-free static (inspector / lab ranker / features.json drift-check)
2. Token-free trace-replay (golden traces → deterministic regression, no new LLM calls — the lab replay keystone, ReplaySolver = dict lookup)
3. Cheap heuristics (Curator-style — distill repeated hot-path LLM calls into cached heuristics; this is exactly FrugalGPT's "LLM approximation" strategy)
4. Minimal targeted live probes (change-scoped subset, not full 15-sample)
5. Rare full external benchmarks (SWE-bench / HumanEval-mini, milestone-only)

Mapping the literature onto OpenAlice, concretely:

• Per-turn model selection = binary routing. Alice runs gpt-5.5 on mvp / gpt-5.4 in prod (see MEMORY provider config). A RouteLLM-style win-probability router could send easy companion-chat turns to a cheaper model and reserve the frontier model for hard reasoning / coding turns — directly cutting the standing per-turn cost. The lab's preference/probe data (lab-harness, lab-scoring) is the natural training corpus for such a router.
• The biggest standing economy is heuristic distillation (ladder rung 3). The one real per-turn LLM drain is mood detection (~190 tok/turn, agents-memory/src/emotional_context.rs::detect_mood_llm). Distilling it to a cheap classifier is textbook FrugalGPT "LLM approximation." But it is soul-adjacent and GATED on NAO — it feeds Alice's emotional/VAD state, so any distillation must pass behavioral-envelope validation, not cosine distance (per [[fusion-and-llm-councils]]'s sibling rule feedback-personality-envelope-not-warmth-reference). This is the honest tension between cost-optimal serving and personality preservation.
• Cascading maps to the lab's probe gate. "Try the token-free path; if it can't decide, escalate to a live probe" is a cascade with a deferral judge. OpenAlice's α is set by the change scope, not a sigmoid threshold.
• Speculative decoding is the one rung not yet on the ladder — it's an inference-serving optimization, and OpenAlice currently serves via the Codex OAuth API (no local weights, no draft-model insertion point). It becomes relevant only if/when Alice self-hosts open weights; until then it's a known free-lunch to bank for later.
• Routing is the per-query sibling of fusion. [[fusion-and-llm-councils]] (OpenRouter Fusion: a budget panel of Gemini-3-Flash + Kimi-K2.6 + DeepSeek-V4 beats GPT-5.5 and Opus 4.6 at ~50% cost) and routing are two answers to the same question. Fusion spends more than one cheap call but less than one frontier call, and wins by combining; routing spends less than the frontier call and wins by selecting. The mature system does both: route the easy 80% to one cheap model, fuse a budget council on the hard 20% — cheaper than frontier on both sub-populations.

See also

• [[fusion-and-llm-councils]] — the complementary "combine many models" answer to the same cost question; the budget-panel-beats-frontier result.
• [[mixture-of-agents]] — the research lineage behind fusion/council layering.
• [[llm-from-scratch]] · [[microgpt-build-an-llm-from-scratch]] — what's inside the models you're routing between.
• [[graphify]] · [[graphrag]] — retrieval/graph context that changes a query's effective difficulty (and so what a router should pick).
• [[agent-memory-systems]] · [[mempalace]] — memory that, like routing, is about not re-paying for work already done.
• [[llm-maintained-wiki]] — the knowledge-library pattern this article lives in.

Sources

• RouteLLM (Ong et al., LMSYS/UC Berkeley) — blog↗ · arXiv:2406.18665↗
• FrugalGPT (Chen, Zaharia, Zou, Stanford) — arXiv:2305.05176↗
• Fast Inference via Speculative Decoding (Leviathan, Kalman, Matias, Google) — arXiv:2211.17192↗
• Dynamic Model Routing and Cascading for Efficient LLM Inference: A Survey — arXiv:2603.04445↗
• OpenRouter — Surpassing Frontier Performance with Fusion — blog↗