Scaling Laws (Chinchilla)

One-line summary. When you train a transformer language model, its loss falls in a smooth, predictable power law as you add parameters, data, and compute. The Chinchilla result (Hoffmann et al. 2022) showed that for a fixed compute budget you should scale model size and training tokens together, roughly equally — about 20 tokens per parameter — overturning the earlier Kaplan (2020) advice to build huge models on relatively little data. Scaling buys you lower loss reliably; whether it buys you "new abilities" is genuinely contested.

What it is (intuition first)

Imagine you're baking bread and you have a fixed amount of money to spend. You can spend it on a bigger oven (more model parameters — capacity to memorize and represent patterns) or on more flour and time (more training tokens — examples the model learns from). A scaling law is the recipe that tells you, given your budget, how to split the money so the bread comes out as good as possible — and, remarkably, lets you predict the taste before you bake a much larger loaf.

That last part is the magic. For most of deep learning, you couldn't say in advance how good a bigger model would be — you just trained it and hoped. Scaling laws changed that. They are empirical observations that the test loss L of a language model behaves like a clean mathematical function of three things:

• N — the number of model parameters (its size / capacity),
• D — the number of training tokens (how much data it sees), and
• C — the total training compute (roughly the product of the two; see below).

Across an astonishing range — Kaplan's curves span "more than seven orders of magnitude" — the loss follows a power law: plot loss against compute on log-log axes and you get a nearly straight line. Straight lines extrapolate. That means you can train a fleet of small, cheap models, fit the line, and forecast how a model 100× bigger will perform before spending the money to build it. This is the single most important reason the field could justify nine-figure training runs: the outcome was, to a useful approximation, predictable.

Two foundational papers define the topic and famously disagreed:

1. Kaplan et al. (2020), "Scaling Laws for Neural Language Models" (OpenAI). Established the power-law form and concluded: given more compute, pour most of it into a bigger model and train on relatively modest data, stopping before convergence. This thinking produced GPT-3 — 175B parameters trained on only ~300B tokens.
2. Hoffmann et al. (2022), "Training Compute-Optimal Large Language Models" (DeepMind), known as the Chinchilla paper. Re-ran the experiment more carefully and found Kaplan was wrong about the allocation: model size and data should grow in lockstep, "for every doubling of model size the number of training tokens should also be doubled." Their 70B Chinchilla, trained on 4× more data than the 280B Gopher at the same compute, beat Gopher, GPT-3, Jurassic-1, and the 530B Megatron-Turing across the board.

The headline takeaway: most large models of the GPT-3 era were badly undertrained — too big for the data they saw. Chinchilla is the correction.

Why it matters

• It turns model training from gambling into budgeting. Before scaling laws, "should we make it bigger or train it longer?" was vibes. After, it's an optimization with a known answer. Every serious pretraining run since 2022 is planned against a Chinchilla-style budget.
• It rewrote what "state of the art" means. Chinchilla proved that a smaller, better-fed model beats a larger, starved one at equal compute. A 70B model outperforming a 530B model is a 7.5×-smaller model winning — which also makes it far cheaper to fine-tune and serve at inference. That reframed the whole cost equation.
• It is the economic engine of the LLM era. The reason labs were willing to spend tens of millions on a single run is that the loss curve was a predictable line. Scaling laws are the financial-modeling tool of frontier AI.
• It's the backdrop to every other technique. Mixture-of-Experts ([[mixture-of-experts]]) exists to cheat the dense FLOP cost that scaling laws impose. Quantization ([[quantization]]) and LoRA ([[lora-and-peft]]) exist to deploy and adapt the large models scaling laws encourage. Test-time-compute reasoning ([[test-time-compute-reasoning]]) is in part a response to scaling laws plateauing on available data. You can't understand the modern stack without this.

How it works (the real mechanics)

The two power-law regimes

When either model or data is the bottleneck and the other is plentiful, loss follows a pure power law. Kaplan parameterized them as:

L(N) ≈ (N_c / N)^α_N      with   α_N ≈ 0.076
L(D) ≈ (D_c / D)^α_D      with   α_D ≈ 0.095

Read the exponents physically: multiplying N by 10 lowers loss by a factor ≈ 0.84 (10^−0.076); multiplying D by 10 lowers it by ≈ 0.80 (10^−0.095). Small exponents — this is the "diminishing returns" of scaling baked into the math. You don't get linear improvement; you get a slowly bending log-log line. Crucially, Kaplan found architecture details (width vs depth, aspect ratio) barely matter "within a wide range" — the law is about scale, not shape.

The compute identity C ≈ 6ND

The bridge between the two regimes is a back-of-envelope accounting identity for transformer training FLOPs:

C ≈ 6 · N · D

The 6 is roughly 2 FLOPs (multiply-add) per parameter for the forward pass + ~4 for the backward pass, per token. So compute is essentially "parameters × tokens × 6." This single equation is why the allocation question — bigger N or bigger D? — is a constrained optimization: you're minimizing L(N, D) subject to 6ND = C.

The Chinchilla parametric loss

Hoffmann et al. fit a single joint function combining both regimes plus an irreducible floor:

L(N, D) = E + A / N^α + B / D^β

with the fitted constants (from the Chinchilla paper):

E = 1.69      (irreducible loss — the entropy of natural language; you can't beat it)
A = 406.4 ,  α = 0.34     (model-size term)
B = 410.7 ,  β = 0.28     (data term)

Interpretation: E ≈ 1.69 nats is the noise floor — even an infinite, infinitely-trained model can't go below it. The other two terms are the reducible loss you buy with parameters and data respectively. Because α and β are close to each other (0.34 vs 0.28), the optimization splits the budget nearly evenly between N and D — which is the entire Chinchilla insight in one line. Contrast with Kaplan, whose exponent ratio pushed the budget toward N.

Solving the constrained optimization → the ~20:1 rule

Minimize L(N, D) under C = 6ND. The solution is itself a pair of power laws in compute:

N_opt ∝ C^a ,   D_opt ∝ C^b ,   with  a = β/(α+β) ,  b = α/(α+β)

Plugging Chinchilla's α, β gives a ≈ b ≈ 0.5 — model size and data each scale as roughly the square root of compute (N* ∝ D* ∝ C^0.5). Equivalently, their ratio D/N stays roughly constant as you scale. Evaluated at the budgets they studied, that constant lands near ~20 tokens per parameter. So the practitioner's rule of thumb:

Train on roughly 20 tokens for every parameter. A 1B model → ~20B tokens. A 70B model (Chinchilla) → ~1.4T tokens.

Compare the two recipes directly:

Pseudocode-in-prose: planning a run

To plan a compute-optimal model:

1. Decide your compute budget C (FLOPs you can afford).
2. Compute N_opt ≈ k_N · C^0.5 and D_opt ≈ k_D · C^0.5 from your fitted (or borrowed) coefficients — or just shortcut via the ~20:1 ratio: solve 6 · N · (20N) = C ⇒ N ≈ sqrt(C / 120), then D ≈ 20N.
3. Sanity-check D_opt is achievable — do you actually have that many high-quality unique tokens? (Often the real constraint; see caveats.)
4. Train. The fitted L(N_opt, D_opt) is your predicted loss; deviation flags a bug in your run.

Why Kaplan and Chinchilla disagreed (the boring, important reasons)

The discrepancy wasn't philosophical — it was experimental hygiene. The leading explanations:

• Learning-rate schedule. Kaplan used a fixed cosine decay length across runs, so models trained for fewer steps than the schedule assumed were effectively under-decayed and looked worse than they should — biasing the fit toward "train less, build bigger." Chinchilla matched the cosine decay length to each run's actual token count. (Later reconciliation work — Porian et al., Pearce et al. 2024 — pins much of the gap on exactly this plus how embedding parameters are counted.)
• Counting embedding parameters. Including/excluding the (large, compute-cheap) embedding matrix in N shifts the exponents. Once you count non-embedding parameters consistently, the two laws move closer together.

The lesson: scaling laws are empirical curve fits, and the fit is only as trustworthy as the experimental protocol underneath it.

Emergent abilities — the loudest debate

A separate strand asks not "does loss go down smoothly?" (it does) but "do capabilities appear suddenly?" Wei et al. (2022) catalogued emergent abilities: tasks (multi-step arithmetic, certain BIG-Bench tasks) where models below some scale score near-random, then sharply jump to competent past a threshold — performance you "could not predict by extrapolating from smaller models."

Schaeffer et al. (2023), "Are Emergent Abilities a Mirage?", pushed back hard. Their claim: emergence is often "a mirage induced by the researcher's choice of metric," not a fundamental change in the model. The mechanism: harsh, discontinuous metrics (exact-match accuracy on a long answer — all-or-nothing) manufacture sharp cliffs, while smooth metrics (token-level log-likelihood, edit distance) on the same models reveal steady, gradual improvement. Swap the metric and the "emergence" can evaporate. They demonstrated they could even conjure fake emergence in vision tasks by choosing a nonlinear metric. This is unresolved: some abilities still look genuinely abrupt under smooth metrics, and "smooth underlying competence" doesn't make the practical threshold ("below 60B it can't do this task at all") any less real to a user.

Key ideas & tradeoffs

• Predictability is the product. The real deliverable of scaling laws isn't a number — it's forecastability. Fit small, predict large, plan budgets. That's worth more than any single coefficient.
• Compute-optimal ≠ deployment-optimal. Chinchilla minimizes loss for a fixed training budget. But you train once and serve forever. If inference dominates lifetime cost (it usually does for a popular model), you should deliberately overtrain a smaller model past its Chinchilla point — Llama-style models train on far more than 20 tokens/param precisely to get a cheaper-to-serve model. Chinchilla answers the wrong question for a production model; it's the training-efficiency optimum, not the total-cost optimum.
• The exponents are small on purpose. ~0.05–0.1 means brutal diminishing returns. Each 10× of compute buys a modest, fixed loss drop. Progress is real but logarithmic — you must keep multiplying inputs to keep subtracting loss.
• Data quality and dedup move the curve. The clean power law assumes a fixed data distribution. Better filtering, dedup, and curriculum shift the whole curve down — sometimes worth more than a compute multiple. Scaling laws describe a given data pipeline.
• It's a law of *loss*, not of *intelligence*. Lower cross-entropy correlates with better downstream behavior but does not equal it. Reasoning, factuality, and alignment are not what the curve directly optimizes.

Honest caveats & open questions

• The Chinchilla fit itself was challenged. Besiroglu, Erdil, Barnett & You (Epoch AI, 2024) attempted to replicate Hoffmann et al. and found problems: the original's parametric estimates were inconsistent with its own other two methods, fit the extracted data poorly, and reported implausibly narrow confidence intervals — "intervals this narrow would require over 600,000 experiments" when the authors "likely ran fewer than 500." Epoch's re-fit reconciled the parametric approach with the other methods. The ~20:1 conclusion survives, but the original paper's precision was overstated. Treat the exact coefficients as soft.
• The "three methods" should agree — and originally didn't cleanly. Chinchilla used three estimation routes: (1) fixing model sizes and varying tokens (training-curve minima), (2) IsoFLOP profiles (fix compute, sweep model size, find the loss-minimizing N at each FLOP level), and (3) the parametric L(N,D) fit above. Their broad agreement is the result's strength; the parametric method's mis-estimated CIs (point above) were the weak link.
• The data wall. ~20:1 assumes you have enough unique, high-quality tokens. Frontier models now want trillions of tokens — and the open web is finite. This is a hard, current constraint driving synthetic data, multi-epoch training (which breaks the single-epoch assumption), and the pivot to [[test-time-compute-reasoning]] (buy capability with inference compute instead of more pretraining data).
• Emergence is genuinely unsettled. The Schaeffer "mirage" critique is strong but not total. Whether sharp capability jumps are real physics or metric artifacts remains an active, consequential question — it bears directly on AI-safety arguments about "surprising" capabilities appearing without warning.
• Architecture-independence has limits. Kaplan's "shape doesn't matter much" holds within a band, but MoE ([[mixture-of-experts]]), state-space models ([[state-space-models]]), and attention variants ([[flash-attention]]) each have their own scaling laws with different constants. The dense-transformer law is not universal across architectures.
• The map is not the territory. These are empirical extrapolations. They held remarkably well from 2020–2024, but a power law fit over today's range carries no guarantee three orders of magnitude out. They could bend.

How it connects to OpenAlice

OpenAlice is a systems-and-orchestration project layered on top of frontier models (Codex/GPT-class and others via routing), not a pretraining lab — so scaling laws shape it mostly as constraints and economics rather than as something we re-derive:

• Why we orchestrate instead of pretrain. Scaling laws explain why a small org rationally builds on existing foundation models: the compute-optimal frontier is a nine-figure capital expense, and the loss-vs-compute curve makes that the domain of a few labs. OpenAlice's leverage is at the layer above the scaling law — routing ([[model-routing]]), councils/fusion ([[fusion-and-llm-councils]], [[mixture-of-agents]]), memory ([[agent-memory-systems]], [[mempalace]]), and test-time reasoning ([[test-time-compute-reasoning]]) — all of which buy capability without paying the pretraining scaling tax.
• Inference economics drive model choice. Because compute-optimal ≠ deployment-optimal, the practically relevant lever for Alice is picking overtrained, inference-cheap models per task. The provider config (default gpt-5.5 on mvp) and model-routing are the operational expression of "spend training compute once, serve forever" — exactly the Chinchilla-vs-deployment tradeoff above.
• The benchmark rig as a tiny scaling instrument. The internal bench platform (bench.blal.pro) measures capability-per-model across the harness — a downstream-task analogue of the same "does scale buy this ability?" question, applied to which model/prompt/harness configuration wins per task rather than to pretraining FLOPs.
• Educationally, this is a foundation node. For OpenAlice Academy this article sits upstream of nearly everything: it's the "why are models the size they are, and why did the field bet on bigness" prerequisite before [[mixture-of-experts]], [[quantization]], [[lora-and-peft]], and [[test-time-compute-reasoning]] (each of which is, in part, a response to a scaling-law pressure).

See also

• [[attention-and-transformers]] — the architecture whose FLOPs the C ≈ 6ND identity counts.
• [[mixture-of-experts]] — decouples total params from active FLOPs to cheat the dense scaling cost.
• [[quantization]] · [[lora-and-peft]] — deploy and adapt the large models scaling laws push you toward.
• [[test-time-compute-reasoning]] — buying capability with inference compute as pretraining data runs short.
• [[state-space-models]] · [[flash-attention]] — alternative architectures with their own scaling constants.
• [[llm-from-scratch]] · [[microgpt-build-an-llm-from-scratch]] — build the small model whose loss curve you'd fit.
• [[model-routing]] · [[fusion-and-llm-councils]] — system-level leverage above the scaling-law layer.