Knowledge Distillation (soft targets, sequence-level, R1-distill, on-policy)

For NAO + anyone who wants the smarts of a giant model running on hardware they actually own. A frontier model is too big and too slow to run on a laptop, a phone, or a $300/mo GPU box. Distillation is the trick that lets a small "student" model copy the behaviour of a big "teacher" model — not by compressing the weights (that's [[quantization]]), but by re-training the small model to imitate what the big one does. The 2025 surprise was that this works shockingly well for reasoning: DeepSeek took an 800k-example dump from their R1 model and turned a plain 7B model into something that beats models 10× its size at competition math. This article is the intuition, the real equations, and the honest caveats. Siblings: [[quantization]] (shrink the weights instead of re-training), [[lora-and-peft]] (cheap way to do the student fine-tune), [[deepseek-architecture]] (where R1-distill comes from), [[rlhf-and-alignment]] + [[rlvr]] (distillation's main rival for teaching reasoning), [[scaling-laws]] (why small-from-big beats small-from-scratch).

The one-sentence idea

Train a small model to match a big model's full output distribution, not just its final answer — the "wrongness pattern" of the big model carries most of the knowledge.

What it is (intuition first)

Imagine a brilliant professor (the teacher) and a bright but inexperienced student. You could teach the student by handing them a textbook with only the correct answers: "Q: capital of France? A: Paris." That's ordinary supervised training on hard labels — one right answer, everything else is wrong.

But the professor knows more than the right answer. Asked to classify a photo of a dog, the professor doesn't just say "dog (100%)". Internally they say "dog 90%, wolf 7%, cat 2.9%, car 0.0001%". That ranking — dog is a lot like a wolf, somewhat like a cat, nothing like a car — is real knowledge about how the world is shaped. Hinton called it dark knowledge: information hiding in the relative sizes of the wrong answers. Hard labels throw all of it away.

Knowledge distillation is: make the student copy the professor's full opinion, wrong answers and all. That softer, richer target teaches the small model far more per example than a one-hot label ever could. You get a small model that generalizes much better than the same small model trained from scratch.

For modern LLMs the same idea scales up. The "opinion" is now a probability distribution over ~128k vocabulary tokens at every single position in a long answer, and for reasoning models it includes the entire chain of thought. Copy that, and the student inherits not just the answers but the way of thinking.

The big mental shift: a small model trained from scratch and the same small model distilled from a strong teacher are not the same model. The distilled one is dramatically better, for free (well — for the cost of teacher inference). This is one of the most reliable "free lunches" in applied ML.

Why it matters

• It's how SLMs (small language models) get good. Almost every strong on-device / small open model you can name was distilled, at least partly, from a bigger sibling. DistilBERT (2019) was 40% smaller and 60% faster than BERT while keeping 97% of its GLUE performance — the canonical proof that you can shrink-by-imitation without falling off a cliff.
• It transfers *reasoning*, not just knowledge. This is the 2025 story. DeepSeek-R1-Distill-Qwen-7B — a 7B model fine-tuned only on R1's outputs — scores 55.5% on AIME 2024 and 92.8% on MATH-500, beating much larger non-reasoning models. The 32B version (72.6% AIME) beats QwQ-32B-Preview and approaches o1-mini. Reasoning turned out to be distillable.
• It's cheaper than RL for small models — and the paper proved it. DeepSeek ran large-scale RL directly on a 32B base (DeepSeek-R1-Zero-Qwen-32B) and it underperformed the distilled version while costing far more compute. Their blunt conclusion: "smaller models relying on large-scale RL require enormous computational power and may not achieve distillation performance." Distill first; RL is the expensive path for the few labs that can train the teacher.
• It's the only legal-ish way to "copy" a closed model's skill. Black-box distillation — prompting GPT-4-class APIs, collecting outputs, and fine-tuning an open model on them — is how a huge fraction of open instruct models were built (Alpaca, Vicuna, and their descendants). The 2024 KD survey frames this as the dominant paradigm: transferring proprietary capability into open models via data augmentation from the teacher.
• It compounds with the other efficiency tricks. Distill to get a small capable model → [[quantization]] to fit it in 4 bits → [[lora-and-peft]] to specialize it cheaply. Three orthogonal multipliers.

How it works (real mechanics)

There are four distinct families. They answer different questions.

1. Logit / response distillation (the original, Hinton 2015)

You have a teacher's logits z and a student's logits v. Soften both through a temperature-scaled softmax (temperature T > 1 flattens the distribution so the small probabilities become visible):

            exp(z_i / T)
p_i(T)  =  ─────────────────        ← teacher soft target
           Σ_j exp(z_j / T)

            exp(v_i / T)
q_i(T)  =  ─────────────────        ← student soft prediction
           Σ_j exp(v_j / T)

At T = 1 this is ordinary softmax. At T = 4 (say), the "dog 90% / wolf 7%" becomes more like "dog 55% / wolf 20%" — the dark knowledge is amplified.

The distillation loss is cross-entropy (≈ KL) between the softened teacher and softened student:

L_soft  =  - Σ_i  p_i(T) · log q_i(T)

In practice you combine it with the ordinary hard-label cross-entropy (at T = 1) so the student still respects ground truth where you have it:

L  =  α · T² · L_soft   +   (1 - α) · L_hard

The `T²` factor is not cosmetic. When you differentiate the soft loss, the gradient magnitude scales as 1/T² (the softmax-with-temperature squashes the gradients). Multiplying by T² puts the soft and hard gradients back on comparable scales so one term doesn't vanish. Hinton's gradient-matching argument: at high T, minimizing L_soft is approximately equivalent to making the student's logits match the teacher's logits in a least-squares sense — the student literally learns to reproduce the teacher's pre-softmax scores.

# Logit distillation, one step (PyTorch-ish)
import torch.nn.functional as F

def distill_loss(student_logits, teacher_logits, labels, T=4.0, alpha=0.5):
    p = F.softmax(teacher_logits / T, dim=-1)            # soft teacher targets
    log_q = F.log_softmax(student_logits / T, dim=-1)    # soft student
    L_soft = -(p * log_q).sum(-1).mean()                 # == KL up to a const
    L_hard = F.cross_entropy(student_logits, labels)     # T=1, ground truth
    return alpha * (T * T) * L_soft + (1 - alpha) * L_hard

This is white-box: you need the teacher's actual logits. Great for two models you control (e.g. a 70B and a 7B you trained), useless against a closed API that only returns text.

2. Feature / representation distillation (DistilBERT-style)

Logits aren't the only thing worth copying — the hidden states carry structure too. DistilBERT (a half-size BERT) trains with a triple loss:

L  =  L_ce     # soft-target distillation over the teacher's output distribution
   +  L_mlm    # the normal masked-language-model objective (keep ground truth)
   +  L_cos    # cosine-embedding loss aligning student & teacher hidden vectors

L_cos pulls the student's internal representation vectors to point the same direction as the teacher's — "think the same way internally," not just "answer the same way." DistilBERT also initializes the student by copying every other layer of the teacher, so it starts in a good place rather than from random. Result: 40% smaller, 60% faster, 97% of the quality. This feature approach (plus its cousin relation distillation, which matches relationships between examples rather than per-example outputs) is the white-box toolkit when you want maximum fidelity and control both models.

3. Sequence-level distillation (Kim & Rush 2016 — the LLM-relevant one)

For language, the output isn't one label — it's a sequence. Naïve word-level KD distills the next-token distribution at every position independently. But that ignores that the model is sampling a whole sentence; the student is graded token-by-token on the teacher's per-step distribution, which still leaves an exposure-bias gap (at inference the student conditions on its own prior tokens, which it never practiced on).

Kim & Rush's sequence-level KD distills over the sequence distribution instead. The clean trick that makes it tractable: approximate the teacher's full sequence distribution by its mode — run beam search with the teacher, take its single best output, and treat that generated sequence as the training target for the student. So you don't train on the human reference; you train on what the teacher would say. Their best student ran 10× faster than the SOTA teacher with little BLEU loss, gaining +4.2 BLEU (greedy) / +1.7 (beam) over a baseline — and it largely eliminated the need for beam search at inference.

This is exactly the recipe behind R1-distill and most "instruct" models today: generate a big corpus of teacher completions, then ordinary supervised-fine-tune the student on them. It needs only teacher text, so it's the black-box workhorse — you can distill from an API you can't see inside.

# Sequence-level distillation (a.k.a. "SFT on teacher generations")
for prompt in dataset:
    target_seq = teacher.generate(prompt)     # the teacher's mode (beam/greedy)
    student.train_step(prompt, target_seq)    # plain next-token cross-entropy
# That's it. The "distillation" is entirely in WHO wrote the targets.

4. On-policy distillation (the 2025–26 frontier)

Sequence-level SFT is off-policy: the student trains on trajectories the teacher generated, which the student will never produce itself. So early student mistakes lead into states the teacher never demonstrated → errors compound (the same exposure-bias problem, just at the trajectory level).

On-policy distillation fixes this: let the student sample its own rollouts, then have the teacher grade each token of the student's own output — typically via the per-token reverse-KL between student and teacher distributions. The student practices in the states it actually visits and learns to recover from its own mistakes, with a dense per-token signal (unlike RL's single sparse end-of-episode reward).

# On-policy distillation, conceptual
for prompt in dataset:
    traj = student.sample(prompt)                 # student's OWN rollout
    for t, token in enumerate(traj):
        # dense per-token signal: how far is student from teacher HERE?
        loss_t = reverse_KL(student.dist(traj[:t]), teacher.dist(traj[:t]))
    update(student, sum(loss_t))

Thinking Machines Lab reports it lifting a math student from 60% → 70% on AIME'24 in ~150 steps, at 9–30× lower cost than RL, and in a self-distillation setting learning an RL-trained policy 7–10× faster than RL itself. The trade: it's more complex than SFT and you need the teacher live in the loop (and white-box-ish access for the token-level KL). It's becoming the standard post-training step after off-policy SFT.

Key ideas & tradeoffs

White-box vs black-box (the survey's key axis): white-box uses logits/hidden states → higher fidelity (logit + feature KD), only for models you control; black-box uses only generated text → works against closed APIs, and its engine is data augmentation (have the teacher produce diverse, instruction-rich data). Most open instruct models are black-box-distilled from GPT-4-class teachers.

Temperature `T` is the main knob in logit KD. Too low → you're basically back to hard labels (no dark knowledge). Too high → the distribution is mush and you amplify noise. T ∈ [2, 5] is the usual range; the right value is task- and teacher-dependent.

Distillation vs RL for reasoning (the DeepSeek result, restated because it matters): if a strong reasoning teacher already exists, distilling it into a small model beats running RL on that small model directly — cheaper and better. RL ([[rlvr]] / [[grpo]]) is how you make the teacher; distillation is how you cheaply spread its skill. They're complementary, not competing.

You can't exceed the teacher (mostly). Standard distillation makes the student approach — not surpass — the teacher. The well-known exception is Born-Again Networks / self-distillation, where a student of the same architecture sometimes slightly beats its teacher (the soft targets act as a regularizer). On-policy + RL on top can also push a distilled student past its SFT ceiling. But "distillation alone, student > teacher" is the rare case, not the rule.

Honest caveats & open questions

• The student inherits the teacher's flaws. Biases, hallucination patterns, refusal quirks, and factual errors all distill right along with the skill. The student can't know which teacher outputs were wrong — dark knowledge includes dark misknowledge. Garbage teacher → confident garbage student.
• Reasoning distillation may be partly "format mimicry." R1-distilled models learn to produce long chain-of-thought in the right shape. How much is genuine transferable reasoning vs. learning the style of reasoning that happens to correlate with correct answers on benchmark distributions is an open question — and a reason to trust benchmark deltas less than they look.
• Benchmark contamination & overfitting to the teacher's distribution. Training on teacher generations can quietly narrow the student to the teacher's habits and the eval's distribution. The headline AIME/MATH numbers are real but the generalization outside those distributions is less certain. Treat with [[llm-evaluation]] skepticism.
• Legal & ToS gray zone. Black-box distillation from a commercial API often violates that provider's terms of service ("don't use outputs to train a competing model"). Technically trivial, legally and ethically contested. This is an unsettled area, not a solved one.
• "Capability density" / true compression limit is unknown. How small can a student go before the teacher's skill genuinely can't fit? Distillation pushes the frontier but there's no clean theory for where it stops — it's measured empirically, model by model.
• On-policy distillation is young. The cost/quality numbers above are from a handful of 2025–26 reports and a fast-moving literature; recipes (which KL direction, how to mix with off-policy SFT and RL, how to weight tokens) are not yet standardized. Promising, not settled.

How it connects to OpenAlice

• The efficiency stack is one pipeline. Distillation, [[quantization]], and [[lora-and-peft]] are the three orthogonal levers for running capable models on blal.de's no-GPU Hetzner box. Distill a big teacher → a small student, quantize the student to 4-bit GGUF, then LoRA-specialize it per task. Each multiplies the others.
• Alice's reasoning could be a distillation target, not just a prompt. Today Alice rides frontier APIs (gpt-5.x / Codex OAuth). Distillation is the bridge to a local Alice: collect her best traces (the same data the benchmark + lab rigs already capture) and sequence-level-distill them into a small on-prem student — the [[deepseek-architecture]] R1-distill recipe applied to our own teacher.
• It reframes the bench platform. bench.blal.pro measures capability; a distillation loop would use those traces as the training corpus. The [[autoresearch]] / lab rigs that already generate, score, and store model outputs are 90% of an off-policy distillation data pipeline — what's missing is the student-fine-tune step.
• On-policy distillation ≈ our self-improvement loop, done right. The lab's RL experiments ([[rlvr]], [[grpo]]) are the expensive teacher-making path. The cheaper move for spreading a capability to a smaller, faster Alice variant is on-policy distillation — dense per-token teacher feedback on Alice's own rollouts, at a fraction of RL cost. Worth a design note before any small-model push.
• Atlas itself wants small distilled models. The ingestor/inspector do lots of cheap, repetitive classification (symbol kinds, file roles, semantic tags). A distilled SLM for those narrow jobs would cut the per-call cost of keeping the index cron-fresh — classic vertical distillation from a frontier teacher into a task-specialist student.

Sources

1. Hinton, Vinyals, Dean — Distilling the Knowledge in a Neural Network (2015) — https://arxiv.org/abs/1503.02531 (the original; soft targets, temperature, `T²`, dark knowledge)
2. Kim & Rush — Sequence-Level Knowledge Distillation (2016) — https://arxiv.org/abs/1606.07947 (word- vs sequence-level; teacher-mode targets; the SFT-on-generations recipe)
3. Sanh et al. — DistilBERT (2019) — https://arxiv.org/abs/1910.01108 (triple loss = soft-target + MLM + cosine; 40% smaller / 60% faster / 97% quality)
4. DeepSeek-AI — DeepSeek-R1 (2025) — https://arxiv.org/abs/2501.12948 (distill reasoning into 1.5B–70B via 800k SFT samples; distillation > RL for small models)
5. Thinking Machines Lab — On-Policy Distillation (2025) — https://thinkingmachines.ai/blog/on-policy-distillation/ (reverse-KL on student rollouts; 9–30× cheaper than RL)
6. Xu et al. — A Survey on Knowledge Distillation of Large Language Models (2024) — https://arxiv.org/abs/2402.13116 (white-box vs black-box; algorithm / skill / verticalization; data augmentation as KD engine)