Synthetic data generation

What it is (intuition first)

Every capable model is, in the end, a function of the data it was trained on. The problem the field ran into is blunt: high-quality human data is finite, expensive, and — for frontier-scale pretraining — nearly exhausted. Synthetic data generation is the answer that stuck: instead of paying humans to write instructions, solutions, and conversations, you use a language model to produce them, then train (or evaluate) on the result. The data is "synthetic" not because it is fake but because its author is a model rather than a person.

The intuition that makes this work at all is that a strong model already contains far more usable knowledge than it reliably exhibits. A base model has read the internet; it knows how to write a SQL tutorial, a tricky algebra problem, a polite refusal — it just hasn't been shaped to do so on demand. Synthetic data generation is the discipline of extracting that latent competence into explicit, trainable examples: prompt the model in the right way, harvest its outputs, filter the good from the bad, and feed the survivors back as supervision. Done well, a 175B model can teach a 7B model most of what it knows ([[distillation]]); done well, a model can even teach a better version of itself by generating problems harder than any in its original training set.

The mental model:

Generate candidates (prompt a model for instructions / problems / solutions) → Filter them (dedup, validity, reward/verifier, quality+diversity) → Train on the survivors (SFT, preference, or RL). The whole craft is in the generate prompt and the filter — raw generation alone drifts toward bland, repetitive, low-diversity slop.

The thing to hold onto: synthetic data is not free quality. It is a lever — it amplifies and reshapes what a model already knows, and (as the model-collapse work below shows) it can amplify the model's flaws just as faithfully if you remove the human anchor.

Why it matters

It breaks the human-data bottleneck. Instruction tuning originally depended on hand-written instruction sets — "limited in quantity, diversity, and creativity," as the Self-Instruct paper put it. Synthetic generation turns a few hundred seed examples into tens or hundreds of thousands, cheaply, on demand. This is the reason the open instruction-tuning ecosystem exists.
It is how small models inherit big-model competence. Distillation-style synthetic data (a strong teacher writes the answers; a small student trains on them) is the dominant way [[small-language-models]] punch above their parameter count. Most "it's tiny but surprisingly good" models are tiny models trained on a larger model's synthetic output.
It targets capabilities you can't scrape. You can't web-scrape "50,000 logical reasoning problems matched to verified solutions." You can synthesize them. For math, code, tool-use, and reasoning, synthetic data with verifiable answers is the substrate that feeds [[rlvr]] and [[self-play-and-verifier-driven-training]].
It is the raw material for evaluation too. Synthetic test sets (held-out, freshly generated) sidestep [[benchmark-contamination]] — but only if the generator isn't the model under test, which is its own subtle trap ([[llm-evaluation]]).

How it works (real mechanics)

1. Self-Instruct — bootstrap from the model's own generations

The foundational recipe (Wang et al., ACL 2023, arXiv 2212.10560) starts from a small seed set of human-written tasks (the paper used 175) and runs an iterative bootstrap: prompt the model to generate new instructions from the seeds, then for each instruction generate an input and an output, then filter — remove instructions too similar to existing ones (ROUGE-L overlap threshold), malformed instances, and so on — and add the survivors back to the task pool to seed the next round. Applied to vanilla GPT-3, this produced a +33% absolute improvement on Super-NaturalInstructions, putting it on par with InstructGPT-001 — with almost no human labeling beyond the seeds. Alpaca (Stanford) popularized the recipe by running it against a stronger teacher. The lasting lesson: a model can supervise itself if you control generation diversity and ruthlessly filter.

2. Distillation & Evol-Instruct — teach with a stronger model, then make it harder

Two upgrades dominate the modern recipe:

Distillation framing. Rather than the model teaching itself, a stronger model (the teacher) writes the outputs and a weaker/smaller model trains on them. Mechanically identical to Self-Instruct's instance generation, but the supervision quality is bounded by the teacher, not the student ([[distillation]]). This is where most open-model instruction data actually comes from.
Evol-Instruct (WizardLM, Xu et al., ICLR 2024, arXiv 2304.12244) attacks the complexity axis. Plain synthetic instructions cluster around easy, generic tasks. Evol-Instruct prompts an LLM to evolve existing instructions along two axes: in-depth evolving (add constraints, deepen, concretize, increase reasoning steps, complicate the input) and in-breadth evolving (mutate into a new, related task). Starting from Alpaca's 52K and running ~4 evolution rounds yielded ~250K instructions of graded difficulty; fine-tuning LLaMA on them produced WizardLM, which the authors report beat human-created instructions on a complexity-balanced test bed. The insight that generalizes: diversity and difficulty are things you must deliberately engineer into the generator — they do not emerge for free.

3. Persona-driven generation — scale diversity by changing who asks

Even Evol-Instruct hits a diversity ceiling: keep prompting one model and outputs converge. Persona Hub (Chan et al., Tencent AI Lab 2024, arXiv 2406.20094) reframes the problem. Inject a persona into the prompt ("you are a pediatric nurse…", "you are a competitive-math coach…") and the same model surfaces a different slice of its latent knowledge each time. The paper auto-curated 1 billion personas from web text via Text-to-Persona (infer a persona that would write/read a given text) and Persona-to-Persona (derive related personas through interpersonal relationships), then used them as "distributed carriers of world knowledge" to synthesize 50K math problems, 50K logical-reasoning problems, 50K instructions, NPCs, tools, and more. The technique is elegant and the headline figures are vendor/self-reported (a Tencent technical report) — "1B personas ≈ 13% of the world's population" is a framing, not a validated capability claim. Flagged as such. The transferable idea is real: vary the generating perspective to escape mode collapse in the data.

4. Rejection sampling — generate many, keep only what passes a verifier

For tasks with a quality signal (a reward model, unit tests, a math checker), the strongest pattern is rejection-sampling fine-tuning (a.k.a. RAFT / iterative best-of-n; RLHFlow/RAFT). For each prompt, sample N candidate completions, score them (reward model or verifier), keep only the top one(s), and fine-tune on those — the same SFT loss, but the data is now self-generated and filtered to the model's own best behavior. LLaMA-2 used exactly this (sample K, score with a reward model, train on the winners) before its PPO stage, and Llama 3's report leans heavily on rejection-sampled data. The variant where you don't train — just pick the best of N at inference — is best-of-N ([[sampling-and-decoding-strategies]], [[test-time-compute-reasoning]]). For verifiable domains this loop is essentially how synthetic data merges into [[rlvr]] and [[process-outcome-rewards]]: the verifier is the filter, and only correct trajectories survive.

5. Quality + diversity filtering — the real work

Across every recipe above, generation is cheap and filtering is where quality is won or lost. The standard toolbox:

Dedup / near-dedup — n-gram or embedding similarity ([[embeddings]]) to drop near-duplicate instructions (Self-Instruct's ROUGE-L gate is the canonical example); this is what protects diversity.
Validity / format checks — discard malformed instances, empty outputs, instruction-input mismatches.
Reward / verifier filtering — keep only high-reward or test-passing completions (rejection sampling).
LLM-as-judge filtering — score candidates with a judge model and threshold ([[llm-evaluation]]); powerful but inherits the judge's biases.
Decontamination — strip anything overlapping eval benchmarks ([[benchmark-contamination]]).

The recurring failure mode is optimizing quality while destroying diversity: aggressive reward filtering converges the dataset onto a narrow, high-scoring mode, and the trained model becomes confidently bland. Quality and diversity are in tension, and managing that tension is the craft.

Key ideas & tradeoffs

Generation is easy; filtering is the moat. Every serious recipe is 10% prompt and 90% filter (dedup, validity, reward, decontam). The quality of synthetic data is almost entirely the quality of the filter.
Diversity must be engineered, not assumed. Left alone, a generator collapses toward its own mode. Evol-Instruct (complexity) and Persona Hub (perspective) are two different deliberate answers to the same diversity problem.
A student rarely exceeds its teacher — on the teacher's distribution. Pure distillation is bounded by the teacher. You exceed the ceiling only when an external signal enters: a verifier, unit tests, real-world reward ([[rlvr]], [[self-play-and-verifier-driven-training]]). Synthetic-from-self with no external anchor is where collapse risk concentrates.
Verifiable domains are the sweet spot. Where correctness is checkable (math, code), rejection sampling turns synthetic data into near-free, high-quality supervision. Where it isn't (open-ended writing, "helpfulness"), you fall back to a judge model and inherit its blind spots.
Cheaper than humans, but not free of bias. Synthetic data faithfully reproduces — and can amplify — the generator's biases, refusals, and stylistic tics. You are not removing human judgment; you are baking one model's judgment into the next.

Honest caveats & open questions

Model collapse is real — and its scope is contested. Shumailov et al. (Nature 2024) showed that indiscriminately training generative models on recursively generated data causes irreversible degradation: the tails of the distribution vanish first, then within a few generations outputs converge to repetitive nonsense. This is the canonical warning. But the framing is contested: Gerstgrasser/Schaeffer et al. (2024, arXiv 2404.01413, NeurIPS) show the collapse result depends on replacing real data with each generation's synthetic data — and that if you instead accumulate synthetic data alongside the original real data, test error has a finite upper bound independent of iteration count, i.e. collapse no longer occurs. Since web-scale data is not deleted en masse in practice, the practical danger is narrower than the headline. The honest synthesis: replacement collapses, accumulation with a real-data anchor mostly doesn't — keep real data in the mix and don't train purely on the latest model's output.
The "1B personas" / "beats human instructions" numbers are self-reported. Persona Hub is a Tencent technical report; WizardLM's superiority claims are the authors' own human/auto evals. Real techniques, vendor-flavored magnitudes — treat the methods as solid and the numbers as advocacy.
Decontamination is harder than it looks. A generator trained on the web can emit near-copies of benchmark items; synthetic eval sets are only contamination-safe if the generator provably hasn't seen the benchmark ([[benchmark-contamination]]).
Judge-filtered data inherits the judge. LLM-as-judge filtering scales beautifully and smuggles in the judge model's biases, length preferences, and sycophancy ([[llm-evaluation]], [[verifiers-and-reward-hacking]]). A reward model that can be gamed produces synthetic data that games it.
Diversity loss is silent. Aggregate quality metrics go up while the dataset quietly narrows; you often don't notice the lost long tail until the trained model fails on an out-of-mode input. There is no single number that catches this.

How it connects to OpenAlice

OpenAlice's training and evaluation substrate is shaped by exactly these mechanics:

The bench + fixture-runner loop is rejection sampling in spirit. OpenAlice runs an internal UI-bench platform and module-specific fixture-runners; generating agent trajectories, scoring them, and promoting only the green ones is the generate→filter→train loop applied to agent behavior rather than instruction text ([[agent-evaluation]], [[self-play-and-verifier-driven-training]]).
The AGI-coding capability is verifier-filtered synthesis. Alice's autonomous repo→PR work measured against SWE-bench is a verifiable domain: tests are the filter, so synthetic coding trajectories that pass tests are exactly the rejection-sampled, high-quality supervision the literature recommends ([[rlvr]], [[agentic-rl-long-horizon-coding]]).
Personality preservation is the anti-collapse constraint, named differently. The house rule that Alice's Soul/Identity content is preserved verbatim and that personality drift is a feature, not a target is, mechanically, a refusal to let a model recursively overwrite its own behavioral anchor with synthetic self-output — the "keep the real-data anchor, don't train purely on the latest generation" lesson from the collapse debate.
Distillation down the cost ladder. OpenAlice's cost-ladder research rig (strong models generating, smaller models serving) is distillation-style synthetic data in production form: a stronger model's outputs become the cheaper model's training signal ([[distillation]], [[small-language-models]], [[model-routing]]).

The throughline: synthetic data is the lever that lets OpenAlice manufacture the supervision it can't scrape — provided every generated example passes a real filter (a test, a verifier, a human-anchored eval) before it is allowed to shape a model.

Synthetic data generation — self-instruct, distillation & Evol-Instruct, persona-driven scale, rejection sampling, quality+diversity filters, and the model-collapse risk