Atlas
kb://library/nanochatstable2026-06-16

nanochat — full-stack ChatGPT in ~8k lines

libraryeducationllm-from-scratchkarpathypretrainmidtrainsftrlgrpotokenizerkv-cacheinferenceend-to-endcapstone

nanochat — full-stack ChatGPT in ~8k lines

The capstone of the from-scratch path. Where [[microgpt-build-an-llm-from-scratch]] shows the algorithm in ~200 dependency-free lines and [[llm-from-scratch]] is the laptop workshop, nanochat is the whole assembly line — tokenizer → pretrain → midtrain → SFT → RL → inference engine → web UI — that takes raw internet text in one end and serves a talking ChatGPT-clone out the other. ~8,300 lines, ~$100, ~4 hours on one GPU node. Andrej Karpathy's answer to "what is the minimum complete thing that is recognizably ChatGPT?"

What it is

nanochat is a single, readable, end-to-end codebase that trains and serves a small ChatGPT-style model from scratch. It is the successor to nanoGPT (which only did the pretraining part) — nanochat adds everything that turns a raw next-token predictor into a chatbot you can actually talk to: a custom tokenizer, the full post-training ladder (chat fine-tuning + reinforcement learning), an inference engine with a KV-cache and tool-use, and a web frontend.

The headline framing — "the best ChatGPT that $100 can buy" — is half joke, half thesis. The default speedrun.sh runs the entire pipeline on an 8×H100 node (~$24/hr) in 3h 51m for ~$92.40 and produces a working, if tiny, conversational model. It does this not by being clever about cost but by being honest about scale: the model it produces is, in Karpathy's own words, a "micro model", "a little kindergartener" — knowledgeable in patches, charming, and reliably wrong at anything requiring real reasoning.

Crucially, nanochat is one cohesive repo, not a framework. No plugin system, no 40-field config object, no model zoo. ~8,300 lines across ~44 files, with ~2,000 lines of that being the Rust tokenizer. You are meant to read all of it and fork it. It is the announced capstone project for Karpathy's LLM101n course — the thing you build after you understand every piece.

Why it matters

Most "build an LLM" material stops at pretraining — you get a base model that autocompletes text, and the leap from there to ChatGPT (a model that follows instructions, holds a conversation, uses tools, and can be RL-tuned) is left as an exercise or buried inside an industrial framework (TRL, Megatron, nemo, etc.) with thousands of layers of abstraction.

nanochat's contribution is making the full post-training stack legible. For the first time you can read, in one sitting, the actual code that:

  • turns FineWeb-EDU text into a base model (pretrain),
  • teaches it the chat format, multiple-choice answering, and tool-use (midtrain),
  • aligns it to clean conversations (SFT),
  • and squeezes out extra math ability with a from-scratch GRPO loop (RL).

That arc — base → SFT → RL — is the same one used by frontier labs and described abstractly in [[rlhf-and-alignment]]. nanochat is the smallest place where you can see it run concretely, end-to-end, with a report card at the bottom. For anyone trying to understand how Alice-class agents are actually made (not just prompted), it is the single best reference implementation of the training ladder.

It is also a strong, honest baseline. Karpathy's design rule is that the out-of-the-box numbers should be a real bar to beat, not a toy — so the repo ships with a speedrun leaderboard and a reproducible report.md. That makes it a research scaffold, not just a tutorial.

How it works

The whole thing is driven by one script, speedrun.sh, which chains the stages below. The design genius is that one integer — `--depth` — sets everything else. Pick the number of transformer layers and the code derives width, head count, learning rate, batch size, weight decay, and how many tokens to train on (Chinchilla-style). The $100 run is depth=20 → ~560M parameters, 1280 hidden channels, 10 heads of dim 128.

0. The model (nanochat/gpt.py) — a modern transformer, not GPT-2

This is the most instructive file. It is not vanilla GPT-2; it quietly folds in roughly a decade of architecture improvements (see [[attention-and-transformers]] for the baseline these modify):

  • RoPE positional encoding — apply_rotary_emb(q, cos, sin), no learned position embeddings (see [[positional-encoding]]).
  • Grouped-Query Attention (GQA) — separate n_head (query heads) and n_kv_head (key/value heads), shrinking the KV-cache. See [[flash-attention]] for why this matters at inference.
  • QK-norm — q, k = norm(q), norm(k) after rotation, for training stability.
  • RMSNorm, pre-norm — F.rms_norm() applied before attention and MLP in each Block (no LayerNorm, no biases anywhere: bias=False).
  • ReLU² MLP — x = F.relu(x).square() instead of GELU. Cheaper, works fine.
  • Untied embeddings — wte and lm_head are separate weights, initialized differently (std=0.8 for embeddings vs std=0.001 for the head).
  • Logit softcap — logits = softcap * tanh(logits / softcap), bounding outputs to a stable range.
  • Sliding-window attention — a per-layer window_pattern (e.g. "SSSL" = three short-context layers then one long), so most layers attend locally.
  • Value embeddings + per-layer residual scalars (ve_gate, resid_lambdas, x0_lambdas) — ResFormer-style tricks that blend the original input back into deeper layers.
  • Flash Attention 3 on Hopper GPUs, with an SDPA fallback.

The takeaway: a "small" educational model today is a substantially better transformer than GPT-2 was, and nanochat shows exactly which knobs changed.

1. Tokenizer — rustbpe (Rust, vocab 65,536)

A from-scratch byte-pair encoding tokenizer written in Rust (built via Maturin), trained on the data. Karpathy rejected existing Python BPE trainers as "too bloated." Vocab is 65,536 (2¹⁶), compression ~4.8 chars/token. For inference it hands off to OpenAI's tiktoken for speed. This is the concrete companion to the [[tokenization]] article — the same algorithm, but production- shaped and fast. Special chat tokens (<|user_start|>, <|assistant_start|>, <|python_start|>, <|output_start|>, …) are reserved here and become load- bearing in every later stage.

2. Pretrain — scripts/base_train.py

Trains the base model on re-shuffled FineWeb-EDU shards (~24GB for the speedrun). Distinctive choices:

  • Muon + AdamW hybrid optimizer. model.setup_optimizer() routes the transformer's matrix weights to Muon (a momentum-orthogonalized optimizer that's become a speedrun favorite) and the embeddings / unembedding / scalars to AdamW. Different parameter geometries, different optimizers.
  • Depth-driven scaling laws. model_dim snaps to a multiple of head_dim; num_heads = model_dim // head_dim; target training tokens follow a Chinchilla-style target_param_data_ratio × params; batch size grows as D^0.383 off a measured depth=12 reference. This is [[scaling-laws]] turned into one line of code per quantity.
  • Three-phase LR: linear warmup → constant → warmdown over the final 65% to ~5% of peak, via get_lr_multiplier(it). LR also scales with batch size (η ∝ √(B/B_ref)).
  • Muon momentum schedule 0.85→0.97→0.90; GC manually disabled after warmup; optional FP8 matmuls on H100.
  • bfloat16 compute, fp32 master weights for optimizer stability.

3. Midtrain — scripts/mid_train.py

The under-appreciated stage, and the one that turns "autocomplete" into "chat." Same training loop as pretrain, but the data is now structured conversations rendered with the special tokens. It mixes three objectives at once (~568K rows):

  • SmolTalk (~460K rows) — general multi-turn conversational behavior; teaches the <|user_start|>…<|assistant_start|>… protocol.
  • MMLU auxiliary-train (~100K) — explicitly teaches multiple-choice answering (A/B/C/D), which the base model has never seen formatted.
  • GSM8K (~8K) — math problems whose solutions use <|python_start|>…<|python_end|> blocks, teaching the model tool-use: emit a Python expression, get the result back, continue. This is the seed of the agentic behavior explored in [[test-time-compute-reasoning]].

4. SFT — supervised fine-tuning

Further training on higher-quality conversations, with the key fix that the data is rendered in exactly the same format used at inference time — closing the train/inference mismatch that otherwise degrades chat quality. After SFT you have a usable assistant.

5. RL — scripts/chat_rl.py — GRPO, stripped to the studs

The reinforcement-learning stage, run on GSM8K, where reward = did the model get the math answer right? This is the most pedagogically valuable RL code in the wild because Karpathy strips GRPO down to its essentials and annotates every deletion:

  1. No trust region / no KL — "no KL regularization to a reference model."
  2. On-policy, so no PPO ratio + clip — "we are on policy, so there's no need for PPO ratio+clip."
  3. DAPO-style token-level normalization (not sequence-level).
  4. Advantage = `rewards − mu` — just subtract the group mean, no z-score.

What's left is essentially REINFORCE with a group-relative baseline:

# generate num_samples (16) completions per question via engine.generate_batch()
advantages = rewards - mu            # group-relative baseline, no std division
pg_obj = (logp * advantages.unsqueeze(-1)).sum()
loss   = -pg_obj                     # normalized by valid tokens

This is the cleanest on-ramp to understanding the family in [[rlhf-and-alignment]]: see what GRPO is, then add back the trust region and clipping to recover the full algorithm. The reward bumps GSM8K from 0.0455 (SFT) to 0.0758 (RL) — small in absolute terms, but a real, measurable lift from RL on a 560M model.

6. Inference — nanochat/engine.py

The serving half, designed for speed, not training:

  • `KVCache` stores keys/values in (B, T, H, D) layout (what FA3 wants), pre-allocated across all layers, with per-row cache_seqlens (int32 for FA3). Avoids recomputing attention over the whole prefix every token — the standard inference optimization (see [[flash-attention]]).
  • `Engine.generate()` runs the two-phase prefill → decode loop: one big forward pass over the prompt, then a batched token-at-a-time decode that yields (token_column, token_masks). It can prefill once and fan out to num_samples by replicating the batch=1 cache — which is exactly what the RL rollout needs.
  • Tool-use mid-generation: when the decoder emits <|python_end|>, the engine decodes the buffered expression, runs use_calculator(expr) (a sandboxed evaluator that blocks dangerous ops and **), and injects the result back wrapped in <|output_start|>…<|output_end|> so generation continues with the answer. The calculator round-trip happens inside the sampling loop.
  • `sample_next_token()`: temperature 0 → argmax; else temperature scale → optional top-k → multinomial.

7. Web UI

A ChatGPT-like web frontend (plus a CLI chat) serves the SFT/RL model so you can actually talk to the thing you built. This is the "full-stack" payoff — the loop closes from raw text shards to a chat window.

Key ideas & tradeoffs

  • One dial (`--depth`) over a config explosion. Every other hyperparameter is derived, so the model is always compute-optimal and you can't misconfigure it. Tradeoff: you lose fine-grained control, which is the point — it's a baseline, not a research harness.
  • Read-the-whole-thing > reuse-a-library. Custom Rust BPE, hand-rolled optimizer wiring, hand-rolled RL. More code to maintain, but nothing is hidden. The opposite philosophy from a TRL/Megatron stack.
  • Modern transformer as the default. RoPE + GQA + RMSNorm + ReLU² + QK-norm + sliding windows ship by default — a quiet education in what actually changed since GPT-2.
  • The full ladder, miniaturized. pretrain → midtrain → SFT → RL is the real frontier recipe, just at 560M instead of 560B. The structure generalizes even though the scale doesn't.
  • Cost as a forcing function. The $100 budget forces every honest tradeoff to the surface: you feel why frontier models cost millions, because you watch a $92 model fail at multiplication.
  • A leaderboard, not a trophy. Shipping report.md + a speedrun board makes the baseline contestable — it invites improvement instead of admiration.

Honest caveats

  • The model is genuinely weak. This is the most important honest point. The $100 depth=20 report card (Karpathy's own, from discussion #1):

| Metric | BASE | SFT | RL | |----------------|--------|--------|--------| | CORE | 0.2219 | — | — | | ARC-Easy | — | 0.3876 | — | | ARC-Challenge | — | 0.2807 | — | | MMLU | — | 0.3151 | — | | GSM8K | — | 0.0455 | 0.0758 | | HumanEval | — | 0.0854 | — | | ChatCORE | — | 0.0884 | — |

GSM8K ~5–8%, HumanEval ~9% — it barely does grade-school math or trivial Python. It is roughly GPT-2-grade knowledge with a chat skin. Karpathy is explicit: it's "a fancy autocomplete," "a little kindergartener," and is "not so sure about the color of the sky yet." Do not expect a usable assistant — expect a legible one.

  • Beware mismatched numbers in the wild. Several blog posts and community reports quote a depth=20 model at CORE ~0.41 / MMLU ~45% — those come from larger or longer community runs (or are simply errors), not the $100 speedrun. The authoritative figures are the table above. (I flag this because the web search surfaced both; the discrepancy is real and I did not reconcile them by guessing.)
  • The hardware floor is real. "$100" assumes an 8×H100 node. Without that (or spot pricing ~$15, or a much slower single-GPU run) the wall-clock and cost change a lot. nanochat is single-node, not laptop-scale — that's [[llm-from-scratch]]'s job.
  • RL is a teaching simplification. The stripped GRPO ("close to REINFORCE") is deliberately not production GRPO/PPO — no KL, no clip, no reference model. Excellent for learning, not a drop-in for frontier RLHF.
  • It's a snapshot of a fast field. The exact architecture choices (FA3, Muon, FP8, sliding-window pattern) are state-of-2025-era and will drift. Read it for the shape of the pipeline, not as a frozen recipe.
  • Source-fetch failures (full disclosure): Karpathy's discussion #1 page returned a 404 on raw fetch but its content was retrievable via the GitHub repo view (report card cross-checked there and against MarkTechPost). The typevar.dev deep-dive returned 403 Forbidden and was not used. The mid_train.py raw file 404'd on direct fetch; midtrain details here come from the README, engine.py (special-token handling), and the MarkTechPost breakdown of the data regimen — not fabricated.

How it connects to OpenAlice + the Academy ladder

The Academy ladder. nanochat is the capstone of the from-scratch path — the article you read last, after the pieces click individually:

  1. [[math-for-ml-foundations]] → [[neural-network-from-scratch]] — the calculus and backprop underneath.
  2. [[microgpt-build-an-llm-from-scratch]] — the transformer algorithm in ~200 lines, no dependencies.
  3. [[llm-from-scratch]] — a real (tiny) GPT trained on a laptop in PyTorch; you write every component.
  4. nanochat (here) — the whole assembly line: the same transformer, plus tokenizer + the full post-training ladder + a serving engine + a UI.

Along the way it touches nearly every other library article as a runnable example: [[tokenization]] (rustbpe), [[attention-and-transformers]] + [[positional-encoding]] + [[flash-attention]] (the model & KV-cache), [[scaling-laws]] (the --depth derivations), [[rlhf-and-alignment]] (the GRPO loop), and [[test-time-compute-reasoning]] (tool-use + RL-on-math). It is not a [[mixture-of-experts]] model — it's a dense transformer — so it's the clean baseline against which the MoE and [[deepseek-architecture]] articles describe departures.

Connection to OpenAlice / Alice. Alice is an agent built on hosted frontier models (Codex / gpt-5.x), so nanochat is not infrastructure we run in production — and it should not be confused with the "Alice can do autonomous coding" capability, which is an agentic capability on top of large models, not a from-scratch training stack. What nanochat is, for OpenAlice, is the single best shared mental model of how the models underneath Alice are actually made: why post-training (SFT → RL) matters more than raw pretraining for agentic behavior, why tool-use is trained in (the <|python_start|> pattern is exactly the calculator/interpreter tool-loop shape Alice uses), and why a small model's "reasoning" is so brittle. When we reason about model choice, RL alignment, or why a given behavior is hard to elicit, nanochat is the ground-truth artifact to point at. It's a teaching engine for the org, not a dependency.