nanoGPT — Karpathy's minimal GPT pretraining repo

For NAO + anyone who wants to pretrain a real GPT, not just read about one. Where [[microgpt-build-an-llm-from-scratch]] shows the algorithm in ~200 dependency-free Python lines, and [[llm-from-scratch]] is the scaffolded learn-by-typing workshop, nanoGPT is the production-shaped minimal repo: the two ~300-line files (model.py, train.py) that actually reproduce GPT-2 (124M) on a real corpus, with torch.compile, mixed precision, and multi-GPU training. It is the bridge between "I understand a transformer" and "I trained one that's competitive with OpenAI's 2019 release for ~$10 of cloud GPU."

The one-sentence idea

A complete, fast, GPT-2-grade pretraining stack — model, training loop, data prep, sampling, finetuning — fits in two small files you can read in an afternoon, and it really reproduces GPT-2 (124M). Karpathy calls it "the simplest, fastest repository for training/finetuning medium-sized GPTs" and describes it as a rewrite of his earlier teaching repo minGPT that "prioritizes practical functionality over educational value."

That tradeoff is the whole point. minGPT and [[microgpt-build-an-llm-from-scratch]] are for understanding; nanoGPT is for doing the real thing at small scale. It is deliberately hackable: there are no abstractions, no framework, no config system beyond plain Python files — you are meant to fork it and edit the model.

What it is

Two repos, one lineage:

• `karpathy/nanoGPT` — the repo itself. model.py (~300 lines, the GPT) and train.py (~300 lines, the loop), plus sample.py, bench.py, config/ (plain-Python config files), and data/*/prepare.py tokenization scripts. Headline result: it reproduces GPT-2 124M on OpenWebText in ~4 days on a single 8×A100 40GB node, reaching validation loss ~2.85.

• `karpathy/build-nanogpt` — the teaching companion to the 4-hour YouTube lecture "Let's reproduce GPT-2 (124M)" (youtu.be/l8pRSuU81PU). It builds the same thing from an empty file, one clean git commit at a time, so you can git checkout to any point in the video. Its train_gpt2.py trains on FineWeb-Edu and evals on HellaSwag. Karpathy's framing: "we start with empty file and end up with a GPT-2 (124M) model." The lecture's claim is that with modern data + software, the 124M reproduction now costs ~1 hour and ~$10 on a rented GPU box (Lambda Labs).

Both are MIT-licensed. Neither covers chat finetuning — the README is explicit that GPT-2/GPT-3 are "simple language models… all they do is 'dream' internet documents," and that reproducing ChatGPT-style behavior is out of scope (that is what nanochat is for — see Caveats↗).

Why it matters

• It demystifies pretraining. "Pretraining a GPT" sounds like a thing only OpenAI/DeepMind can do. nanoGPT shows the loop is small: get tokens, predict the next one, backprop, repeat with a cosine LR schedule. The hard part is scale and engineering, exactly the lesson of [[scaling-laws]].
• It's the de-facto reference implementation. When people say "a GPT-2-style decoder," they usually mean this architecture: pre-LayerNorm blocks, fused QKV, 4× MLP, weight-tied embeddings. Countless forks, speedruns, and re-impls (litGPT, TinyLlama, nanochat, the modded-nanogpt speedrun, llm.c) descend from it. It is a shared vocabulary.
• It's honest about cost. The README ships real baselines (GPT-2 124M val loss 3.12, up to GPT-2-XL 1558M at 2.54) so you can check whether your training run is actually working — a rarity in tutorials.
• It's a teaching ladder, not just a repo. build-nanogpt's commit-by-commit structure + the lecture is one of the best on-ramps to real LLM engineering in existence. This is precisely the kind of artifact the OpenAlice Academy wants to stand on (see the Academy ladder↗).

How it works

model.py — the GPT (architecture walkthrough)

The model is a standard decoder-only transformer ([[attention-and-transformers]]). Config lives in a dataclass:

# GPTConfig defaults = GPT-2 124M
block_size: int = 1024      # max context length
vocab_size: int = 50304     # 50257 GPT-2 BPE tokens, padded up to a multiple of 64 for speed
n_layer: int = 12
n_head: int = 12
n_embd: int = 768
dropout: float = 0.0
bias: bool = True           # biases in Linears/LayerNorms (can disable, slightly faster/better)

The forward pass: token embeddings (wte) + learned absolute positional embeddings (wpe, see [[positional-encoding]]) → n_layer Blocks → final LayerNorm → lm_head linear to vocab logits.

Block — pre-LayerNorm residual. Each layer is the now-canonical "Pre-LN" form (LayerNorm before each sublayer, not after — more stable to train than the original "Post-LN" transformer):

x = x + self.attn(self.ln_1(x))   # attention sublayer
x = x + self.mlp(self.ln_2(x))    # MLP sublayer

CausalSelfAttention. One Linear computes Q, K, V together, then splits:

q, k, v = self.c_attn(x).split(self.n_embd, dim=2)

Each is reshaped to (B, n_head, T, head_dim). nanoGPT uses flash attention when available — torch.nn.functional.scaled_dot_product_attention (PyTorch ≥ 2.0, see [[flash-attention]]) — and falls back to a manual implementation with an explicit causal mask (self.bias[:,:,:T,:T] == 0) that blocks attending to future tokens. The causal mask is what makes it a next-token predictor.

MLP. A 2-layer feed-forward with the standard 4× expansion and GELU:

self.c_fc   = nn.Linear(n_embd, 4 * n_embd, bias=bias)  # up-project
# gelu
self.c_proj = nn.Linear(4 * n_embd, n_embd, bias=bias)  # down-project

LayerNorm is hand-rolled only so the bias can be made optional (PyTorch's built-in nn.LayerNorm doesn't support bias=None).

Two details that matter a lot for training to actually work:

1. Weight tying. The input embedding and output projection share one weight matrix — self.transformer.wte.weight = self.lm_head.weight. Saves ~38M params at 124M scale and is a well-known regularizer.
2. Scaled residual init. Most weights init N(0, 0.02²), but the output projection of each residual sublayer is scaled by 1/√(2·n_layer): std = 0.02 / math.sqrt(2 * config.n_layer). Without this, the residual stream's variance grows with depth and deep models destabilize. This is one of the GPT-2 paper's quiet but critical tricks.

`configure_optimizers` builds AdamW with two parameter groups: every 2D tensor (matmuls, embeddings) gets weight decay; every 1D tensor (biases, LayerNorm gains) gets none. It uses the fused AdamW kernel on CUDA when available.

`from_pretrained('gpt2')` loads the real OpenAI GPT-2 weights from Hugging Face, transposing the old TensorFlow Conv1D weights into nn.Linear layout — so you can sample from or finetune actual GPT-2 without training anything.

train.py — the loop (mechanics walkthrough)

This is where "minimal" earns its keep. The whole pretraining recipe is here.

Data loading is a memmap, not a DataLoader. prepare.py tokenizes a corpus into a flat train.bin / val.bin of np.uint16 token IDs. get_batch memory-maps it and grabs random windows:

ix = torch.randint(len(data) - block_size, (batch_size,))
x = torch.stack([data[i   : i+block_size]   for i in ix])
y = torch.stack([data[i+1 : i+1+block_size] for i in ix])  # targets = inputs shifted by 1

No epochs, no shuffling machinery — just uniform random offsets over a giant token stream, moved to GPU with pin_memory() + non_blocking=True. Simple and fast.

Gradient accumulation = a 0.5M-token batch on commodity hardware. GPT-2 was trained with a ~0.5M-token batch, which won't fit in memory. nanoGPT fakes it: gradient_accumulation_steps = 5 * 8 micro-steps of batch_size=12 × block_size=1024 tokens, summing gradients before one optimizer step (12 × 1024 × 40 ≈ 0.5M). Loss is divided by the accumulation count so it averages correctly. Under DDP the count is divided by world size so the global batch stays fixed.

Mixed precision. Forward passes run under torch.amp.autocast in bfloat16 (or float16 with a GradScaler on older GPUs). bf16 is the modern default — same exponent range as fp32, no loss-scaling needed.

LR schedule = linear warmup + cosine decay. warmup_iters = 2000 linear ramp to learning_rate = 6e-4, then cosine decay over lr_decay_iters = 600000 down to min_lr = 6e-5:

coeff = 0.5 * (1.0 + math.cos(math.pi * decay_ratio))  # 1 → 0
lr = min_lr + coeff * (learning_rate - min_lr)

Gradient clipping at grad_clip = 1.0 via clip_grad_norm_ (after unscaling, before stepping). `estimate_loss()` averages over eval_iters = 200 batches on both splits for a low-variance train/val number. Checkpoints (model + optimizer + config + iter_num) save to out_dir/ckpt.pt; init_from='resume' continues.

`torch.compile(model)` JITs the graph — Karpathy reports iteration time dropping from ~250ms to ~135ms (nearly 2×). Disable with --compile=False on unsupported platforms (e.g. CPU/MacBook quick start).

DDP — multi-GPU in a few lines

Distributed Data Parallel is bolted on without a framework:

init_process_group(backend='nccl')      # one process per GPU
# read RANK / LOCAL_RANK / WORLD_SIZE from env
model = DDP(model, device_ids=[ddp_local_rank])

Launch with torchrun --standalone --nproc_per_node=8 train.py config/train_gpt2.py, or across nodes with --nnodes/--node_rank/--master_addr. A neat optimization: model.require_backward_grad_sync is toggled so gradients only all-reduce on the final accumulation micro-step, not every one — cutting inter-GPU traffic ~40×.

The three runnable on-ramps

1. Shakespeare char-level (laptop/A100, ~3 min): prepare.py → python train.py config/train_shakespeare_char.py → sample.py. A 6-layer, 6-head, 384-dim model hits val loss 1.4697. There's even a pure-CPU config for a MacBook.
2. GPT-2 124M reproduction (8×A100, ~4 days → val ~2.85): python data/openwebtext/prepare.py then the torchrun command above.
3. Finetune real GPT-2: config/finetune_shakespeare.py starts from OpenAI's checkpoint with a smaller LR. Or just sample from gpt2-xl with no training.

Key ideas & tradeoffs

• Readability over abstraction. Two flat files you can hold in your head beat a configurable framework you can't. The cost: nanoGPT is a starting point to fork, not a library to depend on.
• "Simulate the big batch" via gradient accumulation is the single most reusable idea here — it decouples the statistical batch size (matters for optimization) from the physical one (limited by VRAM).
• bf16 + `torch.compile` + flash attention are the three levers that turned a multi-week 2019 training run into an afternoon. None changes the math; all change the wall-clock. This is [[scaling-laws]] from the engineering side.
• The init/tying/Pre-LN details aren't optional polish — drop the residual-init scaling and a deep model won't converge. "Minimal" still encodes hard-won tricks.
• Plain-Python configs (a .py file you exec) instead of YAML/argparse jungle: override anything from the CLI (--block_size=64) or a config file.

Honest caveats

• It is now officially deprecated. As of a November 2025 README update, Karpathy marks nanoGPT "very old and deprecated" and points to its successor, nanochat — "the best ChatGPT that $100 can buy." nanochat is a full end-to-end pipeline (Rust BPE tokenizer → FineWeb pretraining → midtraining on conversational data → SFT on MMLU/GSM8K/HumanEval → optional RL → web UI) that trains a chat model in ~4 hours for ~$100. nanoGPT only covers pretraining. If you want the chatbot, study nanoGPT to understand the base model, then move to nanochat for the assistant pipeline (which is RLHF territory — see [[rlhf-and-alignment]]).
• It reproduces 2019-era GPT-2, not a 2026 frontier model. No RoPE ([[positional-encoding]] covers the modern alternative to GPT-2's learned absolute positions), no [[mixture-of-experts]], no GQA, no long-context tricks, no [[state-space-models]]. The architecture is the floor of modern LLMs, useful precisely because it's the shared baseline — see [[deepseek-architecture]] for how far the frontier has moved past it.
• It's a base model: it "dreams documents." Out of the box it autocompletes internet text; it does not follow instructions or chat. That's not a bug — it's the definition of pretraining.
• The headline costs assume cloud A100/H100s. The "$10 / 1 hour" and "~4 days on 8×A100" numbers are real but hardware-dependent; on a single consumer GPU the 124M reproduction is slow, and on CPU only the toy Shakespeare config is practical.
• Source-fidelity note for this article: architecture/loop details are quoted from the live model.py and train.py on master and the two READMEs; exact default values (e.g. lr_decay_iters = 600000) are GPT-2-config defaults and can differ in other config files. No fetch failed during research; the lecture itself (a 4-hour video) was summarized from the repo + Karpathy's own announcement, not watched frame-by-frame.

How it connects to OpenAlice + the Academy ladder

OpenAlice's Alice is an agent built on a frontier hosted LLM — we do not pretrain our own base model. So nanoGPT's value here is pedagogical and diagnostic, not operational:

• The Academy "build an LLM" ladder. nanoGPT is the natural rung 3 of a four-step climb every OpenAlice engineer should be able to walk: 1. [[neural-network-from-scratch]] — backprop, the engine under everything. 2. [[microgpt-build-an-llm-from-scratch]] — the whole GPT algorithm in ~200 pure-Python lines (no PyTorch). Understand it. 3. nanoGPT (this article) — the same model in real PyTorch, fast, multi-GPU, reproducing GPT-2 124M. Pretrain it. The hands-on twin is [[llm-from-scratch]]. 4. nanochat — the full pretrain→SFT→RL→serve pipeline. Make it a chatbot (then read [[rlhf-and-alignment]] for the alignment stage).
• Shared vocabulary for reading any model. Once you've internalized nanoGPT's Block, fused QKV, 4× MLP, weight tying, and cosine schedule, every model card and architecture diagram in the lab becomes legible — including the advanced detours: [[flash-attention]] (the kernel nanoGPT already calls), [[positional-encoding]] (what replaced its learned wpe), [[mixture-of-experts]] / [[deepseek-architecture]] (where the frontier went), [[tokenization]] and [[embeddings]] (what prepare.py and wte actually do), and [[scaling-laws]] (why "just make it bigger" is the rest of the story).
• A cheap, honest sandbox. When the lab needs to test an idea about training dynamics — an LR schedule, a loss tweak, a sampling change ([[test-time-compute-reasoning]]) — nanoGPT's Shakespeare config gives a full train→eval→sample loop in minutes on one box, with published baselines to check against. It's the lab bench, not the product.

Bottom line: nanoGPT is the canonical "I actually pretrained a GPT" repo — deprecated in favor of nanochat for building a chatbot, but still the clearest two files in existence for seeing how next-token pretraining really runs. In the OpenAlice Academy it's the rung where the transformer stops being a diagram and becomes a training run you can launch.