minGPT — the minimal, readable GPT (architecture teaching reference)

For NAO + anyone who wants to *read* a GPT — not the algorithm in raw Python ([[microgpt-build-an-llm-from-scratch]]), and not the fast pretraining rig ([[nanogpt]]), but the clean, idiomatic *PyTorch* model that maps one-to-one onto the [[attention-and-transformers]] diagram. minGPT is the version you open when you want to point at a line and say "that is the causal mask, that is the residual stream, that is the MLP." It is Karpathy's "small, clean, interpretable and educational" GPT in ~300 lines, the ancestor of nanoGPT, and still the clearest single file for learning the GPT-2 architecture itself.

The one-sentence idea

A faithful PyTorch re-implementation of OpenAI's GPT where the entire model — embeddings, multi-head causal self-attention, transformer blocks, and the language-model head — is ~300 readable lines, deliberately written for humans to learn from, not for speed. Karpathy's framing of GPT itself is in the README: "feed a sequence of indices into a Transformer, and a probability distribution over the next index in the sequence comes out." minGPT is the smallest honest codebase that does exactly that with real GPT-2 weights.

minGPT is semi-archived (since Jan 2023). Karpathy points new work to nanoGPT, which "prioritizes practical functionality over educational value." So the rule of thumb is: read minGPT to understand, run [[nanogpt]] to pretrain. This article is the understand half.

What it is

minGPT is three small files plus a couple of demo projects:

Demo projects in projects/ and notebooks:

• demo.ipynb — GPT + Trainer on a toy sorting task (the "hello world").
• projects/adder — trains a GPT from scratch to do integer addition (an homage to the GPT-3 paper's arithmetic experiments).
• projects/chargpt — a character-level LM you can train on any text file.
• generate.ipynb — loads pretrained GPT-2 and samples text from a prompt.

Install is the standard editable clone:

git clone https://github.com/karpathy/minGPT.git
cd minGPT && pip install -e .

The key design choice is the split: `model.py` is GPT; `trainer.py` is just PyTorch boilerplate that could train any network. That separation is itself a teaching move — it tells you which parts are "the idea" and which parts are plumbing every neural net needs. (This is the same line nanoGPT later draws between model.py and train.py.)

Why it matters

Three reasons it earns a permanent spot in the library:

1. It is the clearest architecture map. If you've read the concept in [[attention-and-transformers]] and want to see it as code that runs and loads the real GPT-2-124M checkpoint, minGPT is the most direct correspondence. Class names and tensor shapes are chosen to be legible, not clever.

1. It's the canonical ancestor. nanoGPT is explicitly a rewrite of minGPT. Understanding minGPT means you can read nanoGPT, [[nanochat]], and most "GPT in N lines" repos instantly — they share this skeleton. minGPT became so widely referenced (courses, books, blogs) that Karpathy froze it precisely because so many people learn from it.

1. It places the right rung on the learning ladder. [[microgpt-build-an-llm-from-scratch]] shows the algorithm with its own autograd and zero dependencies (you learn what backprop is). minGPT assumes PyTorch gives you autograd and lets you focus purely on the transformer. [[nanogpt]] then assumes you know the transformer and focuses on training it fast and for real. minGPT is the middle rung.

How it works (real mechanics + file walkthrough)

The whole model is "indices in → next-token distribution out." Here is each moving part as it actually appears in model.py.

Config and model variants

GPT.get_default_config() returns a config you populate with a model_type (e.g. 'gpt2') or explicit n_layer / n_head / n_embd. The class ships a table of named sizes, from tiny to GPT-2-XL:

• gpt-nano — 3 layers, 3 heads, n_embd 48 (trains on a CPU in minutes; this is the one you debug with)
• gpt-micro, gpt-mini — slightly larger toys
• gpt2 — 12 layers, 12 heads, n_embd 768 → the real 124M model
• … up to gpt2-xl — 48 layers, 25 heads, n_embd 1600

You set vocab_size (50257 for GPT-2's BPE) and block_size (1024 = max context length). Dropout defaults are 0.1 on embeddings, attention, and residual paths.

NewGELU — the activation

A self-contained module implementing the tanh-approximation GELU used by the original BERT/GPT TensorFlow code:

0.5 * x * (1.0 + tanh( sqrt(2/π) * (x + 0.044715 * x³) ))

It's spelled out by hand (rather than nn.GELU) so the file matches OpenAI's numerics exactly — a small but honest detail.

CausalSelfAttention — the heart

This is the one block worth reading line by line. The mechanics:

1. One fused QKV projection. A single c_attn = nn.Linear(n_embd, 3*n_embd) produces query, key, and value in one matmul, then .split() into Q, K, V. (Doing it in one layer is just efficiency; conceptually it's three projections.)
2. Reshape into heads. Each of Q/K/V goes from (B, T, C) to (B, n_head, T, head_size) so the heads attend in parallel.
3. Scaled dot-product scores. att = (Q @ Kᵀ) / sqrt(head_size) → shape (B, n_head, T, T).
4. Causal mask. A lower-triangular tril buffer (registered, not a parameter) sets every "future" position to -inf before softmax, so token t can only see tokens ≤ t. This single masked_fill is what makes the model autoregressive — the difference between GPT and BERT lives here.
5. Softmax + dropout over the last dim, then att @ V to mix values.
6. Re-assemble heads back to (B, T, C) and project out through c_proj, with residual dropout.

That's the entire attention mechanism — no flash kernels, no fused ops. (For the fast version see [[flash-attention]]; nanoGPT swaps in scaled_dot_product_attention.)

Block — the repeating unit

The transformer block is pre-norm (LayerNorm before each sub-layer), with two residual connections:

x = x + attn(ln_1(x))     # attention sub-block
x = x + mlp(ln_2(x))      # MLP sub-block

The MLP is the standard 4× expansion: Linear(n_embd → 4*n_embd) → NewGELU → Linear(4*n_embd → n_embd) → Dropout. The x = x + ... form makes the residual stream explicit — each block reads from and writes to a shared channel, which is exactly the mental model that makes deep transformers trainable.

GPT.forward — putting it together

1. Token IDs → wte token-embedding lookup → (B, T, n_embd).
2. Positions 0..T-1 → wpe learned position embedding (GPT-2 uses learned, absolute positions — contrast the rotary/relative schemes in [[positional-encoding]]).
3. Sum + embedding dropout → run through the stack of Blocks → final ln_f.
4. lm_head (a Linear(n_embd → vocab_size)) projects to logits.
5. If targets are passed, it computes the cross-entropy loss (logits vs. the next token) in one line — that's the whole training objective.

Weight init and the optimizer trick

Two details that are easy to miss but matter:

• Init. All linears N(0, 0.02); LayerNorms to weight 1 / bias 0. Residual projections (c_proj) get scaled init N(0, 0.02/sqrt(2·n_layer)) — this keeps the residual stream's variance from blowing up as depth grows (a GPT-2 paper detail faithfully reproduced).
• `configure_optimizers`. Parameters are split into two groups: matmul weights get weight decay (L2), while biases, LayerNorm params, and embeddings get *no* weight decay. Decaying a LayerNorm gain or an embedding is the kind of silent mistake this method exists to prevent. It returns an AdamW optimizer with betas (0.9, 0.95).

from_pretrained — loading real GPT-2

A classmethod pulls official GPT-2 checkpoints (small→XL) from Hugging Face and copies the weights in. The one wrinkle: HF's GPT-2 uses Conv1D layers whose weights are transposed relative to nn.Linear, so the loader transposes those specific tensors while shape-checking everything. After this you have a working GPT-2 you can sample from — proof the minimal model is the same architecture, not a toy lookalike.

generate — autoregressive sampling

The decode loop: crop context to block_size → forward to get logits → take the last position → divide by temperature → optional top-k filter → softmax → multinomial sample (or argmax if greedy) → append the token → repeat. This ~15-line loop is the entire inference story; the same shape appears in every LLM serving stack.

trainer.py — generic, on purpose

The Trainer contains no GPT-specific code — "boilerplate that could apply to any arbitrary neural network." Defaults: device auto-detect, num_workers=4, batch_size=64, learning_rate=3e-4, betas=(0.9,0.95), weight_decay=0.1, grad_norm_clip=1.0. The run() loop is a clean five-step rhythm: pull a batch from a DataLoader (sampling with replacement, so it's an infinite stream) → forward to get loss → backward() → clip_grad_norm_ → optimizer.step(). A small callback system (add_callback / set_callback / trigger_callbacks, firing on_batch_end) lets demos plug in logging/checkpointing/eval without touching the loop — a tidy lesson in keeping the training core untouched.

bpe.py — GPT-2's tokenizer, demystified

A refactor of OpenAI's GPT-2 encoder, and a great read for [[tokenization]]:

• bytes_to_unicode() maps all 256 bytes to printable Unicode so intermediate strings never contain control chars (this is why GPT-2 tokens show up as Ġ for space, etc.).
• A regex pre-tokenizer (contractions, ?\p{L}+, ?\p{N}+, punctuation, whitespace) splits text into chunks before merging.
• bpe() greedily merges the lowest-ranked bigram (per vocab.bpe) until no merge applies, memoizing results; encode/decode round-trip text ↔ the 50,257 integer IDs. get_encoder() lazily downloads encoder.json + vocab.bpe into ~/.cache/mingpt/.

Key ideas & tradeoffs

• Readability over speed — explicitly. No torch.compile, no flash attention, no mixed precision, no DDP. Every tradeoff is "make the code clearer." That is the feature, and the exact line nanoGPT later crosses for "teeth."
• Model vs. infrastructure separation. model.py (the idea) vs. trainer.py (plumbing) vs. bpe.py (data prep). Knowing which file owns what is half the education.
• Single fused QKV + registered causal-mask buffer — small idioms, but they're the ones every later GPT repo copies; learning them here makes other repos free.
• Faithful GPT-2 numerics — hand-written GELU, scaled residual init, two-group weight decay, Conv1D transpose on load. minGPT teaches the details that actually matter, not a hand-wave.
• Tiny configs are first-class. gpt-nano exists so you can train a real GPT on a laptop CPU and watch loss go down — the fastest path from "I read it" to "I ran it."

Honest caveats

• Semi-archived (Jan 2023). It's frozen on purpose. Don't expect new features, modern attention, or current PyTorch idioms. For anything you intend to run at scale, use [[nanogpt]] / [[nanochat]].
• Not fast and not meant to be. Training anything past the toy demos is slow; there's no multi-GPU, no kernel fusion. It's a microscope, not an engine.
• Old-style absolute learned positions. Great for matching GPT-2; not how 2026 models do it (rotary / ALiBi — see [[positional-encoding]]). Read it knowing the field moved on.
• No tokenizer training. bpe.py uses GPT-2's pretrained merges; it doesn't learn a vocabulary from scratch. For training a tokenizer see [[tokenization]].
• Architecture only — no alignment/RLHF, no scaling story. minGPT is the pretraining-objective transformer. The post-training layer is [[rlhf-and-alignment]]; why bigger helps is [[scaling-laws]]. Don't mistake the base model for the whole pipeline.
• My source reads were the README + the three `mingpt/` files via raw GitHub + one WebSearch for the minGPT↔nanoGPT context. All fetches succeeded; no failures to report. Some figures (exact head-table for every variant, precise file line counts) are summarized from those reads — verify against the live repo before quoting a number in anger.

OpenAlice + Academy ladder

Where this sits for us:

• Academy rung 3 of 4 (the "read a real GPT" rung). The intended path: [[neural-network-from-scratch]] / [[math-for-ml-foundations]] → [[microgpt-build-an-llm-from-scratch]] (algorithm, own autograd) → minGPT (the same architecture as clean PyTorch you can point at) → [[nanogpt]] (train GPT-2 for real) → [[nanochat]] (the full chat-model pipeline). [[attention-and-transformers]] is the concept companion to read alongside model.py; [[tokenization]] pairs with bpe.py; [[nn-zero-to-hero]] is the video track that builds straight to this point.
• The canonical "show me the lines" reference. When an OpenAlice agent (or NAO) needs to explain exactly where causal masking / the residual stream / the LM head live, this is the file to cite — small enough to paste, faithful enough to trust.
• Mental model for our own stack. Alice's generation isn't minGPT, but the shape — tokenize → transformer → logits → temperature/top-k sample → append — is the same loop. Reading generate() here is the cheapest way to build intuition for what a serving layer ([[model-routing]], [[quantization]] for the fast/cheap side) is wrapping. For why the production models dwarf this one, [[scaling-laws]]; for the cleverer attention they use, [[flash-attention]].

Bottom line: minGPT is the microscope. Read it to understand the GPT architecture as living PyTorch, then graduate to [[nanogpt]] to actually train one.