microGPT — a whole LLM in ~200 lines, explained

For NAO + anyone learning how LLMs actually work. Simple but smart. Second article in the lab library. Grounded in Karpathy's microGPT (microgpt.py, Feb 12 2026). The point: the algorithm of an LLM is small and knowable; the trillion-dollar part is scale and engineering, not mystery. Karpathy frames it as "the culmination of multiple projects… a decade-long obsession to simplify LLMs to their bare essentials."

The one-sentence idea

A complete GPT — dataset, tokenizer, autograd engine, attention, training, and generation — fits in ~200 lines of pure Python with zero dependencies (no PyTorch, no NumPy, no CUDA, no Hugging Face). Karpathy: "This is the full algorithmic content of what is needed. Everything else is just for efficiency."

That sentence is the whole lesson. Once you've seen the file, an LLM stops being magic and becomes a thing you can hold in your head. The script trains and runs end-to-end on a laptop CPU in about a minute.

What it builds (the 6 pieces)

microGPT trains on 32,033 names (the classic names.txt corpus) and learns to generate new plausible ones. Six parts, each the simplest honest version of a real LLM component:

1. A dataset of documents. Each name is one "document," wrapped in a special BOS token that doubles as both start-of-sequence and the end-of-document delimiter.
2. Tokenizer (char-level). One integer per character — 26 lowercase letters + the BOS token = a 27-token vocabulary. Karpathy calls it "the simplest possible tokenizer." (GPT-4 uses BPE subword tokens over a ~100k vocab — same idea, bigger and learned.)
3. Autograd engine — the clever bit. A scalar Value class implements reverse-mode automatic differentiation from scratch in ~30 lines. This is micrograd, Karpathy's older teaching repo, folded in. Demystifies backprop completely.
4. A GPT-2-shaped transformer, gently simplified (RMSNorm instead of LayerNorm, no biases, ReLU instead of GeLU, one layer).
5. Training loop with the Adam optimizer (bias-corrected moments + linear LR decay) and per-position cross-entropy loss.
6. Inference — autoregressive sampling with a temperature knob.

The numbers that make it click

Same algorithm. The gap between this and a frontier model is scale + engineering (data volume, learned tokenizers, mixed precision, distributed training, post-training/RLHF) — not a fundamentally different idea. Note the random-chance loss of ~3.3 is exactly ln(27): a uniform guess over 27 tokens. Dropping to 2.37 means the model has learned real structure in how letters follow each other in names.

How it works — the real mechanics

1. The autograd Value class (backprop, demystified)

Every number in the network is wrapped in a Value object that records not just its .data but how it was computed:

class Value:
    __slots__ = ('data', 'grad', '_children', '_local_grads')

• .data — the forward-pass scalar.
• .grad — ∂L/∂(this value), filled in during the backward pass (starts at 0).
• ._children — the input Values this one was built from.
• ._local_grads — the local derivative of this op w.r.t. each child.

Each operator (+, *, **, exp, log, relu) returns a new Value that stashes its children and the local partial derivatives. When you finally call loss.backward(), two things happen:

1. Topological sort. A depth-first post-order traversal linearises the whole computation graph so every node comes after all the nodes it depends on.
2. Reverse sweep with the chain rule. Walking that order backwards, each node pushes gradient into its children: child.grad += local_grad * node.grad.

The += is the load-bearing detail. When a value feeds into several downstream ops (the graph branches), gradients from every path must be summed — that's the multivariable chain rule, and accumulation handles it for free. This is literally the same algorithm PyTorch's loss.backward() runs; PyTorch just does it on tensors with C++/CUDA kernels instead of one Python scalar at a time. Seeing it at scalar granularity is the entire point: backprop isn't magic, it's "multiply two numbers, then sum over paths," composed across a graph.

2. RMSNorm — normalization without parameters

microGPT replaces GPT-2's LayerNorm with RMSNorm, and even drops the learnable gain. In prose: compute the mean of the squares of the activation vector, add a tiny epsilon for stability, take the inverse square root, and scale every element by it:

ms = sum(xi*xi for xi in x) / len(x)
scale = (ms + 1e-5) ** -0.5
x = [xi * scale for xi in x]

That's it — it just keeps activation magnitudes well-behaved so training doesn't blow up. Real RMSNorm (e.g. in Llama) adds a learned per-dimension gain vector; microGPT omits it for simplicity.

3. Attention with a KV cache — "where the model routes information"

Each token's embedding is projected through learned weight matrices into a Query, Key, and Value vector. Keys and values are appended to a per-layer cache so position t can attend to all positions ≤ t:

keys[li].append(k);  values[li].append(v)

For each head, the attention logits are scaled dot-products of this position's query against every cached key, divided by √(head_dim):

attn_logits[t] = sum(q_h[j] * k_h[t][j] for j in range(head_dim)) / head_dim**0.5

Softmax those logits into weights, take the weighted sum of the cached values, concatenate across the 4 heads, project through an output matrix attn_wo, and add the residual. The √(head_dim) scaling (= √4 = 2 here) is the same trick from Attention Is All You Need — it keeps the dot-products from growing with dimension and saturating the softmax. Causality is automatic: a position only ever sees keys/values that were cached before it. Attention is routing — it decides which earlier tokens matter for predicting the next one.

4. The MLP — "where the model thinks per position"

After attention, each position passes independently through a two-layer feed-forward net that expands 4× and contracts back:

x = linear(x, mlp_fc1)    # 16 -> 64
x = [xi.relu() for xi in x]
x = linear(x, mlp_fc2)    # 64 -> 16

Plus a residual. Attention mixes information across positions; the MLP does the nonlinear per-position computation. Stack these two sub-blocks with residuals and you have a transformer block. microGPT runs exactly one such block; frontier models stack dozens to a hundred-plus.

5. Training loop + Adam

Parameters initialise as Gaussians with std 0.08. For each of 1,000 steps: pick one name, prepend/append BOS, forward-pass to get per-position logits, compute cross-entropy as the mean of -log(p(target)) across the document, backward() once, then update with Adam:

m = β1*m + (1-β1)*grad          # 1st moment (momentum)
v = β2*v + (1-β2)*grad**2       # 2nd moment (variance)
m_hat = m / (1 - β1**(t+1))     # bias correction
v_hat = v / (1 - β2**(t+1))
p.data -= lr_t * m_hat / (v_hat**0.5 + 1e-8)

with β1=0.85, β2=0.99, base lr=0.01, and linear LR decay to zero: lr_t = lr * (1 - t/num_steps). The bias-correction terms matter early on, when the moving averages m and v are still biased toward their zero initialisation. This is the real Adam (Kingma & Ba, 2014), not a toy.

6. Inference — autoregressive sampling with temperature

Start from BOS, run the forward pass to get 27 logits, divide by temperature, softmax to probabilities, and random.choices() a token weighted by those probs. Feed it back in and repeat until BOS reappears (end of name) or you hit the length cap:

probs = softmax([l / temperature for l in logits])
token_id = random.choices(range(vocab_size), weights=[p.data for p in probs])[0]

Temperature < 1 sharpens the distribution (safer, more repetitive); > 1 flattens it (more diverse, more typos). Karpathy's examples use ~0.5. This is the exact same sampling loop that drives ChatGPT — just over 27 chars instead of ~100k subwords.

How to actually learn it (the train0…train5 ladder)

Karpathy scaffolds the build as a sequence of files, each adding one idea, so you never face more than one new concept at a time. A build_microgpt.py / build gist shows the diffs between consecutive versions:

Reading them in order is the real teaching trick — and a good template for our own library articles (one idea per step, diff-able).

Key ideas & tradeoffs

• Scalar autograd vs. tensor autograd. microGPT computes gradients one scalar at a time, which is gloriously legible and catastrophically slow. PyTorch packs the same math into batched tensor ops on GPU. The algorithm is identical; the difference is 6+ orders of magnitude of throughput.
• Char-level vs. subword tokenization. A 27-char vocab needs zero tokenizer training and keeps the example tiny, but it makes sequences long and throws away the morphological priors that BPE/SentencePiece capture. The simplification is pedagogical, not free.
• RMSNorm-no-gain, no-bias, ReLU. Each simplification trims parameters and code at a small quality cost — fine for names, not what you'd ship.
• One layer, dim 16. Enough to beat the bigram baseline and learn real letter-transition structure; nowhere near enough for language. The point is to see the machine, not to be good.

Honest caveats & limitations

• It is deliberately not production code. Karpathy is explicit: "Understanding microGPT teaches you how LLMs work, but it doesn't get you closer to deploying one." No batching, no GPU, no parallelism, no mixed precision, no checkpointing, no real tokenizer training.
• Tiny task, tiny model. Generating names ≠ language modelling. There's no in-context learning, no reasoning, no instruction-following — those are emergent properties of scale, absent here by design.
• Param-count caveat. The canonical figure is 4,192 parameters (from Karpathy's post). Some secondary write-ups and an auto-summary of the gist quote "~10,000" — that's wrong; trust 4,192 for the default config. Line-count is likewise approximate: "~200" core lines, with some walkthroughs counting ~243 including comments/blanks.
• It teaches *pretraining only*. There's no SFT, no RLHF, no alignment, no tool use — the things that turn a base model into an assistant. For that lineage see nanochat (below).

The Karpathy lineage (where microGPT sits)

microGPT is one rung on a deliberate teaching ladder, each rung trading education for capability:

• micrograd — the scalar autograd engine alone (~150 lines). microGPT is micrograd + a transformer on top.
• minGPT — a clean, educational PyTorch GPT.
• microGPT — (this article) the whole thing in pure Python, zero deps, pretraining only, ~4k params.
• nanoGPT — the practical PyTorch rewrite of minGPT that "prioritizes teeth over education"; reproduces GPT-2 (124M) on OpenWebText in ~4 days on one 8×A100 node. ~300-line train script. (As of Nov 2025, Karpathy marks it superseded by nanochat.)
• nanochat — the full-stack successor (Oct 2025): tokenizer training (Rust BPE) → pretraining on FineWeb → midtraining → SFT → optional RL, plus a web UI. ~8,000 lines; a ~560M model trainable for ~$100 (≈4h on 8×H100). The capstone for Karpathy's LLM101n course.

The throughline: microGPT shows you the core algorithm; nanoGPT shows you real pretraining; nanochat shows you the whole ChatGPT pipeline including alignment.

How it connects to OpenAlice

• Demystification of our whole stack. Every model OpenAlice orchestrates — Codex (gpt-5.x), the Claude agents, the M11 council — is this, scaled. Understanding the 200 lines makes the stack legible: you reason about why models behave as they do (context windows = block_size; sampling temperature; attention = routing; KV cache = why long contexts cost memory) instead of treating them as oracles. When [[model-routing]] picks a model per task, or a prompt overflows the context window, you know exactly what's happening under the hood because you've seen the toy version.
• Sampling/temperature intuition is operational. Alice's per-chat modes and the council's diversity-vs-determinism tradeoffs map directly onto the temperature knob you can read in microGPT's 5-line sampling loop.
• It's the bedrock of the "Alice can code" capability. OpenAlice treats autonomous coding (repo → PR, SWE-bench) as a base AGI capability of Alice, not a separate product. Knowing how the underlying transformer actually computes — autograd, attention, the training loop — is the floor for reasoning about failure modes, context limits, and where fine-tuning would even help.
• A template for the library itself. The train0→train5 "one idea per file" ladder is exactly the [[deep-research]] / [[llm-maintained-wiki]] pattern: build understanding incrementally, keep each step diff-able and honest.
• Siblings. microGPT is one mind's machinery; the council is how you combine several — see [[fusion-and-llm-councils]] and [[mixture-of-agents]]. For the fuller "build a real one" path see [[llm-from-scratch]]. For how models plug into retrieval and memory rather than raw weights, see [[graphrag]], [[agent-memory-systems]], [[mempalace]], and the code-graph work in [[graphify]].

Takeaways

1. The algorithm of an LLM is small and knowable (~200 lines, ~4,192 params).
2. Backprop is not magic — it's the chain rule, accumulated over a topo-sorted autograd graph. The grad += is the multivariable chain rule.
3. Attention routes, the MLP thinks, residuals carry gradients, RMSNorm keeps the magnitudes sane — that quartet is the transformer block.
4. Adam + cross-entropy + temperature sampling are the same training and inference loops used at frontier scale, just without the engineering.
5. Frontier models differ by scale + engineering + post-training, not a secret algorithm. microGPT teaches pretraining; alignment lives one repo over in nanochat.

*Sources: Karpathy, "microGPT" (karpathy.github.io/2026/02/12/microgpt↗)

• the gist↗;

cross-checked against nanoGPT↗ and nanochat↗ for the lineage. Written via the [[deep-research]] ingest→synthesize method (M13).*