makemore — your first language model

The bridge between "I understand backprop" and "I understand GPT." [[nn-zero-to-hero]] starts with a single scalar and its gradient (micrograd). It ends with a working transformer. makemore is the four lectures in the middle, and it is where most people first feel a language model click — because it makes more of something concrete: names. Feed it 32,000 baby names, and it invents new ones that sound real but never existed.

What it is

makemore is a single-purpose, one-file PyTorch tool by Andrej Karpathy. The job description fits in one sentence, and Karpathy's README gives it almost verbatim:

"makemore takes one text file as input, where each line is assumed to be one training thing, and generates more things like it."

Under the hood it is an autoregressive character-level language model. That phrase has three load-bearing words:

• character-level — the vocabulary is the set of distinct characters in your file (for names.txt: the 26 lowercase letters, plus one special start/stop token). There is no [[tokenization]] step, no subword BPE. The "tokens" are literally letters. This is a deliberate teaching choice: it removes one entire layer of machinery so you can see the model itself.
• autoregressive — it predicts the next character given the characters so far, one at a time, left to right. Generation is just: predict, sample, append, feed back in, repeat — until it emits the stop token.
• language model — formally, it learns a probability distribution P(next_char | previous_chars). That is exactly what GPT does. The only differences between makemore-the-transformer and GPT are scale and the vocabulary granularity.

The name is the thesis: it makes more of whatever you give it. Names in → plausible new names out. The bundled dataset, names.txt, is ~32,000 of the most common US baby names from the Social Security Administration's 2018 data. Swap in a file of company names, Pokémon, scientific compounds, or your own data, and it makes more of those instead. License is MIT.

Critically, makemore is not one model. It is a ladder of six models behind a single CLI flag, and that ladder is the whole point.

Why it matters

makemore is the rung of the [[nn-zero-to-hero]] curriculum where abstract backprop becomes a thing that produces output you can read and laugh at. It matters for three reasons:

1. It teaches the *invariant* of language modeling, not one architecture. The training loop, the loss (cross-entropy / negative log-likelihood), the sampling procedure, the train/dev/test discipline — all of that stays identical whether the model inside is a 2-parameter bigram counter or a 200K-parameter transformer. Once you internalize "the model is just a swappable function that maps context → next-char-distribution," GPT stops being magic.

1. It is the honest on-ramp to [[nanogpt]] and GPT. The transformer option in makemore is the same CausalSelfAttention + Block + residual structure you meet again, scaled up, in nanoGPT and in [[microgpt-build-an-llm-from-scratch]]. makemore is where you build that block for the first time, on a toy problem small enough to train on a laptop CPU in minutes. nanoGPT is the same idea pointed at the internet.

1. It compresses ~20 years of LM architecture history into one runnable file. The ladder is a literal timeline: count-based bigram → Bengio 2003 MLP → Mikolov 2010 RNN → Cho 2014 GRU → Vaswani 2017 transformer. You can --type your way through the history of the field and watch the validation loss drop at each rung.

For the OpenAlice Academy specifically, makemore is the "first language model" deliverable: the moment a learner goes from "I can train a classifier" to "I built something that generates novel sequences." Everything Alice's stack does with text is a descendant of this loop.

How it works (real mechanics + file walkthrough)

Everything lives in one hackable file: `makemore.py` (plus names.txt). That is by design — Karpathy frames it as education-first, not a production library. Here is the real anatomy.

The data layer: CharDataset

The dataset class turns lines of text into (x, y) integer tensors.

class CharDataset(Dataset):
    self.stoi = {ch: i + 1 for i, ch in enumerate(chars)}  # char -> int
    # index 0 is reserved as the special START / STOP token

Key design decisions, all load-bearing:

• `stoi` / `itos` — string-to-int and int-to-string maps. Characters get indices 1..N; index `0` is reserved as a single special token that serves as both the start padding and the end-of-sequence marker. Emitting 0 is how the model says "the name is done."
• `block_size` — the maximum context length (the longest word in the file + 1 for the stop token). This is the makemore equivalent of GPT's context window.
• Each example is returned as (x, y) where x is the padded context and y is the next-character targets shifted by one. Inactive / padding positions in y are set to `-1` so the loss can ignore them via ignore_index=-1. This masking trick is what lets variable-length names live in fixed-size tensors.

The model layer: one config, six architectures

A tiny dataclass parameterizes every model:

@dataclass
class ModelConfig:
    block_size: int = None   # context length
    vocab_size: int = None   # number of distinct chars (+1 for the special token)
    n_layer: int = 4
    n_embd: int  = 64
    # (transformer adds n_head)

Each architecture is a nn.Module whose forward(idx, targets) returns (logits, loss). The six rungs, in historical/complexity order:

1. `Bigram` — the simplest possible LM. A single lookup table mapping the previous token directly to logits over the next token. Its block_size is 1 (it only ever looks one character back). This is the neural-net version of the pure counting model from Makemore Part 1 — where you literally build a 27×27 matrix N of bigram counts and normalize rows into probabilities. The neural bigram learns the same table by gradient descent and arrives at the same answer, which is the lecture's "aha": counting and a one-layer net trained on NLL are the same model.
2. `MLP` — the Bengio et al. 2003 architecture. It embeds the previous k characters, concatenates their embedding vectors, and runs them through an nn.Sequential stack with a tanh hidden layer to produce next-char logits. This is the first model with a real notion of distributed representation — characters that behave similarly get nearby embeddings.
3. `RNN` / `GRU` — recurrent models (Mikolov 2010 / Cho 2014). An RNNCell or GRUCell takes the current input and a hidden state and returns the next hidden state, processing the sequence step by step. The hidden state is a fixed-size memory of "everything so far," which removes the MLP's fixed context-window limit — at the cost of sequential, hard-to-parallelize training.
4. `BoW` (bag of words) — a deliberately weak baseline. A CausalBoW block simply averages the embeddings of preceding tokens, using a causal mask "to ensure attention is only applied to the left" (you cannot peek at the future). It is the transformer with the interesting part — content-dependent attention — surgically removed, so you can measure how much that part buys you.
5. `Transformer` — the GPT architecture (Vaswani et al. 2017). It implements CausalSelfAttention (masked self-attention so each position attends only to earlier positions), wrapped in a Block that combines attention + a small MLP with residual connections, stacked n_layer times. The default is a ~200K-parameter transformer — small enough to train fast, structurally identical to the real thing. (For the deep mechanics of why attention works, see the attention-and-transformers article in this library.)

Note on the ladder: the makemore repo ships bigram → MLP → RNN → GRU → BoW → transformer. The [[nn-zero-to-hero]] lectures take a slightly different cut — bigram → MLP → BatchNorm/activations → manual backprop → WaveNet (a deeper, tree-structured / dilated-causal-conv evolution of the MLP). The WaveNet lecture is built in notebooks rather than as a --type in this file, so don't go looking for a --type wavenet flag — it isn't there. The repo and the course are complementary, not identical.

The training loop

Standard, deliberately unremarkable PyTorch — which is the lesson (the loop never changes, only the model inside it does):

optimizer = torch.optim.AdamW(model.parameters(), lr=learning_rate)
...
logits, loss = model(X, Y)          # forward
loss.backward()                     # backprop (the micrograd idea, autograd-scaled)
optimizer.step()                    # update

The loss is cross-entropy with `ignore_index=-1` — i.e. average negative log-likelihood of the true next character, skipping padding positions. Batches are sampled from the CharDataset; an evaluate() helper runs the model over the train and dev splits and returns the mean loss, so you can watch over/under-fitting.

Sampling / generation

# generate(): take a conditioning sequence of indices idx and
# complete the sequence max_new_tokens times.

Generation is the autoregressive loop made literal: run the model, take the logits for the last position, optionally divide by a temperature, optionally keep only the top_k candidates, softmax into probabilities, sample one character, append it, and repeat — stopping when the model emits the 0 (stop) token. Temperature near 0 makes it conservative and repetitive; higher temperature makes it wilder and more error-prone. This is the exact decoding loop a production LLM uses.

Running it

python makemore.py -i names.txt -o names

It only requires PyTorch ("torch") and defaults to the lightweight ~200K-param transformer. The argparse surface is small and tells the whole story:

• --type — bigram | mlp | rnn | gru | bow | transformer (pick your rung)
• --n-layer, --n-head, --n-embd — model size knobs
• --batch-size, --learning-rate, --max-steps — training knobs
• --sample-only — skip training, just generate from a checkpoint (so you can interactively sample names while a separate run is still training)
• --device, --seed — hardware + reproducibility

Key ideas & tradeoffs

• The loop is invariant; only the model is swappable. This is the single most important takeaway. Data → model → cross-entropy loss → AdamW → sample. Master it once on names and you have mastered it for GPT.
• NLL / cross-entropy is "how surprised is the model." Lower loss = the true next character was assigned higher probability. The bigram counting model gives you a concrete floor to beat, and every fancier rung should beat it.
• Context length is a real architectural axis. Bigram sees 1 char. MLP sees a fixed block_size window. RNN/GRU carry an unbounded (but lossy) hidden memory. The transformer sees the whole window at once and learns which earlier chars matter — that selectivity is the entire value of attention, which the BoW baseline (uniform averaging) deliberately lacks so you can measure the gap.
• Embeddings buy generalization. The jump from bigram → MLP is the jump from a raw count table to distributed representations (Bengio's contribution).
• Character-level sidesteps [[tokenization]] entirely. Great for learning; a real constraint for scale (see caveats). The follow-on lecture on tokenization exists precisely because character-level does not scale to real corpora.
• One file, MIT, education-first. No abstraction layers to fight. The tradeoff is that it is not a production framework — it is a microscope.

Honest caveats

• It is a teaching repo, not a library. Karpathy says so plainly: "one hackable file," intended for education. Don't import it into a service. For an actual minimal pretraining stack, use [[nanogpt]].
• Character-level does not scale. On real text, modeling one character at a time wastes the model's context on spelling and makes sequences enormously long. This is why every production LLM uses subword [[tokenization]] — and why the course follows makemore with a dedicated tokenizer lecture. makemore teaches the model; it deliberately avoids the tokenizer problem.
• `names.txt` is a toy. ~32K short, single-token-ish strings from 2018 US SSA data. It is perfect for fast iteration and visible results, and useless as a benchmark of real language ability. Names also carry the biases of that dataset.
• The repo ladder ≠ the lecture ladder. As noted above: the repo's rungs are bigram/mlp/rnn/gru/bow/transformer; the [[nn-zero-to-hero]] lectures route through BatchNorm, manual backprop, and WaveNet instead. If you came from the videos expecting --type wavenet, it lives in the lecture notebooks, not this file. Treat the two as a pair.
• "Generates new names that don't exist" is unverified by the tool. makemore samples plausible strings; it does not check them against the training set or the real world. Some samples will collide with real names. The novelty claim is a vibe, not a guarantee.
• No micrograd article exists in this library yet. The conceptual predecessor — micrograd, the scalar autograd engine that makes loss.backward() comprehensible — is currently covered inside [[microgpt-build-an-llm-from-scratch]] and [[nn-zero-to-hero]] rather than as its own page, so I have linked those instead of a dead [[micrograd]] slug.

OpenAlice + Academy ladder

In the [[nn-zero-to-hero]] spine, makemore is the second through fifth rungs — the entire middle of the climb:

micrograd            → backprop on one scalar      (the engine)
makemore (THIS)      → first language model on names
  ├─ part 1  bigram  → counting == one-layer net, NLL loss
  ├─ part 2  MLP     → Bengio 2003, embeddings, train/dev/test
  ├─ part 3  BatchNorm → activations, gradients, stable deep training
  ├─ part 4  backprop ninja → manual grads, no autograd
  └─ part 5  WaveNet → deeper tree/dilated-conv model
tokenization         → subwords / BPE              ([[tokenization]])
nanoGPT / GPT        → the real transformer at scale ([[nanogpt]])

Recommended path for an Academy learner (beginner → depth):

1. Read [[nn-zero-to-hero]] for the map, then make sure you've done (or read) the micrograd lecture so loss.backward() is not a black box.
2. pip install torch, clone makemore, run python makemore.py -i names.txt -o names --type bigram. Watch the loss. Read the generated names — they'll be bad. That bad baseline is the point.
3. Climb the --type ladder: mlp, then transformer. Watch the dev loss drop and the names get more name-like at each rung.
4. Open makemore.py and find the four pieces: CharDataset, the model forward, the training loop, generate(). Confirm to yourself that only the model changed between rungs.
5. Graduate to [[tokenization]] to learn why character-level doesn't scale, then to [[nanogpt]] and [[microgpt-build-an-llm-from-scratch]] to point the exact same transformer block at real data.

For OpenAlice, this is foundational literacy. Alice's text stack, the attention-and-transformers internals, and the nanochat-class full-stack training work in this library all rest on the four ideas makemore makes tangible: autoregression, cross-entropy, a swappable model, and sampling. If a contributor has built and understood makemore's transformer rung on names, the leap to reading [[nanogpt]] and the rest of the GPT-family articles here is small and concrete rather than mysterious.