Neural Networks: Zero to Hero — the spine of the Academy

The canonical from-scratch curriculum, mapped end to end. This is Andrej Karpathy's free YouTube course nn-zero-to-hero — eight lectures that build, in spelled-out code, from one scalar and its gradient up to a working GPT and the tokenizer underneath it. We adopt it explicitly as the backbone of the OpenAlice Academy: the ladder a person climbs to stop seeing LLMs as magic.

>

The lab already has the destination articles — [[neural-network-from-scratch]] (the atom), [[microgpt-build-an-llm-from-scratch]] (a whole LLM in ~200 lines), [[llm-from-scratch]] (a real PyTorch GPT), [[tokenization]] (how text becomes tokens). This article is the map that puts them in order and ties each rung to the lecture that teaches it.

The one-sentence idea

You can build a GPT from absolutely nothing — a single multiply-and-add — by climbing one honest rung at a time, never skipping a step, never importing a magic box. Each lecture removes exactly one piece of mystery: backprop, then tensors, then a language model, then normalization, then convolution, then attention, then the tokenizer. By the end the entire stack is a thing you can hold in your head.

That is the whole pedagogical bet, and it is the same bet the lab makes about Alice: an LLM is "small and knowable"; the trillion-dollar part is scale and engineering, not a secret (see [[microgpt-build-an-llm-from-scratch]]).

What it is

nn-zero-to-hero is a course of 8 video lectures, each paired with a Jupyter notebook (the repo is 100% notebooks, MIT-licensed, ~23k★). You are meant to watch and type along — pause, re-derive, break it, fix it. There is no hand-waving: every gradient is computed by hand at least once before a framework is allowed to do it for you.

The prerequisites are deliberately low: Python basics and high-school calculus (you need to know what a derivative is; you do not need to be good at it). That low bar is the point — it is the entry to the Academy, not a graduate seminar.

The curriculum is a strict ascent. Each lecture is the simplest honest version of something real, and the next lecture is built on top of the code you just wrote:

And then the course graduates into two production-shaped repos that reuse every idea above at scale: [[nanogpt]] (pretraining a GPT-2-class model) and [[nanochat]] (the full ChatGPT stack — tokenizer → pretrain → SFT → RL → web UI — for ~$100).

Why it matters

Most ML education starts at the wrong altitude: import torch, call model.fit(), watch a number go down, understand nothing. nn-zero-to-hero does the opposite. It starts below the framework — at the level of one scalar value and its derivative — and earns every abstraction by hand before using it. By the time you call loss.backward() in lecture 7, you have written backward() twice (lecture 1 from scratch, lecture 5 by hand through a batchnorm), so the call is no longer faith; it is recognition.

For the lab specifically, this course is load-bearing in two ways:

1. It is the shared mental model. When an article here says "it's just matrix multiplies, softmax, and backprop," this curriculum is why that sentence is true and not a slogan. Every sibling article assumes the reader has, or could climb, this ladder.
2. It is the proof of the lab's core thesis about Alice's coding capability: the substrate is knowable and improvable. You don't need a mystery to build an AGI-shaped system; you need to understand the small pieces and then do the engineering. This course is the cleanest existing proof of "small and knowable."

How it works — a rung-by-rung walkthrough

Lecture 1 — micrograd: the gradient, from nothing

The whole engine is ~100 lines for autograd + ~50 for a tiny neural-net library. It introduces one class, Value, that wraps a single scalar number and remembers how it was produced:

from micrograd.engine import Value
a = Value(-4.0)
b = Value(2.0)
d = a * b + b**3
# ...
g.backward()
print(a.grad)   # ∂g/∂a, computed by walking the graph backward

Each arithmetic op (+, *, **, .relu()) builds a node in a dynamically constructed DAG and stashes a tiny local-derivative closure. .backward() topologically sorts the graph and applies the chain rule from output to inputs — this is reverse-mode automatic differentiation, the algorithm under PyTorch and JAX, stripped to its skeleton. The lecture then stacks Values into a Neuron → Layer → MLP and trains a 2-layer MLP on the two-moons dataset with SGD. The key teaching move: backprop is not a special neural-net trick — it is just the chain rule applied to a graph of + and *. This article's sibling [[neural-network-from-scratch]] walks this same Value/backward() idea to the bone with a worked one-weight example.

Lectures 2–6 — makemore: a language model, grown five times

`makemore`↗ is an autoregressive, character-level language model: feed it a text file (one item per line) and it generates more like it. The course corpus is `names.txt` — the ~32k most common US names — so "documents" are short and a laptop CPU suffices. The repo can run the whole zoo (bigram → MLP → WaveNet → RNN → LSTM → GRU → Transformer), but the lectures walk the first three plus deep-training mechanics:

• Bigram (lecture 2). The model is literally a 27×27 table of "given this character, how likely is each next character." You build it two ways — by counting and by training a single linear layer with gradient descent — and see they converge to the same thing. First contact with torch.Tensor, one-hot encoding, the negative-log-likelihood loss, and sampling. (27 = 26 letters + a start/end token; compare microGPT's 27-token vocabulary, the same trick.)
• MLP (lecture 3, Bengio et al. 2003). Now characters get embeddings in a small vector space; a context window of previous chars is concatenated and pushed through a hidden layer. This lecture is really about the craft: learning-rate finding, train/dev/test splits, diagnosing overfitting, batching.
• Activations, gradients & BatchNorm (lecture 4). The most underrated lecture. It makes you look at the histograms — of activations, of gradients — and see how a naïvely-initialized deep net dies (saturated tanh, vanishing gradients). Then it introduces Batch Normalization as the fix that made deep nets trainable, and the discipline of watching internal statistics, not just the loss.
• Becoming a backprop ninja (lecture 5). You manually backprop through the entire MLP+batchnorm forward pass — cross-entropy, the linear layers, tanh, the batchnorm, the embedding table — with no autograd. This is the lecture that pays off lecture 1: you now trust .backward() because you've reproduced it by hand.
• WaveNet (lecture 6, DeepMind 2016). The flat MLP becomes a hierarchical, tree-like network of dilated causal convolutions — context is fused progressively instead of all at once — and you meet the real torch.nn containment model (Linear, BatchNorm1d, Sequential) by re-implementing it.

Lecture 7 — "Let's build GPT": the one new idea is attention

This is the climax. Starting from an empty file and the tiny-Shakespeare corpus, the lecture builds a decoder-only transformer following Attention Is All You Need. The arc inside the lecture is itself a mini-ladder:

1. a bigram baseline (same idea as makemore lecture 2) to get the training/ sampling scaffold up;
2. the realization that a token should be able to look at past tokens — first as a clumsy average over previous positions with a for-loop;
3. that average rewritten as a lower-triangular matrix multiply (the "mathematical trick" — masking the future with -inf before softmax so position t only attends to ≤ t);
4. softmax over those scores → self-attention, then scaled dot-product attention, then multi-head;
5. stacked into blocks with feed-forward layers, residual connections, and layer normalization — the full transformer.

The companion repo `build-nanogpt`↗ keeps the commits clean and step-by-step and goes all the way to reproducing GPT-2 (124M) — the model is ~300 lines of PyTorch, the training loop another ~300. Everything here is "just" matrix multiplies + softmax + the backprop you already own. See [[attention-and-transformers]] and [[positional-encoding]] for the deep dives, and [[microgpt-build-an-llm-from-scratch]] for the dependency-free ~200-line version of this exact architecture.

Lecture 8 — the GPT Tokenizer: the unglamorous layer that explains the bugs

The final lecture builds Byte Pair Encoding from scratch (`minbpe`↗). Text is UTF-8 → raw bytes; BPE repeatedly finds the most frequent adjacent pair of tokens and merges it into a new token, growing a vocabulary that compresses the text. Three classes climb in fidelity: BasicTokenizer (BPE on raw bytes), RegexTokenizer (splits on a GPT-2/4 regex first so merges don't cross word/number/punctuation boundaries; adds special tokens), and GPT4Tokenizer (reproduces tiktoken's GPT-4 merges exactly). API is train(text, vocab_size) / encode / decode. Karpathy's thesis: "tokenization is at fault" for a startling number of LLM quirks — why models miscount letters, fumble arithmetic, choke on trailing whitespace, behave oddly in non-English. Full treatment in [[tokenization]].

Graduation — nanoGPT and nanochat

The course doesn't end at a toy. Two repos carry every idea above into something you'd actually run:

• [[nanogpt]] — the lean, hackable repo for pretraining a GPT-2-class model (the thing build-nanogpt reproduces in the lecture). ~300+300 lines, no config monsters.
• [[nanochat]] — Karpathy's "best ChatGPT that $100 can buy": the full stack — train your own BPE tokenizer → pretrain → midtrain → SFT → RL → KV-cached inference → a ChatGPT-style web UI — in one minimal, readable codebase. The headline speedrun.sh trains an end-to-end chat model in ~4 hours on an 8×H100 node (~$24/hr ⇒ ~$100); a stronger ~$1,000 / ~42-hour tier exists too. It descends from nanoGPT and borrows the gamified leaderboard idea from modded-nanoGPT (a "time-to-GPT-2" board). This is where the curriculum meets the lab's own thesis about building real systems from knowable parts.

Key ideas & tradeoffs

• No skipped rungs. The defining property: lecture n+1 is built on the code of lecture n. You never import a box you haven't opened. This is slow on purpose and it is why the understanding sticks.
• Compute the gradient by hand, then trust the framework. Backprop is taught twice (lecture 1 from scratch, lecture 5 by hand through batchnorm) before .backward() is used on faith. The framework becomes a labor-saver, not an oracle.
• Look at the histograms, not just the loss. Lecture 4's real lesson is a habit: inspect activations and gradients to diagnose training, rather than staring at a single scalar going down.
• The transformer's only new idea is attention. Everything else (embeddings, MLPs, normalization, residuals, training loop) is already in makemore. Attention is the one thing lecture 7 genuinely adds — and it's still just masked, scaled matrix-multiplies + softmax.
• Tokenization is upstream of "model" bugs. Many failures blamed on the network are actually the tokenizer. Lecture 8 reframes a whole class of LLM weirdness.
• Tradeoff — clarity over performance, always. micrograd is scalar (one number per node) — catastrophically slow versus tensorized autograd, and that's fine: it exists to be read, not run at scale. Same spirit throughout: these are teaching artifacts, deliberately un-optimized.

Honest caveats

• It is a 2022–2023 course (lecture 8 ≈ Feb 2024), and the field moved. The fundamentals — backprop, attention, BPE — are timeless, but you will not learn here: RLHF/DPO alignment ([[rlhf-and-alignment]]), Mixture-of-Experts ([[mixture-of-experts]]), FlashAttention ([[flash-attention]]), state-space models ([[state-space-models]]), quantization ([[quantization]]), LoRA/PEFT ([[lora-and-peft]]), or test-time-compute reasoning ([[test-time-compute-reasoning]]). nanochat patches a little of the modern post-training story (SFT + RL), but this is a foundations course, not a survey of 2026 SOTA. Treat it as the base of the ladder, then climb to the topic articles.
• It demands typing, not watching. Passive viewing teaches almost nothing here; the value is in pausing and re-deriving. Budget real hours per lecture, not minutes.
• Calculus is assumed, gently. You need the idea of a derivative. People who haven't touched it benefit from [[math-for-ml-foundations]] first — that's exactly why it sits one rung below this course in our ladder.
• The graduation repos need real GPUs. micrograd/makemore run on a laptop CPU. build-nanogpt/nanoGPT/nanochat assume serious hardware (multi-GPU, ideally 8×H100). You can read every line for free; running the top of the ladder costs money.
• Source-fidelity note. Numbers here are cross-checked against Karpathy's repos and the lecture descriptions. One automated fetch reported the nanochat speedrun as "~$48 / 2h"; the canonical figure is ~$100 / ~4h on 8×H100 (with a ~$1,000 / ~42h stronger tier), confirmed against the nanochat README and announcement discussion — we use the canonical numbers above.

How it connects to OpenAlice + the Academy ladder

This course is the spine of the OpenAlice Academy — the lab's education path that climbs from "what is a neuron" to the systems that run Alice. The lab already authored the destination articles; nn-zero-to-hero is the ordered route between them. The Academy ladder, made concrete:

math-for-ml-foundations          ← prerequisite (derivative, vector, matrix)
        │
neural-network-from-scratch      ← Academy rung 0  (≙ micrograd, lecture 1: the atom)
        │
nn-zero-to-hero (THIS)           ← the spine: lectures 1→8, no skipped rungs
   ├─ micrograd ............ backprop          → neural-network-from-scratch
   ├─ makemore ............. LM, MLP, BN, conv
   ├─ build GPT ............ attention/transformer → attention-and-transformers,
   │                                                  positional-encoding
   ├─ tokenizer ............ BPE              → tokenization
   └─ graduate ............. nanoGPT, nanochat → nanogpt, nanochat
        │
microgpt-build-an-llm-from-scratch  ← the whole LLM in ~200 lines (no deps)
llm-from-scratch                    ← a real ~10M-param PyTorch GPT on a laptop
        │
modern topics: rlhf-and-alignment · mixture-of-experts · flash-attention ·
               quantization · lora-and-peft · test-time-compute-reasoning ·
               state-space-models · scaling-laws
        │
lab systems: fusion-and-llm-councils · mixture-of-agents · model-routing ·
             agent-memory-systems · mempalace · graphrag · embeddings

Why the lab cares, concretely: the same from-scratch ethos drives Alice's own coding capability. The lab's stated view is that an LLM is not a black box — its algorithm is "small and knowable" ([[microgpt-build-an-llm-from-scratch]]), and the hard, valuable part is scale and engineering. nn-zero-to-hero is the cleanest existing proof of that thesis: it walks a learner from one scalar's gradient to a working GPT without ever invoking magic. The very loop that taught one weight to descend a loss is, at scale, what lets Alice predict the next token, write code, and improve from feedback.

For a person joining the lab or NAO onboarding a collaborator: start at [[math-for-ml-foundations]] if calculus is rusty, then [[neural-network-from-scratch]], then walk these eight lectures in order. Come out the other side and [[microgpt-build-an-llm-from-scratch]], [[llm-from-scratch]], [[attention-and-transformers]], and [[tokenization]] all read as review — and the whole stack stops being magic and becomes a thing you can hold in your head.

Sources

• [karpathy/nn-zero-to-hero](https://github.com/karpathy/nn-zero-to-hero) — the course repo (8 lectures, notebooks, MIT). Primary source for the curriculum arc and per-lecture contents.
• [karpathy/micrograd](https://github.com/karpathy/micrograd) — ~100-line scalar autograd + ~50-line nn lib; the Value class, .backward(), and two-moons MLP of lecture 1.
• [karpathy/makemore](https://github.com/karpathy/makemore) — character-level autoregressive LM on names.txt; the bigram→MLP→WaveNet→…→Transformer zoo of lectures 2–6.
• [karpathy/build-nanogpt](https://github.com/karpathy/build-nanogpt) — clean, step-by-step from-scratch reproduction of GPT-2 (124M); the code spine of lecture 7 ("Let's build GPT").
• [karpathy/minbpe](https://github.com/karpathy/minbpe) — minimal BPE (BasicTokenizer / RegexTokenizer / GPT4Tokenizer, train/encode/decode); the GPT-tokenizer lecture 8.
• [karpathy/nanochat](https://github.com/karpathy/nanochat) — full-stack ChatGPT clone (tokenizer→pretrain→SFT→RL→web UI) for ~$100; the graduation repo, with the canonical $100 / ~4h on 8×H100 speedrun confirmed via its README + announcement.