LLM101n — Let's build a Storyteller

The course this whole library is shaped like. If [[nn-zero-to-hero]] is Karpathy's video path from a single neuron to GPT, LLM101n is the planned full curriculum — 17 chapters that take a complete beginner from a bigram counting table all the way to a deployed, multimodal Storyteller AI that writes and illustrates little stories, "similar to ChatGPT, from scratch in Python, C and CUDA, with minimal computer science prerequisites." It is the most on-mission item in this library: it is the ladder. Guiding quote on the repo: "What I cannot create, I do not understand. — Richard Feynman."

⚠️ Honesty up front: as of this writing (2026-06-16) the full course does not exist yet. The repo↗ is archived/read-only with a note: "this course does not yet exist. It is currently being developed by Eureka Labs. Until it is ready I am archiving this repo." What we document below is the published syllabus plus the real, shipping code Karpathy released around it ([[nanochat]], nanoGPT, micrograd, makemore, minBPE) that the chapters will almost certainly be built from. Treat this as a map of the intended course, not a walkthrough of finished lessons.

What it is

LLM101n: "Let's build a Storyteller" is the flagship course of Eureka Labs, the AI-native education company Andrej Karpathy announced in July 2024 after leaving OpenAI/Tesla. The stated deliverable is concrete and motivating: by the end, the student has built a Storyteller AI Large Language Model — a working web app, comparable to a small ChatGPT, that lets a user "create, refine and illustrate little stories with the AI." Everything is built bottom-up from first principles in Python, C, and CUDA, assuming minimal CS background.

Two things make it different from a normal ML MOOC:

1. One end-to-end artifact, not disconnected exercises. Every chapter is a load-bearing part of the same final system. Chapter 1 is a bigram table; chapter 16 deploys an inference API and web app; chapter 17 adds image generation so the storyteller can illustrate. The Storyteller is the thread that ties tokenization, attention, optimization, and RLHF into one object you can actually run.

1. AI-native delivery. Eureka Labs' thesis is a teacher + AI teaching assistant symbiosis: a human expert designs the course materials, and an AI TA guides each student through them at their own pace. Karpathy's framing — "make it easy for anyone to learn anything" — and the recursive joke that the course teaches you to build a smaller version of the very AI TA that's helping you.

The published syllabus is 17 chapters + an appendix. The repo is currently a README-only placeholder; the lessons (planned as runnable Jupyter notebooks and from-scratch C/CUDA code) are still being authored.

Why it matters

For the OpenAlice Academy this is the reference spine. Most of the from-scratch entries in this library — [[micrograd]]-style autograd, [[makemore]]-style character models, [[tokenization]], [[nanochat]] — are chapters of LLM101n that Karpathy happened to ship early as standalone repos and lectures. The course is the integration story that says why each piece exists and in what order a human should meet them.

It matters specifically because:

• It's a complete dependency graph of "what you must understand to build a modern LLM," ordered pedagogically. You can read the chapter list as a checklist for any AGI-coding or alice-core engineer who claims to understand the stack.
• It refuses black boxes. The Feynman epigraph is operational: you don't import torch.nn.Transformer, you write attention, layernorm, and a BPE tokenizer yourself. That is exactly the depth the rest of this library aims for.
• It spans the *whole* lifecycle — not just "train a transformer" but device/precision/distributed performance, kv-cache and quantization for inference, SFT + RL for alignment, deployment, and multimodality. Most courses stop at "here's attention." LLM101n keeps going until there's a product.

How it works — the 17-chapter progression

The course is bottom-up: each chapter introduces the minimum new machinery needed and reuses everything before it. Below is the published syllabus (verbatim chapter titles + the parenthetical sub-topics from the README), with the real mechanics of what each chapter teaches and the existing Karpathy code each one maps onto.

Foundations — from counting to backprop (Ch 1–3)

1. Bigram Language Model (language modeling) — The simplest possible LM: count how often character b follows character a, normalize to probabilities, sample. No gradients yet. Establishes the core objects — vocabulary, next-token distribution, sampling, negative-log-likelihood loss — that every later chapter optimizes. This is the opening of Karpathy's makemore series; there is no standalone makemore page in this library yet, so see [[nn-zero-to-hero]] for the video walkthrough.

1. Micrograd (machine learning, backpropagation) — A tiny scalar autograd engine (~150 lines): a Value that records the operations done to it and can .backward() to push gradients through the graph via the chain rule. This is the single most important conceptual chapter — once you've built backpropagation by hand, every framework afterward is just a fast, vectorized version of this. There is no dedicated micrograd page in this library yet; the from-scratch autograd idea is covered in [[neural-network-from-scratch]] and [[nn-zero-to-hero]].

1. N-gram model (multi-layer perceptron, matmul, gelu) — Generalize the bigram to predict from n previous tokens, then replace the lookup table with a small MLP: embedding → matmul → nonlinearity (GELU) → logits. This is the Bengio-2003 neural LM. Now you have a trainable neural network whose weights are learned by the micrograd-style backprop from Ch 2.

The transformer — attention, tokens, and stable training (Ch 4–7)

1. Attention (attention, softmax, positional encoder) — The heart of the course. Self-attention as content-based, position-aware mixing of token representations: queries · keys → softmax weights → weighted sum of values. Deep-dive companions in this library: [[attention-and-transformers]], [[positional-encoding]], and [[embeddings]] for the representations attention operates on.

1. Transformer (transformer, residual, layernorm, GPT-2) — Stack attention + MLP blocks with residual connections and layernorm into the full GPT-2 architecture. This is the point where the toy becomes the real thing. The reference implementation is Karpathy's nanoGPT ([[nanogpt]]); the minimal pedagogical version is [[microgpt-build-an-llm-from-scratch]], and the laptop-scale workshop is [[llm-from-scratch]].

1. Tokenization (minBPE, byte pair encoding) — How raw text becomes the integer tokens the model actually sees. Implements Byte Pair Encoding from scratch (Karpathy's minbpe): iteratively merge the most frequent byte pair. This is the deceptively important chapter — see [[tokenization]] for the full treatment, including why tokenization is the source of so many LLM "bugs."

1. Optimization (initialization, optimization, AdamW) — Why training is hard and how to make it stable: weight initialization, learning-rate schedules, and the AdamW optimizer. The difference between a model that trains and one that diverges.

Need for Speed — make it fast and big (Ch 8–10)

This trilogy is what most "build a GPT" tutorials skip, and it's where the C/CUDA half of the course lives.

1. Need for Speed I: Device (device, CPU, GPU) — How computation actually runs on hardware; moving tensors between CPU and GPU; why the GPU wins.

1. Need for Speed II: Precision (mixed precision training, fp16, bf16, fp8) — Trade numerical precision for speed and memory. Mixed-precision training, the fp16/bf16/fp8 zoo, loss scaling. See [[quantization]] for the inference-side cousin of these ideas.

1. Need for Speed III: Distributed (distributed optimization, DDP, ZeRO) — Train across many GPUs: data-parallel (DDP) and the ZeRO family of memory-sharding tricks that make billion-parameter training fit. Background on why you'd push this far lives in [[scaling-laws]].

Data, inference, and a real product (Ch 11–13)

1. Datasets (datasets, data loading, synthetic data generation) — Where the training signal comes from: building data loaders, and synthetic data generation (using an LLM to make training data for an LLM) — directly relevant to producing the stories the Storyteller learns from.

1. Inference I: kv-cache — Generation is autoregressive and would recompute the whole context every token; the kv-cache stores past keys and values so each new token is cheap. The single most important inference optimization.

1. Inference II: Quantization (quantization) — Shrink the trained model to int8/int4 so it runs cheaply at serve time. Full treatment in [[quantization]].

Alignment — turn a text-predictor into an assistant (Ch 14–15)

1. Finetuning I: SFT (supervised finetuning, PEFT, LoRA, chat) — A pretrained model predicts internet text; supervised fine-tuning on curated chat examples teaches it to follow instructions and adopt the storyteller persona. LoRA/PEFT ([[lora-and-peft]]) make this cheap by training small adapter matrices instead of all the weights.

1. Finetuning II: RL (reinforcement learning, RLHF, PPO, DPO) — Align the model to preferences with RLHF (PPO) and the simpler DPO. This is how the storyteller learns to produce stories people actually like, not just plausible text. Full context in [[rlhf-and-alignment]].

Ship it + see it (Ch 16–17)

1. Deployment (API, web app) — Wrap the model in an inference API and a web app — the ChatGPT-like front end where a user types a prompt and gets a story back. This is the "Storyteller" becoming a thing other people can use.

1. Multimodal (VQVAE, diffusion transformer) — Extend beyond text so the storyteller can illustrate: VQVAE to tokenize images and a diffusion transformer to generate them. The same next-token / denoising machinery, now over image tokens.

Appendix

A list of "further topics to work into the progression above": programming languages (Assembly, C, Python), data types, tensors, deep-learning frameworks (PyTorch / JAX), neural-net architectures (GPT, Llama, MoE — see [[mixture-of-experts.md]] in this library), and additional multimodal content. These are the cross-cutting concerns Karpathy notes don't fit cleanly into one chapter.

The real code behind the (not-yet-written) course

Because the course README is a placeholder, the honest way to "do" LLM101n today is to study the standalone artifacts Karpathy shipped that are its chapters:

The most important development since the syllabus was published is nanochat (Oct 2025). Karpathy described it as a "strong, readable, hackable baseline … and an anchor for the upcoming LLM101n course," and it is widely understood as the capstone codebase the later chapters will be built around: a single ~8,300-line repo that does tokenizer → pretrain → midtrain → SFT → RL → kv-cached inference engine → web UI, training a ~560M-param ChatGPT clone in ~4 hours / ~$100 on one 8×H100 node. In other words: chapters 1–10 of LLM101n teach you to understand the transformer; [[nanochat]] is what chapters 11–17 assemble. If you want the closest thing to "the finished LLM101n project" that exists right now, it's nanochat.

Key ideas & tradeoffs

• Build-to-understand over use-the-library. The whole pedagogy is the Feynman epigraph: you implement backprop, attention, BPE, and a kv-cache by hand before you're allowed to trust a framework. The tradeoff is speed of getting to a working model vs depth of understanding — LLM101n deliberately chooses depth, which is exactly the bet the OpenAlice library makes too.

• One artifact threads the whole curriculum. A concrete, slightly whimsical goal (a storyteller that writes and draws little stories) is a strong motivational anchor — it's narrow enough to be buildable end-to-end yet touches text generation, alignment, deployment, and multimodality. The risk is that the "story" framing is mostly motivational dressing; the underlying machinery is a general small-LLM stack.

• Full lifecycle, not just training. Including Need-for-Speed (device / precision / distributed), inference (kv-cache / quantization), and deployment is what separates this from "here's a transformer" courses. It's also what makes it genuinely hard to finish — those chapters are where C and CUDA enter.

• Bottom-up ordering is the product. The sequence bigram → micrograd → MLP → attention → transformer → tokenizer → optimization is itself the lesson: each rung is the minimum new idea on top of the previous. This ordering is the thing most beginners get wrong on their own.

Honest caveats

• The course is not finished and may never ship in its announced form. The repo has been archived/read-only with the "does not yet exist" note since 2024. Eureka Labs has shipped components and tooling but, as of 2026-06, no enrolled, end-to-end LLM101n. Do not promise NAO or anyone that "the Karpathy course" is a thing you can take today — it isn't.

• The 17-chapter list is a published *plan*, not a table of contents of existing lessons. Chapter titles and sub-topics quoted here are real (straight from the README), but no per-chapter notebooks/text are in the repo. Anything beyond the titles below the surface is our reconstruction from Karpathy's adjacent public work, and is flagged as such.

• Some sources blur "planned" and "done." Several third-party write-ups (blogs, wiki mirrors) describe LLM101n as if the course content exists, often paraphrasing the README's aspirations as features. We treat the GitHub README as ground truth and external articles as context only.

• Fetch notes (this article): the GitHub repo README and raw README, the DeepWiki overview, and Silicon Republic's Eureka Labs piece all fetched cleanly and corroborate each other. Techopedia's LLM101n explainer returned HTTP 403 and was dropped — its claims are not reflected here. The nanochat↔LLM101n capstone link is from web search summaries of Karpathy's own nanochat announcement (X/GitHub); it is consistent across multiple secondary sources but the exact "anchor for LLM101n" wording is his, paraphrased in coverage rather than fetched verbatim from a primary doc.

• C/CUDA chapters are the most speculative. The "Need for Speed" trilogy and the multimodal VQVAE/diffusion chapter are the least represented by existing standalone Karpathy code, so they're where this map is thinnest.

OpenAlice + the Academy ladder

LLM101n is the reference syllabus for the OpenAlice Academy — the spine that orders the rest of this library. A learner (or a new alice-core / AGI-coding engineer) can walk it as a rung ladder, substituting our existing pages and the real Karpathy repos for the not-yet-written chapters:

1. Numbers → networks. [[math-for-ml-foundations]] → [[neural-network-from-scratch]] → [[nn-zero-to-hero]] (micrograd + makemore = LLM101n Ch 1–3).
2. The transformer. [[attention-and-transformers]] + [[positional-encoding]] + [[embeddings]] → [[microgpt-build-an-llm-from-scratch]] → [[nanogpt]] (Ch 4–5), with [[tokenization]] for Ch 6.
3. Make it real. [[scaling-laws]] and [[quantization]] cover the Need-for-Speed + inference chapters (Ch 8–13).
4. Make it an assistant. [[lora-and-peft]] + [[rlhf-and-alignment]] for SFT/RL (Ch 14–15).
5. The capstone. [[nanochat]] is the closest existing artifact to the finished Storyteller (Ch 11–17 assembled) — train it end-to-end and you've effectively done the LLM101n project.

For OpenAlice specifically, this curriculum is the honest answer to "what does it take to build Alice's brain from scratch." Per NAO's directive that autonomous coding is a base AGI capability of Alice grown on the existing substrate, LLM101n is the literacy test: an engineer who has internalized these 17 chapters understands every layer they're standing on — from the gradient flowing through a single Value to the kv-cache serving a token to a user. That is the depth this library exists to reach.

See also

• [[nn-zero-to-hero]] — Karpathy's video course; the taught version of LLM101n Ch 1–6.
• [[nanochat]] — the full-stack ChatGPT clone widely treated as the LLM101n capstone codebase.
• [[nanogpt]] / [[microgpt-build-an-llm-from-scratch]] / [[llm-from-scratch]] — the transformer chapter, at three levels of polish.
• [[tokenization]] — LLM101n Ch 6 (minBPE), in depth.
• [[attention-and-transformers]] · [[positional-encoding]] · [[embeddings]] — the machinery of Ch 4–5.
• [[lora-and-peft]] · [[rlhf-and-alignment]] · [[quantization]] · [[scaling-laws]] — the "make it a real, aligned, fast product" chapters.