llm.c — GPT training in raw C/CUDA

One sentence. llm.c trains GPT-2 / GPT-3-class language models in plain C and hand-written CUDA — no PyTorch, no Python interpreter at training time — and uses that minimalism to make every byte of memory and every GPU kernel visible and editable, culminating in reproducing the 1.5B-parameter GPT-2 for roughly $672 on a single 8×H100 node in 24 hours.

This is the systems / performance capstone of the from-scratch ladder. Where [[nanogpt]] teaches you what a GPT is in ~600 lines of readable PyTorch, llm.c strips the framework away entirely and asks: what is the machine actually doing, kernel by kernel, byte by byte, to make that math run fast on a GPU? If [[nanogpt]] is the spec, llm.c is the metal.

What it is

llm.c (by Andrej Karpathy, MIT-licensed) is a single-repo implementation of GPT pretraining with a deliberate constraint stated in the README: "LLM training in simple, pure C/CUDA — no need for 245MB of PyTorch or 107MB of cPython." The whole training stack is ~5,000 lines of code. It compiles in seconds, has a constant memory footprint, trains in mixed precision (bf16), is bitwise deterministic, scales multi-node over NCCL, and hovers around ~50% MFU (Model FLOPs Utilization — the fraction of the GPU's theoretical peak FLOP/s you actually keep busy).

The repo is organized as a teaching gradient from "obvious" to "fast":

The philosophy is explicit and worth internalizing: keep the root folder simple and readable; allow complexity only in dev/. Karpathy notes he will accept a cuBLAS call for a real speedup but resists adding "500 lines of complex C code" for marginal gains — the mainline is a teaching artifact first and a benchmark second.

Why it matters

1. It collapses the abstraction stack. PyTorch hides ~five layers between your loss.backward() and the silicon: autograd, the dispatcher, ATen, cuBLAS/cuDNN, and the CUDA runtime. llm.c removes all of them. You write the backward pass by hand. You allocate the activations by hand. When the loss is wrong, there is no framework to blame — the bug is in your chain rule. This is the single most clarifying exercise in deep-learning systems.

2. It is the canonical "training got cheap" demonstration. In 2019 GPT-2 (1.5B) was a frontier model OpenAI initially withheld. Five years later llm.c trains an equivalent for ~$672 (Karpathy, July 2024). The cost collapse came from three compounding forces, all visible in the code: compute (H100 tensor cores), software (cuBLASLt, cuDNN/FlashAttention, bf16 — see [[flash-attention]]), and data quality (FineWeb-EDU). This is the most concrete artifact behind the [[scaling-laws]] story and the "compute is the input" framing.

3. It is bitwise deterministic and self-verifying. Because there is a PyTorch reference and C/CUDA implementations of the same math, llm.c can assert that its forward and backward passes match PyTorch to tolerance. Determinism + a golden reference is exactly the discipline a serious systems project (or an autonomous coding agent) needs.

4. It is faster than the framework it replaces. With cuBLASLt, TF32, CUDA graphs, and kernel fusion (PR #89, contributor ademeure), the GPU path beat PyTorch Nightly by ~7% in the mainline and ~2× in early experiments — proof that the abstraction tax is real and recoverable when you control the kernels.

How it works (real mechanics + code walkthrough)

The CPU reference (train_gpt2.c) — read this first

This ~900-line file is the Rosetta Stone. Everything else is a faster spelling of it. Its structure:

Four structs hold the entire model.

• GPT2Config — hyperparameters: max_seq_len, vocab_size, num_layers, num_heads, channels.
• ParameterTensors — 16 weight tensors: token embeddings wte, position embeddings wpe, the two layernorms' weights/biases per block, the fused QKV projection, the attention output projection, and the MLP's two linear layers.
• ActivationTensors — 23 intermediate buffers for the forward pass (encoded input, layernorm outputs, attention scores/probs, GELU activations, logits, probs, losses).
• GPT2 — the master struct: config + params + grads + AdamW optimizer state (m and v moment buffers) + activations + current batch.

Memory is one contiguous block, by hand. malloc_and_point_parameters() allocates one slab for all weights, then sets each struct pointer into offsets within it. Activations are allocated the same way, lazily on the first forward pass. This is the trick behind the "constant memory footprint" claim — there is no allocator churn per step, just one buffer reused forever. Reading this is the moment the phrase "model weights" stops being abstract: they are a single float* you could memcpy to disk.

The forward pass is one function per layer primitive, each a tight loop that mirrors the math:

encoder_forward     // out[b,t,:] = wte[token] + wpe[t]   (sum token + positional embeddings)
layernorm_forward   // normalize over channels, then scale (gamma) and shift (beta)
matmul_forward      // y = x @ W^T + b   (with 8x loop unrolling on CPU)
attention_forward   // causal multi-head self-attention in 4 passes:
                    //   (1) Q·K^T scores, (2) softmax over the causal window,
                    //   (3) weighted sum of V, (4) concat heads
gelu_forward        // approximate GeLU
residual_forward    // elementwise add (the skip connection)
softmax_forward     // numerically stable: subtract the row max before exp
crossentropy_forward// negative log-likelihood of the true next token

gpt2_forward() simply chains these: embed → for each of N blocks do ln → attn → residual → ln → mlp → residual → final layernorm → logits → softmax → loss. Variable naming is terse-but-honest: B batch, T time/sequence, C channels, NH num-heads, hs head-size, Vp padded-vocab. Comments cite the PyTorch docs they reproduce (reference: torch.nn.LayerNorm).

The backward pass is the chain rule, written out longhand. gpt2_backward() reverses the forward order and accumulates gradients into the parameter-gradient slab. The kickoff comment is the whole insight in one line: "we kick off the chain rule by filling in `dlosses` with `1.0f/(BT)"* — i.e. the mean over the batch is just a constant the gradient starts from. The attention backward is again 4 passes, mirroring forward in reverse (softmax gradient → value gradient → query/key gradients). Writing attention_backward` by hand once teaches more about autograd than a year of using it.

The optimizer is AdamW, ~20 lines. gpt2_update() walks the single parameter slab, maintains bias-corrected first/second moments (m, v), applies the update with decoupled weight decay, and that's it. No optimizer abstraction, no parameter groups — just a loop over a float array.

The loop loads a checkpoint, spins up a DataLoader over the .bin token files, then for each step: forward → backward → update, with periodic validation-loss evals and autoregressive sampling (XORshift RNG + multinomial over the vocabulary) so you can watch the model learn to babble.

From CPU to GPU: the dev/cuda bench

The CPU file is correct but slow. The GPU work happens kernel-by-kernel in dev/cuda/, where each layer primitive gets its own standalone .cu file with multiple progressively-faster versions and inline documentation. This is the part that turns "I know CUDA syntax" into "I can make a kernel fast." The optimization arc (largely from PR #89 onward):

• matmuls → cuBLASLt, with the bias and GELU fused into the epilogue so you don't round-trip through global memory just to add a bias.
• attention → fused scale + softmax kernels, and optionally cuDNN FlashAttention (off by default to keep compile time and dependencies low; enable with make train_gpt2cu USE_CUDNN=1). Flash attention is the memory-bandwidth win explained in [[flash-attention]].
• TF32 path so cuBLASLt/cuBLAS numerics match PyTorch when you want exact parity.
• CUDA graphs + custom streams to overlap CPU launch overhead with GPU work.
• Kernel fusion generally — collapse layernorm's mean/variance/normalize/scale/shift into fewer global-memory passes.

Concrete effect cited in PR #89: a step on an RTX 4090 dropped from ~65ms to ~34ms — competitive with PyTorch's ~36ms — and on H100s the mainline runs ~7% faster than PyTorch Nightly at ~50% MFU.

Mixed precision + distribution (train_gpt2.cu)

The bleeding-edge trainer adds: bf16 master compute with fp32 where it matters for stability; gradient accumulation to hit a large effective batch on limited VRAM; multi-GPU via NCCL + MPI with a ZeRO-style optimizer-state shard; and multi-node with three rendezvous options (OpenMPI, shared filesystem, or TCP sockets). It stays bitwise deterministic — a non-trivial engineering feat once you have async reductions across 8 GPUs.

The reproductions (the headline results)

The 124M command (Discussion #481) shows the knobs that map 1:1 onto the math above:

./train_gpt2cu \
  -b 64 \        # micro-batch size (per-GPU, fits in VRAM)
  -t 1024 \      # sequence length T
  -d 524288 \    # total batch ≈ 0.5M tokens (via gradient accumulation)
  -l 0.0006 \    # max learning rate 6e-4
  -u 700 \       # warmup steps
  -c 0.1 \       # weight decay
  -v 250 -n 5000 # validate every 250, checkpoint every 5000

The 1.5B command (Discussion #677) is the same shape, just deeper and longer: -b 16 -t 1024 -d 1048576 -l 0.0006 -u 700 -x 32000 -e "d48" under mpirun -np 8 — a depth-48 model, 1M-token batches, 32k steps.

Key ideas & tradeoffs

• Readability *is* the architecture. The single contiguous parameter/activation slab isn't just a perf trick — it makes "what is a model" inspectable. You can dump weights with one fwrite. Frameworks hide this; llm.c makes it the first thing you see.
• Hand-written backward as pedagogy. You will never again treat .backward() as magic after writing attention_backward and matmul_backward by hand. This is the irreplaceable lesson.
• Golden reference + determinism = trustworthy systems. Every C/CUDA path is asserted against PyTorch. This is the pattern every serious training stack (and every autonomous coding agent generating systems code) should copy.
• MFU is the real scoreboard. ~50% MFU means half the GPU's FLOPs are wasted to memory bandwidth, launch overhead, and non-matmul work. The whole dev/cuda arc is a war to claw that number up — and it teaches you why matmuls fuse with their epilogues and why attention must be memory-aware ([[flash-attention]]).
• Cost is a software artifact, not just a hardware one. The $672 number is as much about FineWeb-EDU data quality and cuDNN/bf16 as it is about H100 silicon. Better data → fewer tokens to a given loss → fewer dollars.
• Tradeoff: mainline simplicity vs. peak speed. Karpathy deliberately leaves performance on the table in the root folder to keep it teachable, pushing the gnarly optimizations into dev/. That's a feature, not a bug — but it means the "fast" path (train_gpt2.cu) is genuinely harder to read than train_gpt2.c.

Honest caveats

• C/CUDA, not C++ niceties. This is raw pointer arithmetic and manual memory management. A single off-by-one in an offset corrupts weights silently. The flip side of "no framework" is "no guardrails."
• Pretraining only. llm.c reproduces the base model. There is no instruction tuning, no RLHF, no chat ([[rlhf-and-alignment]] is a separate stage). The 1.5B model babbles fluent text; it is not a chatbot. For the chat-grade successor that does fine-tune, see [[nanochat]] (Karpathy's later project, which trains a GPT-2-grade chat model for ~$73 in ~3h on 8×H100 — a different, higher-level pedagogy).
• The dollar figures assume cloud H100/A100 access. "$672" is real but presupposes you can rent an 8×H100 node (Karpathy used Lambda at ~$14–$28/hr depending on node). On a single consumer GPU the 124M run is "4–24 hours"; the 1.5B run is impractical.
• It is a moving target / research codebase. llm.c is bleeding-edge and was refactored heavily over 2024. LOC counts (~5k total, ~900–1000 for train_gpt2.c), flags, and even the optional cuDNN path drift between commits. Treat exact numbers here as accurate-as-of-mid-2024 and re-check the README for your checkout.
• Not where you start. Coming in cold to CUDA + hand-written backprop simultaneously is brutal. Do [[nanogpt]] (or [[microgpt-build-an-llm-from-scratch]] / [[llm-from-scratch]]) first so you know what the code computes before you fight how fast it computes it.
• `dev/cuda` depth varies. Some kernels have three polished, documented versions; others are a single working implementation. The "every kernel is beautifully documented" claim is aspirational at the edges.
• Related-but-distinct: llama2.c. Karpathy's earlier llama2.c (Llama-2 inference in a single ~700-line C file, no CUDA) is a sibling project often confused with this one. There is no llama2c article in this library yet; if you want the inference-in-pure-C story, that's the repo to read — it's the inference counterpart to llm.c's training focus.

OpenAlice + Academy ladder

Where it sits in the from-scratch ladder. This is the capstone of the systems track, the rung after you understand the model and want to understand the machine:

neural-network-from-scratch  →  micrograd / nn-zero-to-hero        (autograd, by hand)
        ↓
attention-and-transformers  →  positional-encoding  →  tokenization (the architecture)
        ↓
nanogpt / microgpt / llm-from-scratch                              (a GPT in readable PyTorch)
        ↓
flash-attention  +  scaling-laws  +  quantization                 (the perf & cost theory)
        ↓
★ llm.c ★   — the theory, made real in C/CUDA: kernels, mixed precision, MFU, $672
        ↓
nanochat                                                          (post-training: chat, RL, ~$73)

It pairs directly with three siblings:

• [[nanogpt]] — the PyTorch original llm.c is a C port of. Read them side by side: same math, two languages, two abstraction levels.
• [[flash-attention]] — the single most important kernel optimization llm.c can switch on (USE_CUDNN=1). The "why attention is memory-bound" intuition lives there; the "here's the kernel call" lives here.
• [[scaling-laws]] — llm.c's reproduction table (124M→350M→1.5B, $20→$200→$672) is scaling laws you can run. The cost-vs-capability curve stops being a chart and becomes a shell command.

Also relevant: [[quantization]] (the precision spectrum bf16/fp8 that llm.c lives on the bf16 end of), [[tokenization]] (the .bin uint16 token streams llm.c consumes), and [[nanochat]] (the next rung up — what to do after you have a pretrained base).

Why this matters for Alice / OpenAlice specifically. Per the org's standing direction, autonomous repo→PR coding is a base AGI capability of Alice, measured against SWE-bench-style harnesses — not a separate product. llm.c is a near-perfect target domain and training mirror for that capability, for three reasons:

1. It is a self-verifying systems codebase. PyTorch-reference parity + bitwise determinism is exactly the kind of golden-test scaffold an agent needs to safely modify low-level code. An agent that can keep test_gpt2.cu green while editing a CUDA kernel is demonstrating real systems competence — the kind our bench platform (bench.blal.pro) is built to measure.
2. It is the honest floor on "how cheap is a model." When Alice reasons about cost, compute budgets, or whether to train vs. call an API, the $672/124M-$20 numbers here are the ground truth, not vibes. They anchor the [[scaling-laws]] reasoning in dollars.
3. It is the right depth for a curriculum. For any agent (or human engineer) onboarding to the lab's ML stack, the llm.c → nanochat path is the canonical "from kernel to chatbot" ramp. It connects the abstract ([[attention-and-transformers]]) to the concrete (a float* of weights you can fwrite) in a way no framework tutorial can.

Academy use: assign train_gpt2.c as the close-reading exercise (trace one token through gpt2_forward, then derive attention_backward on paper before reading it), then have the student flip USE_CUDNN=1 and measure the MFU delta themselves. The lesson lands when the abstraction tax becomes a number on their own screen.