Atlas
kb://library/microgradstable2026-06-16

micrograd — scalar autograd from scratch

libraryeducationautogradbackpropagationfrom-scratchneural-networkchain-rulereverse-modekarpathyfundamentalsacademy

micrograd — scalar autograd from scratch

The smallest possible thing that is genuinely backpropagation. Roughly 150 lines of Python — a ~100-line autograd Value engine and a ~50-line neural-net library on top — that trains a real neural network the same way PyTorch does, just one scalar at a time. Karpathy's own framing: "These 94 lines of code are everything that is needed to train a neural network. Everything else is just efficiency."

>

This is the deepest pedagogical ROI on the whole ladder. If [[neural-network-from-scratch]] teaches you what a network is and [[math-for-ml-foundations]] teaches you the calculus, micrograd is where you finally see the gradient flow — node by node, closure by closure — with nothing hidden. A beginner who reads this to the end and reimplements the Value class understands reverse-mode automatic differentiation, which is the engine under every model in this library.

What it is

micrograd (karpathy/micrograd, MIT) is a tiny scalar-valued automatic differentiation engine plus a small neural-network library with a PyTorch-like API. Two files do the real work:

  • `micrograd/engine.py` — the Value class (~100 lines). A Value wraps a single Python float, and arithmetic on Values silently records a computation graph (a DAG) of how each value was produced. Call .backward() on the final value and it walks that graph in reverse, filling in .grad (the derivative of the final value with respect to that node) on every node.
  • `micrograd/nn.py` — Neuron, Layer, MLP (~50 lines). A multi-layer perceptron built entirely out of Value operations, so the whole network is one big expression graph that .backward() can differentiate end-to-end.

There is no NumPy in the hot path, no tensors, no vectorization, no GPU. Every neuron weight is its own Value; every multiply-add in a dot product is a separate node in the graph. That is the entire point: it is explicit to the point of being slow, so that nothing is magic.

It is also the companion code to Karpathy's first *Neural Networks: Zero to Hero* lecture ("The spelled-out intro to neural networks and backpropagation: building micrograd"), which builds the whole thing live from an empty file — covered in this library as [[nn-zero-to-hero]].

Why it matters

Modern deep-learning frameworks (PyTorch, JAX, TensorFlow) are enormous, and their autograd is buried under tensor abstractions, dispatch layers, and CUDA kernels. It is entirely possible to train models for years and never understand how the gradients get computed. micrograd removes every layer of that. What is left is the irreducible core:

  1. A graph of operations is built as a side effect of doing forward math.
  2. The chain rule is applied locally at each operation.
  3. Reverse-mode means you compute the output first, then propagate sensitivities backwards from output to inputs.

That trio is backpropagation, and it is identical in micrograd and in a 70-billion parameter LLM. The only difference upward from here is efficiency — batching scalars into tensors, fusing ops, running on accelerators. So micrograd is the conceptual floor: once you have it, every other article in this library ([[attention-and-transformers]], [[flash-attention]], [[microgpt-build-an-llm-from-scratch]], [[nanogpt]]) is "the same gradient, made fast."

For the OpenAlice Academy this is the keystone rung: the cheapest possible artifact that delivers a permanent mental model. You can read all 150 lines in one sitting.

How it works (real mechanics + code walkthrough)

The Value — a scalar that remembers where it came from

Every Value stores its number plus the bookkeeping needed to differentiate it:

class Value:
    """ stores a single scalar value and its gradient """

    def __init__(self, data, _children=(), _op=''):
        self.data = data            # the actual number (forward value)
        self.grad = 0               # d(final output) / d(self), filled in by backward()
        self._backward = lambda: None   # local backprop fn for the op that made this Value
        self._prev = set(_children)     # the Values this one was computed from (graph edges)
        self._op = _op                  # which op produced it ('+', '*', 'ReLU'…) — for debugging/viz

Four ideas are packed in here:

  • data is the forward number.
  • grad is the gradient, initialized to 0 and accumulated during backprop.
  • _prev is the set of parent nodes — the edges of the DAG, built as you compute.
  • _backward is a closure: a per-node function that knows how to push gradient from this node to its parents. This is the trick that makes the whole thing tiny.

Operations build the graph and capture a local derivative

Each operator does three things: compute the forward result, record the parents, and define a _backward closure that applies the chain rule for that one operation. Add:

    def __add__(self, other):
        other = other if isinstance(other, Value) else Value(other)
        out = Value(self.data + other.data, (self, other), '+')

        def _backward():
            self.grad  += out.grad
            other.grad += out.grad
        out._backward = _backward

        return out

For c = a + b, the local derivatives are dc/da = 1 and dc/db = 1, so each parent simply inherits out.grad (chain rule: local-derivative × upstream-grad = 1 × out.grad). Multiply is the product rule:

    def __mul__(self, other):
        other = other if isinstance(other, Value) else Value(other)
        out = Value(self.data * other.data, (self, other), '*')

        def _backward():
            self.grad  += other.data * out.grad   # d(a*b)/da = b
            other.grad += self.data  * out.grad   # d(a*b)/db = a
        out._backward = _backward

        return out

Power and ReLU complete the set of primitives:

    def __pow__(self, other):
        assert isinstance(other, (int, float)), "only supporting int/float powers for now"
        out = Value(self.data**other, (self,), f'**{other}')

        def _backward():
            self.grad += (other * self.data**(other-1)) * out.grad   # d(x^n)/dx = n·x^(n-1)
        out._backward = _backward
        return out

    def relu(self):
        out = Value(0 if self.data < 0 else self.data, (self,), 'ReLU')

        def _backward():
            self.grad += (out.data > 0) * out.grad   # gradient passes only where active
        out._backward = _backward
        return out

Notice there is no `__sub__`, `__truediv__`, or `tanh` primitive — those are derived from the four above:

    def __neg__(self):        return self * -1
    def __sub__(self, other): return self + (-other)
    def __truediv__(self, o): return self * o**-1

Subtraction is "add a negation," division is "multiply by a reciprocal." So the entire autograd surface is just `+`, `*`, `, and relu** — four local derivatives, and everything else composes. (The __radd__/__rmul__/__rsub__ etc. are there so that 2 * x and 1 + x` work, i.e. when a raw Python number is on the left.)

backward() — topological sort, then reverse

The magic step. You can only correctly back-propagate into a node after every node that depends on it has been processed. micrograd guarantees that with a topological sort of the DAG, then walks it in reverse:

    def backward(self):
        # topological order of all of the children in the graph
        topo = []
        visited = set()
        def build_topo(v):
            if v not in visited:
                visited.add(v)
                for child in v._prev:
                    build_topo(child)
                topo.append(v)
        build_topo(self)

        # go one variable at a time and apply the chain rule to get its gradient
        self.grad = 1                 # seed: d(output)/d(output) = 1
        for v in reversed(topo):
            v._backward()

Two lines carry the whole algorithm:

  1. self.grad = 1 — the seed. The derivative of the output with respect to itself is 1. This is what kicks the chain rule off.
  2. for v in reversed(topo): v._backward() — visit nodes from output back to inputs, and at each node run its captured closure to push gradient one hop further.

Why += everywhere (the most important subtlety)

Every _backward accumulates with +=, never =. This is the multivariable chain rule: if a Value is used in more than one place (e.g. c += c + 1 reuses c), its total gradient is the sum of the contributions from each downstream path. Using = would silently overwrite one path's gradient with another's — the single most common bug when people reimplement autograd. (It also means you must zero gradients between training steps, which is what zero_grad() below is for.)

This is exercised by the README's stress example, where values are reused and combined in tangled ways:

from micrograd.engine import Value
a = Value(-4.0); b = Value(2.0)
c = a + b
d = a * b + b**3
c += c + 1
c += 1 + c + (-a)
d += d * 2 + (b + a).relu()
d += 3 * d + (b - a).relu()
e = c - d
f = e**2
g = f / 2.0
g += 10.0 / f
print(f'{g.data:.4f}')   # 24.7041   (forward)
g.backward()
print(f'{a.grad:.4f}')   # 138.8338  (dg/da)
print(f'{b.grad:.4f}')   # 645.5773  (dg/db)

The repo's only test (test/test_engine.py) recomputes exactly this graph in PyTorch and asserts the data and grad match to numerical tolerance — i.e. micrograd is validated against the real thing. Run it with python -m pytest (PyTorch required).

The neural net — an MLP made of pure Values (nn.py)

Now the same Value machinery is wrapped into a familiar network. The base Module just defines gradient zeroing and a parameter list:

class Module:
    def zero_grad(self):
        for p in self.parameters():
            p.grad = 0
    def parameters(self):
        return []

A Neuron is a list of weight Values, a bias, and a forward pass that is literally a dot product plus bias, optionally passed through ReLU:

class Neuron(Module):
    def __init__(self, nin, nonlin=True):
        self.w = [Value(random.uniform(-1,1)) for _ in range(nin)]
        self.b = Value(0)
        self.nonlin = nonlin

    def __call__(self, x):
        act = sum((wi*xi for wi,xi in zip(self.w, x)), self.b)  # Σ wᵢxᵢ + b, all Value ops
        return act.relu() if self.nonlin else act

    def parameters(self):
        return self.w + [self.b]

That sum((wi*xi ...), self.b) line is doing something profound and easy to miss: every wi*xi is a Value.__mul__ that adds a node to the graph, and the running sum chains them with Value.__add__. So a single neuron's forward pass silently constructs a little subgraph that backward() already knows how to differentiate. No gradient code is written here at all — it is inherited entirely from engine.py.

Layer is just a list of neurons run on the same input; MLP chains layers, making every layer ReLU except the last (the output stays linear so it can represent any real number):

class Layer(Module):
    def __init__(self, nin, nout, **kwargs):
        self.neurons = [Neuron(nin, **kwargs) for _ in range(nout)]
    def __call__(self, x):
        out = [n(x) for n in self.neurons]
        return out[0] if len(out) == 1 else out
    def parameters(self):
        return [p for n in self.neurons for p in n.parameters()]

class MLP(Module):
    def __init__(self, nin, nouts):
        sz = [nin] + nouts
        self.layers = [Layer(sz[i], sz[i+1], nonlin=i!=len(nouts)-1)
                       for i in range(len(nouts))]
    def __call__(self, x):
        for layer in self.layers:
            x = layer(x)
        return x
    def parameters(self):
        return [p for layer in self.layers for p in layer.parameters()]

The full training loop

demo.ipynb trains a binary classifier on the two-moons dataset with an MLP of two 16-node hidden layers (MLP(2, [16, 16, 1])), an SVM max-margin loss, and plain SGD. The loop is the universal four-step pattern that every deep-learning trainer reduces to:

model = MLP(2, [16, 16, 1])

for k in range(100):
    # 1. forward: build the graph, compute the loss
    scores = [model(x) for x in inputs]
    loss   = svm_max_margin_loss(scores, labels)   # a single Value at the graph's root

    # 2. backward: zero old grads, then fill in fresh ones
    model.zero_grad()
    loss.backward()

    # 3. update: nudge every parameter downhill (SGD)
    lr = 1.0 - 0.9 * k / 100
    for p in model.parameters():
        p.data -= lr * p.grad

zero_grad() first (because grads accumulate with +=), loss.backward() to populate every p.grad, then p.data -= lr * p.grad to step. That is gradient descent, in full, with zero framework support. The decision boundary it learns on the moons is the payoff visual in the notebook.

Seeing the graph (trace_graph.ipynb)

A draw_dot() helper renders any Value's graph with Graphviz — each node showing its forward data (left) and grad (right), each op as an interior node. Visualizing nn.Neuron(2)(x) makes the abstract DAG concrete and is the single best way to believe that a neuron is just a handful of +/* nodes.

Key ideas & tradeoffs

  • Closures as the backward pass. Instead of a giant if op == '+': ... elif op == '*': ... dispatch, each op captures its own gradient rule in a _backward closure at construction time. This is the elegant heart of the design and what keeps it ~100 lines.
  • Define-by-run (dynamic graph). The graph is built as the forward math runs, so Python control flow (loops, branches) "just works." This is exactly PyTorch's eager model — micrograd is a faithful miniature of it, not a toy with a different paradigm.
  • Reverse-mode is the right default for ML. One backward pass computes gradients of a single scalar loss w.r.t. all parameters at once. (Forward-mode would need one pass per parameter — fine for one input/many outputs, catastrophic for many params/one loss.) micrograd makes this asymmetry tangible.
  • Local derivatives compose into global ones. You only ever specify the derivative of +, *, **, relu. The chain rule, applied along the topo order, assembles those locals into the gradient of an arbitrarily deep network. This is the whole conceptual unlock.
  • Tradeoff — clarity bought with extreme inefficiency. One Value and one Python closure per scalar means millions of tiny objects and pointer chases for any real model. It is, deliberately, the slowest possible correct autograd. Tensors exist to amortize this; see [[flash-attention]] for the opposite extreme of squeezing the hardware.

Honest caveats

  • It will not scale, by design. Per-scalar objects + Python closures make micrograd thousands of times slower and more memory-hungry than a tensor framework. Never reach for it to train anything real — its job is understanding, not throughput.
  • Minimal op set. Only +, *, **(int/float), and relu are primitives (plus the derived -, /, neg). No tanh, exp, log, softmax, matmul, conv, etc. The Zero-to-Hero lecture adds tanh/exp to teach more derivatives, but the published repo is intentionally leaner. __pow__ only supports constant (non-Value) exponents.
  • No batching, no broadcasting, no shapes. Every concept that tensor frameworks add (axes, strides, broadcasting, in-place ops, mixed precision) is absent. Going from micrograd to PyTorch is a real conceptual jump in engineering, even though the autograd idea is unchanged.
  • `backward()` is recursive. build_topo recurses over the graph; a very deep graph could hit Python's recursion limit. Irrelevant for teaching, would matter at scale.
  • You must `zero_grad()` yourself. Because gradients accumulate (+=), forgetting to zero between steps silently corrupts training — a real footgun that the design makes you feel, which is part of the lesson.
  • Validated only against one PyTorch graph. The single test is strong evidence the core is correct, but it is not a broad test suite — treat the repo as a teaching artifact, not production-hardened code.
  • Line-count is approximate. "~150 lines," "94 lines," "~100 + ~50" all float around (Karpathy's own tweet says 94); exact counts depend on how you count blanks. The point is the order of magnitude, not a precise number.

OpenAlice + Academy ladder

micrograd is the autograd rung of the lab's education path — the place where "a network learns" stops being a slogan and becomes a mechanism you can read end to end. Recommended order:

  1. [[math-for-ml-foundations]] — the calculus prerequisite: derivatives, the chain rule, gradients. micrograd is that chain rule turned into running code.
  2. [[neural-network-from-scratch]] — what a neural network is and why it learns, with the gradient still mostly conceptual.
  3. micrograd (this article) — how the gradient is actually computed: reverse-mode autodiff, made fully explicit. The keystone.
  4. [[nn-zero-to-hero]] — Karpathy's lecture series; lecture 1 is building micrograd live, then it climbs from here to language models.
  5. [[microgpt-build-an-llm-from-scratch]] → [[nanogpt]] / [[nanochat]] — the same backward() idea, now over tensors and attention ([[attention-and-transformers]], [[positional-encoding]]), scaled to a real LLM.

Why ATLAS keeps this in the library: it is the highest-leverage 150 lines in ML education. Any agent or contributor who wants to truly reason about training — gradient flow, vanishing/exploding gradients, why zero_grad exists, what an optimizer step is — should be able to reconstruct the Value class from memory. Everything heavier in this library inherits its semantics; understanding it here, cheaply, pays off everywhere above it. Karpathy's line is the mandate: everything else is just efficiency.

Sources