Atlas
kb://library/neural-network-from-scratchstable2026-06-16

Build a basic neural network from scratch

libraryeducationneural-networkfrom-scratchbackpropagationgradient-descentperceptronautogradfundamentalsacademy

Build a basic neural network from scratch

The foundational "small" entry point of the lab's education path. This is the rung below the LLM. Master this — a network that learns to draw a line between dots — and the rest of the ladder ([[microgpt-build-an-llm-from-scratch]] → [[llm-from-scratch]] → transformers/attention) is just the same idea, scaled up. A complete beginner who reads this to the end should understand backpropagation.

>

Grounded in a from-scratch video NAO supplied ("Делаю нейросеть с нуля", GNcGPw_Kb_0), cross-checked against 3Blue1Brown, Michael Nielsen's book, and Karpathy's micrograd — the scalar-autograd lineage that connects directly up to microGPT.

The one-sentence idea

A neural network is a function that learns from examples. You don't write the rules; you show it pairs of (input, correct answer), and it adjusts its own internal numbers until it gets the answers right. That adjusting is "learning," and the algorithm that does it is backpropagation + gradient descent. Everything below is just unpacking that sentence honestly.

The from-scratch video says it best: it would take "just a couple of Python libraries and a few lines of code" to use a neural net — but the author wants to "understand all of this from the very beginning," so he writes one completely from scratch in ~86 lines. That's the right instinct, and it's the lab's instinct too (see How it connects to OpenAlice).

1. What a neural network is (intuition first)

Forget biology and brains for a moment. Picture a task: you have dots scattered on a 2D plane — some red, some blue — and you want a rule that, given a new dot's coordinates (x, y), says "red" or "blue."

A neural network is a machine that finds that rule by itself. You feed it the dots you already know the colour of. At first it guesses randomly. Each time it's wrong, it nudges its internal knobs a tiny bit toward "less wrong." Do that thousands of times and the knobs settle into values that draw a sensible boundary line between red and blue. In the video you can literally watch this: the author clicks to place dots, and a line wobbles, rotates, and blurs as the network searches for where to split them.

So: a network is a function with adjustable knobs (called *weights* and *biases*), plus a procedure for tuning those knobs from examples. Nothing mystical. The same machine that separates dots is — at enormous scale — what predicts the next word in [[microgpt-build-an-llm-from-scratch]].

Why "learns from examples" matters: classic programming is you writing the if-rules. A neural net is the opposite — you provide examples and let the rules emerge. That's why the video can solve the Iris flower problem without knowing what the four input numbers even mean: "we don't need to know what they mean — we just need the network to learn to split the data into three classes."

2. The neuron / perceptron (weights, bias, activation)

The atom of a network is a single neuron (the oldest version is the perceptron, invented ~60 years ago — exactly what the video builds).

A neuron does three things:

  1. Weighted sum. It takes inputs x₁, x₂, … and multiplies each by its own number — a weight w₁, w₂, … — then adds them up.
  2. Add a bias. It adds one more number, the bias b.
  3. Activation. It passes the result through an activation function σ.

In one line:

output = σ(w₁·x₁ + w₂·x₂ + … + wₙ·xₙ + b)

The video walks this by hand: send 0.8 in along a connection with weight 1 → stays 0.8; the same 0.8 down another connection with weight 0.5 → becomes 0.4; send 0.6 down a weight of -2 → becomes -1.2; then sum everything arriving at the next neuron. That weighted-sum-then-sum is the entire forward mechanic. The inputs (the dot's x, y) are values you supply; the weights (drawn blue in the video) are "unknown to us, and we need to find them. Training the network is finding the values of these weights."

Why the bias is not optional (the video's best moment)

The video discovers the bias the hard way, and it's the clearest beginner lesson in the whole thing. With only weights, the network's dividing line always passes through the origin (0,0). Why? Because the inputs at the origin are x=0, y=0, and "no matter how we add or multiply zeros, they stay zero." So the line can rotate, but it's pinned to the centre. The moment you add a bias neuron — a neuron whose value is always 1, with its own trainable weights — the line can shift off the origin, and the network "immediately started recognising situations where the dots aren't centred."

Mental model: weights rotate/tilt the boundary; the bias shifts it. You need both, exactly like y = m·x + b needs both a slope and an intercept.

The activation function

If a neuron only did the weighted sum, a whole network would collapse into one big linear function — it could only ever draw straight lines. The activation σ adds a gentle bend, which is what eventually lets stacked neurons draw curved boundaries.

  • The video uses a smooth S-curve (a sigmoid-style function).
  • Nielsen's book formalises the sigmoid neuron: same weights and bias as a perceptron, but output σ(w·x + b) where `σ(z) = 1 / (1 + e^(−z))`. Sigmoid squashes any number into the range (0, 1).
  • Karpathy's micrograd uses the even simpler ReLU: relu(z) = max(0, z).

The crucial property (Nielsen): a sigmoid neuron is built so that small changes in the weights cause only small changes in the output. That smoothness is what makes learning possible — it lets us ask "if I nudge this weight a hair, does the answer get a little better or a little worse?" A hard 0/1 perceptron can't answer that; a smooth neuron can.

3. Layers + the forward pass

One neuron draws one line. To do anything interesting you stack neurons into layers:

  • Input layer — your raw numbers (the dot's x, y; or 784 pixels for a digit).
  • Hidden layer(s) — the middle. The video starts with no hidden layer (just input→output) and hits a wall (next section), then adds a hidden layer and the network "got smarter."
  • Output layer — the answer (one neuron for "red vs blue"; ten neurons for the digits 0–9, one per class).

The forward pass is just: feed the input numbers in on the left, apply the neuron rule layer by layer, read the answer off the right. 3Blue1Brown's mental image: "a neuron is a thing that holds a number," and the network is those numbers lighting up, layer by layer, until the output layer's brightest neuron is the prediction.

Matrix-free → matrix view

For one neuron with inputs x₁, x₂:

z = w₁·x₁ + w₂·x₂ + b
a = σ(z)

A layer is just many neurons doing this at once, each with its own weight row. Stack those rows and the whole layer becomes one matrix multiply plus a vector add:

z = W·x + b          # W is the weight matrix, b the bias vector
a = σ(z)             # σ applied element-wise

This matrix form is only a repackaging — the arithmetic is identical to doing each neuron by hand. But it's why GPUs matter: a matrix multiply is the one operation hardware does blisteringly fast, so the entire field is built on it. (microGPT's attention is, under the hood, the same W·x + b trick repeated.)

4. The loss — what "wrong" means as a number

To improve, the network needs to measure how wrong it is, as a single number. That number is the loss (or cost).

The simplest honest version is quadratic cost (mean-squared error). For one example, if the network outputs activations a and the correct answer is y:

C = ½ · Σⱼ (aⱼ − yⱼ)²

In words: for each output neuron, take (what it said − what it should have said), square it (so over- and under-shooting both count as positive error, and big mistakes hurt more than small ones), sum across all output neurons. Loss = 0 means perfect; bigger loss = more wrong. The video tracks exactly this: a red "error" curve that should go down as training proceeds, alongside a blue "accuracy" curve that goes up.

The whole goal of training is now crisp: find the weights and biases that make the loss as small as possible.

Real classifiers usually swap quadratic cost for cross-entropy loss (microGPT uses it), which trains faster on probability outputs — but the idea is identical: one number measuring wrongness, which we want to minimise.

5. Backpropagation + gradient descent — the heart

This is the engine. The video is refreshingly honest about it: after watching "a huge amount of material" and even implementing it, the author "still can't say I understood it 100%." That's normal. Let's make it land.

Gradient descent — the intuition

Imagine the loss as a hilly landscape, where your position is set by all the weights and biases, and your altitude is the loss. You want the lowest valley. You can't see the whole map, but you can feel the slope under your feet. So you take a small step downhill, then feel again, then step again. Repeat. That's gradient descent.

The "slope under your feet" for a single weight is the gradient ∂C/∂w — it answers: "if I increase this weight a tiny bit, how much does the loss change?" The update rule is just "step opposite the slope":

w ← w − η · (∂C/∂w)          # for every weight
b ← b − η · (∂C/∂b)          # for every bias

Here `η` (eta) is the learning rate — your step size. Too small and learning crawls; too big and you overshoot the valley and bounce around. (The video exposes learning rate as a tunable knob for exactly this reason.) The minus sign is the whole point: the gradient points uphill (toward more loss), so we move the other way.

Backpropagation — computing all those slopes efficiently

Gradient descent needs ∂C/∂w for every weight. A big network has thousands. Computing each one separately would be hopeless. Backpropagation is the trick that gets all of them in a single backward pass, using the chain rule from calculus.

The chain rule, in one sentence: if A affects B and B affects C, then A's effect on C is `(A's effect on B) × (B's effect on C)`. You multiply the local effects along the path. Backprop applies this from the loss backward toward the inputs: start at the output where you know the error, then walk backward, at each step multiplying by the local slope, handing each neuron its share of the blame. Forward = compute the answer; backward = compute who's responsible for the error and by how much.

The clearest way to see it: micrograd's scalar autograd

Karpathy's micrograd is the gold-standard beginner demystifier — ~100 lines that implement backprop over individual scalar numbers. Every number is wrapped in a Value that remembers two things: its data, and a tiny function _backward saying how to push gradient to its parents. Each operation knows its own local derivative. Verbatim from engine.py:

def __mul__(self, other):                 # multiplication: out = self * other
    out = Value(self.data * other.data, (self, other), '*')
    def _backward():
        self.grad  += other.data * out.grad   # ∂out/∂self  = other  (chain rule ×)
        other.grad += self.data  * out.grad   # ∂out/∂other = self
    out._backward = _backward
    return out

def __add__(self, other):                 # addition: out = self + other
    out = Value(self.data + other.data, (self, other), '+')
    def _backward():
        self.grad  += out.grad                # ∂out/∂self  = 1  → gradient passes straight through
        other.grad += out.grad                # ∂out/∂other = 1
    out._backward = _backward
    return out

Read those two _backward closures and you've read the chain rule made literal: `+` copies the incoming gradient to both inputs; `×` routes each input's gradient scaled by the *other* input. The top-level backward() just topologically sorts the graph, sets the output's gradient to 1.0, and calls every _backward in reverse:

def backward(self):
    topo = []                       # topological order of all nodes
    # ... depth-first build of topo ...
    self.grad = 1                   # seed: ∂C/∂C = 1
    for node in reversed(topo):     # walk backward through the graph
        node._backward()            # each node hands gradient to its parents
One subtlety worth catching: the gradients use +=, not =. If a value feeds into two places, its gradient is the sum of both paths — that's the multivariate chain rule, and forgetting it is the #1 backprop bug.

A tiny worked example (do this by hand)

Take the simplest learnable thing: one input x = 2, one weight w = 3, target y = 10, quadratic loss.

Forward:
  p  = w · x            = 3 · 2 = 6        (prediction)
  e  = p − y            = 6 − 10 = −4      (error)
  L  = e²               = 16              (loss)

Backward (chain rule, multiply local slopes from L back to w):
  ∂L/∂e = 2·e           = 2·(−4) = −8
  ∂e/∂p = 1                                (since e = p − y)
  ∂p/∂w = x             = 2                 (since p = w·x)
  ∂L/∂w = (∂L/∂e)·(∂e/∂p)·(∂p/∂w) = −8 · 1 · 2 = −16

Update (η = 0.01):
  w ← w − η·(∂L/∂w) = 3 − 0.01·(−16) = 3.16

The gradient is negative, so the update raises w from 3 toward the value (5) that would make p = w·x = 10 exactly right. Run that loop and w climbs 3 → 3.16 → … → 5, and the loss falls to 0. That is a neural network learning, in full, on one weight. Real networks do the identical thing across thousands of weights at once — that's all backprop is.

6. A minimal training loop (pseudocode)

Putting it together, the entire algorithm is this loop:

initialize all weights and biases to small random numbers   # the video: "fill with random numbers"

repeat many times (each pass = an "epoch"):
    for each batch of examples:
        # ---- forward pass ----
        prediction = network.forward(inputs)        # §3
        loss       = cost(prediction, correct_answer) # §4

        # ---- backward pass ----
        gradients  = backprop(loss)                  # §5: ∂loss/∂each weight & bias

        # ---- update (gradient descent step) ----
        for each parameter p:
            p = p − learning_rate * gradients[p]     # step downhill

    log: loss should trend DOWN, accuracy UP

That's it. Forward → measure → backward → nudge, over and over. Nielsen's stochastic gradient descent (SGD) refines just one detail: instead of using the whole dataset for each step, use a small mini-batch (e.g. the video feeds digits "in packs of 100"). It's a noisier but far faster estimate of the downhill direction, shuffled each epoch.

7. Honest caveats — toy vs. real

The video is wonderfully candid about where a from-scratch net struggles, and these limits are the reason the field looks the way it does:

  • Linear separability. With no hidden layer, the network can only draw a single straight line. Put red and blue dots in an XOR-like pattern that needs two lines and it "simply can't do it." The fix — add a hidden layer — is the entire reason deep networks exist. More neurons/layers = more bends in the boundary.
  • Training is finicky. The video shows the net "getting stuck," the boundary "flickering," solutions appearing then vanishing. Convergence depends on initialization, learning rate, and luck. This is real, not a bug in his code.
  • Overfitting. Trained on only 100 digit images, the net hit 100% accuracy on those — but only by memorizing them ("learning them by heart instead of learning general features"). On new digits it fails. The cure is more/augmented data and regularization. (The video even notes the standard trick: shift/scale/resize each digit to enlarge the dataset artificially.)
  • Vanishing gradients. Stack many sigmoid layers and the backward gradients get multiplied toward zero — early layers stop learning. This is why modern nets prefer ReLU, normalization, and residual connections (microGPT uses RMSNorm + ReLU for exactly this reason).
  • Why frameworks exist. Your hand-written autograd is correct but slow. PyTorch / JAX do the same math (W·x + b, backprop) but vectorized on GPU, with battle-tested optimizers (Adam) and numerical-stability fixes. You learn from scratch to understand; you build real things on frameworks. That's the whole arc of this library.

8. Where this leads — the ladder

You now hold the universal primitive. The rest of the lab's education path is the same machine, scaled and specialized:

  1. This article — a neuron, a layer, a loss, backprop. The atom.
  2. [[microgpt-build-an-llm-from-scratch]] — the next rung. A whole LLM in ~200 lines. Its autograd Value class is literally micrograd from §5; its training loop is §6 with cross-entropy + Adam. The only genuinely new idea is attention (a way for tokens to look at each other) stacked into a transformer. If you understood backprop here, microGPT is reachable.
  3. [[llm-from-scratch]] — the hands-on companion: a real ~10M-parameter GPT in PyTorch you train on a laptop in ~45 minutes. Same primitives, now on a framework doing the heavy lifting you hand-rolled here.
  4. Transformers / attention — the architecture that made LLMs work. Still just matrix multiplies + softmax + backprop. No magic was ever added; only scale and engineering (see microGPT's "everything else is just for efficiency").

Side paths that build on the same foundation: [[fusion-and-llm-councils]] and [[mixture-of-agents]] (combine many trained models), [[model-routing]] (pick the right one per query), and once a model is running, [[agent-memory-systems]] / [[mempalace]] (give it memory) and [[graphrag]] (give it grounded knowledge).

How it connects to OpenAlice

This article is the entry rung of the planned OpenAlice Academy — the lab's education path that climbs from "what is a neuron" all the way to the systems that run Alice. It exists for a specific reason: the same from-scratch ethos drives Alice's own coding capability. The lab's view is that an LLM is not a black box — its algorithm is "small and knowable" ([[microgpt-build-an-llm-from-scratch]]), and the trillion-dollar part is scale and engineering, not mystery. Understanding backprop on one weight is the first concrete proof of that philosophy.

The same loop that taught one weight to go 3 → 5 is, at scale, what lets Alice predict the next token, write code, and improve from feedback. Start here, climb the ladder, and the whole stack stops being magic and becomes a thing you can hold in your head.

Sources

  • ["Делаю нейросеть с нуля" / Making a neural network from scratch](https://www.youtube.com/watch?v=GNcGPw_Kb_0) (GNcGPw_Kb_0) — the primary, NAO-supplied video. ~86-line from-scratch perceptron; source of the dots/bias/linear-separability/overfitting intuition.
  • [3Blue1Brown — But what *is* a neural network?](https://www.3blue1brown.com/lessons/neural-networks/) — "a neuron is a thing that holds a number"; layers, hidden layers, the MNIST framing.
  • [Michael Nielsen — Neural Networks and Deep Learning, ch.1](http://neuralnetworksanddeeplearning.com/chap1.html) — perceptron vs. sigmoid neuron, σ(z)=1/(1+e⁻ᶻ), quadratic cost, gradient descent & SGD with mini-batches. The precise math.
  • [Karpathy — micrograd](https://github.com/karpathy/micrograd) — ~100-line scalar autograd; the Value class, _backward closures, and backward() quoted in §5. The bridge straight up to [[microgpt-build-an-llm-from-scratch]].