Math for ML: Linear Algebra, Calculus & Probability

The pitch. An LLM is, underneath, three pieces of high-school-plus math wearing a trench coat: linear algebra (move and mix numbers in bulk), calculus (figure out which direction makes the model less wrong), and probability (turn scores into "how sure am I"). If you genuinely understand y = Wx, the chain rule, and "softmax turns numbers into probabilities," you understand the load-bearing 80% of what happens inside GPT. This article takes you from "what's a vector" to "here's why the gradient of softmax+cross-entropy is just p − y" — and ties every piece to where it actually shows up in a transformer.

What it is (intuition first)

Three branches of math, each answering one question a neural network constantly asks.

1. Linear algebra — "how do I transform a whole bunch of numbers at once?" A neural network is mostly multiplying lists of numbers by grids of numbers. A vector is just an ordered list ([0.3, -1.2, 5.0]) — think of it as a point or an arrow in space. A matrix is a grid of numbers that transforms vectors: feed it a vector, it spits out a new vector (rotated, stretched, projected). The single most-run operation on the planet right now is y = W·x — "transform input x by weight matrix W." That's a matrix-vector multiply, and a transformer does billions of them per token.

2. Calculus — "if I nudge a knob, does the model get better or worse, and by how much?" A model has millions/billions of knobs (weights). Training = find knob settings that make predictions good. Calculus gives us the derivative: the answer to "if I increase this one knob a tiny bit, how much does my error change?" The derivative is slope = sensitivity. The gradient is just "all the derivatives at once, one per knob" — a giant arrow pointing in the direction that increases error fastest. So we step the opposite way. That's the entire training loop.

3. Probability — "I have raw scores; how confident should I be?" The model outputs raw numbers ("logits") like [2.1, 0.5, -1.0] over possible next tokens. Those aren't probabilities — they don't sum to 1, some are negative. Softmax squashes them into a proper probability distribution [0.7, 0.2, 0.1]. Expectation ("weighted average") and cross-entropy ("how surprised was I by the truth") then let us measure and minimize how wrong those probabilities are.

That's it. Linear algebra moves the numbers, calculus tells us how to fix the numbers, probability tells us how to score the numbers. Everything below is the rigorous version of these three sentences.

Why it matters

It's not optional decoration — it *is* the model. A transformer layer is: a matmul (linear algebra) → a softmax over attention scores (probability) → another matmul → a nonlinearity. Training it is one long application of the chain rule (calculus). Remove any of the three and there is no LLM.
Debugging requires it. "Loss is NaN" almost always traces to a numerical-stability issue in softmax/log (probability + floating point). "Loss won't go down" is a gradient/learning-rate problem (calculus). "My shapes don't match" is linear algebra. You cannot diagnose these from vibes.
Every modern technique is a re-mix of these primitives. LoRA ([[lora-and-peft]]) is low-rank matrix factorization. Quantization ([[quantization]]) is about preserving matmul fidelity at fewer bits. Attention ([[attention-and-transformers]]) is a softmax-weighted expectation over value vectors. Flash-attention ([[flash-attention]]) is the same math reordered to be numerically stable and memory-cheap. Knowing the foundation means new papers read as variations, not magic.

How it works (the real mechanics)

Part 1 — Linear algebra: vectors, dot products, matmul

Objects and notation (following Goodfellow et al., Deep Learning ch. 2):

Scalar: a single number, s ∈ ℝ (e.g. a learning rate).
Vector: an ordered array, bold lowercase x, with elements x₁…xₙ. If all entries are real, x ∈ ℝⁿ. Geometrically: a point/arrow in n-dimensional space.
Matrix: a 2-D grid, bold uppercase A ∈ ℝ^{m×n} (m rows, n columns). Entry A_{ij} = row i, column j.
Tensor: same idea with more than two axes (e.g. a batch of token embeddings is a 3-D tensor [batch, sequence, dim]). "Tensor" in deep learning just means "n-dimensional array."

The dot product — the atom of everything. For two vectors of the same length:

a · b = Σᵢ aᵢ·bᵢ = a₁b₁ + a₂b₂ + … + aₙbₙ   →  a single scalar

Intuition: the dot product measures alignment. Big positive = vectors point the same way; zero = perpendicular ("unrelated"); negative = opposing. This is exactly why attention uses query · key — it's literally asking "how aligned is this query with that key?" (see [[attention-and-transformers]], [[embeddings]]). Cosine similarity is just a dot product of length-normalized vectors.

Matrix multiplication (matmul) — dot products in bulk. C = A·B where A is m×n and B is n×p gives an m×p matrix, and:

C_{ij} = Σ_k A_{ik}·B_{kj}     # row i of A dotted with column j of B

The inner dimensions must match (the n). This "shapes must line up" rule is the #1 source of beginner errors. The mental model: each output entry is one dot product; a matmul is a whole table of dot products computed at once. GPUs exist primarily to do this fast.

The workhorse: a linear layer. y = W·x + b. Here x is the input vector, W the learned weight matrix, b the bias vector. This single line, stacked with nonlinearities, is most of a neural network ([[neural-network-from-scratch]]).

Broadcasting. Adding a bias vector b to every row of a matrix C = A + b is written compactly without literally copying b — the vector is implicitly added to each row (Goodfellow ch. 2). Numpy/PyTorch/JAX all do this; it's a notation/efficiency convenience, not new math.

Other pieces you'll meet: transpose (Aᵀ)_{ij} = A_{ji} (flip across the diagonal — turns rows into columns, needed constantly to make shapes line up); norms (the "length" of a vector, e.g. L2 norm ‖x‖₂ = √(Σ xᵢ²), used for normalization and weight decay); identity/inverse matrices (the "1" and "1/x" of matrix-land, central to classical linear systems but used less directly inside trained nets).

Part 2 — Calculus: derivatives, the chain rule, backprop

Derivative = slope = sensitivity. For a function f(x), the derivative df/dx answers: "if I nudge x by a tiny ε, how much does f change?" Answer: f(x+ε) ≈ f(x) + ε·(df/dx). A few you must know cold:

d/dx (xⁿ) = n·x^(n-1)
d/dx (eˣ) = eˣ
d/dx (ln x) = 1/x
d/dx σ(x) = σ(x)·(1 − σ(x))     # sigmoid σ(x)=1/(1+e^{-x}); derivative in terms of itself

The chain rule — the single most important equation in deep learning. If z = g(x) and f = f(z) (f composed with g), then:

df/dx = (df/dz)·(dz/dx)

In words: multiply the local slopes along the path. A neural network is just a deeply nested composition f(g(h(…(x)))), so its derivative is a long product of local slopes. That product, computed efficiently, is backpropagation.

Gradient = vector of partial derivatives. When f depends on many inputs, the gradient ∇f = [∂f/∂x₁, ∂f/∂x₂, …] collects the slope w.r.t. each one. It points in the direction of steepest increase of f.

Backpropagation = chain rule over a computational graph (Karpathy, CS231n optimization-2). Represent the computation as a graph of simple operations ("gates"). Each gate, during the forward pass, computes its output; during the backward pass, it receives the gradient of the loss w.r.t. its output, multiplies by its local gradient, and passes the result to its inputs. The beautiful part: each gate only needs to know its own local derivative. The three canonical gates:

Add gate f = x + y: local grads are both 1 → distributes the incoming gradient unchanged to both inputs.
Multiply gate f = x·y: ∂f/∂x = y, ∂f/∂y = x → swaps the inputs and scales the incoming gradient.
Max gate f = max(x,y): gradient is 1 for the larger input, 0 for the other → routes the gradient only to the winner (this is exactly how gradients flow through ReLU).

Pseudocode-in-prose for one training step:

Forward: push input through the graph, caching intermediate values, until you get a scalar loss.
Seed: set d(loss)/d(loss) = 1 at the output.
Backward: visit gates in reverse topological order; each multiplies the incoming "upstream" gradient by its cached local gradient and sends it to its inputs.
Accumulate at forks: if a variable feeds multiple downstream gates, its gradients add up (+=, never =). This is the multivariable chain rule and a notorious bug source — overwriting instead of accumulating silently corrupts training (CS231n).

The reason backprop is efficient (one backward pass costs ~the same as one forward pass, regardless of how many millions of parameters) is that it reuses shared subexpressions instead of recomputing each parameter's derivative from scratch. This efficiency is what makes training billion-parameter LLMs feasible at all.

Part 3 — Gradient descent: using the gradient to learn

You have ∇L (gradient of loss w.r.t. every weight). The gradient points uphill (toward more loss). So step downhill — the opposite direction (Wikipedia, Gradient descent):

θ_{t+1} = θ_t − η · ∇L(θ_t)

θ = all the model's parameters (weights).
η (eta) = the learning rate / step size. Too small → painfully slow convergence; too large → you overshoot the minimum and the loss diverges (often → NaN). Choosing η is the central practical knob.
Why negative gradient? Because f decreases fastest in the direction of −∇f — that's literally the definition of the gradient (steepest-ascent direction), so its negation is steepest descent.

Under nice (convex) conditions this produces a non-increasing sequence L(θ₀) ≥ L(θ₁) ≥ … converging to a minimum. Real neural nets are non-convex — no such guarantee — yet it works astonishingly well in practice (an honest open question; see caveats).

Stochastic gradient descent (SGD). Computing ∇L over the entire dataset every step is too expensive. Instead, estimate the gradient from a small random mini-batch. It's noisier but vastly cheaper, and the noise can even help escape bad spots. SGD (and its momentum/adaptive descendants like Adam) is "the most basic algorithm used for training most deep networks today." Adam adds per-parameter adaptive step sizes and momentum, but the core is still θ ← θ − η·(scaled gradient).

Part 4 — Probability: softmax, cross-entropy, expectation

Softmax — raw scores → probability distribution (CS231n, linear-classify). Given logits z = [z₁…z_k]:

softmax(z)_j = e^{z_j} / Σ_k e^{z_k}

Properties: every output is in (0,1) and they sum to 1 — a valid probability distribution. Exponentiating makes everything positive and amplifies differences (the biggest logit dominates). In an LLM, the final layer applies softmax over the entire vocabulary to produce P(next token | context). Attention also uses softmax — over alignment scores — to decide how much each previous token contributes ([[attention-and-transformers]]).

Numerical-stability trick (you WILL hit this). e^{z} overflows for large z. Since softmax is invariant to adding a constant to all logits, subtract the max first:

softmax(z)_j = e^{z_j − m} / Σ_k e^{z_k − m},   where m = max_k z_k

Now the largest exponent is e⁰ = 1 — no overflow. This identity (e^{f + logC}/Σ e^{f + logC} with logC = −max f) is mandatory in every real softmax implementation. The same stability concern, applied to the whole attention computation, is a core motivation for [[flash-attention]].

Cross-entropy loss — "how surprised was I by the truth?" For a single example with true class y_i, where p = softmax(scores):

L_i = −log( e^{f_{y_i}} / Σ_j e^{f_j} ) = −f_{y_i} + log Σ_j e^{f_j}

Equivalently L_i = −log(p_{y_i}) — the negative log-probability the model assigned to the correct answer. Perfect confidence in the truth (p=1) → loss 0; assigning near-zero probability to the truth → loss → ∞. This is the loss LLMs minimize during pretraining: at every position, "what's the negative log-prob you assigned to the token that actually came next?"

The information-theoretic view: cross-entropy H(p,q) = −Σ_x p(x)·log q(x) measures the average surprise of the true distribution p under your predicted distribution q. Minimizing it is equivalent to minimizing the KL divergence between predicted and true distributions (CS231n). It connects directly to [[scaling-laws]] (cross-entropy loss is the quantity that scales smoothly with model/data size) and to RLHF reward modeling ([[rlhf-and-alignment]]).

The punchline gradient — why this exact pairing is everywhere. Combine softmax and cross-entropy and differentiate the loss w.r.t. the logits, and the messy softmax Jacobian and the log's derivative cancel almost everything, collapsing to:

∂L / ∂z_j = p_j − y_j          # (predicted prob) minus (one-hot truth)

That is: gradient = "what you predicted" minus "what was true." Clean, cheap, and intuitive — if you over-predicted a class, push its logit down by the over-prediction; the correct class gets p − 1 (push it up). This elegant cancellation (derived in Bendersky's softmax write-up and standard everywhere) is the reason softmax+cross-entropy is the default classification head — it makes the backward pass trivial and well-behaved.

Expectation & probability glue (Wikipedia, Expected value). A probability distribution assigns weights summing to 1 over outcomes. Expectation is the probability-weighted average:

E[X] = Σᵢ xᵢ·pᵢ      (discrete)        E[X] = ∫ x·f(x) dx   (continuous)

Linearity of expectation — E[X+Y] = E[X]+E[Y], E[aX]=a·E[X], always, even when X and Y are dependent — is the most-used property and underpins why averaging mini-batch losses gives an unbiased estimate of the full-dataset gradient. Variance Var(X)=E[(X−E[X])²] measures spread (gradient-noise analysis, weight-init scaling). Attention's output is literally an expectation: a softmax-weighted average (expectation) of value vectors. Sampling from an LLM = drawing from the softmax distribution; temperature rescales logits (z/T) before softmax to make that distribution flatter (more random) or sharper (more deterministic).

Key ideas & tradeoffs

Vectorize everything. The same computation written as explicit loops vs. as matmuls differs by 10–100× in speed on a GPU. "Think in matrices, not loops" is the core practical skill linear algebra buys you.
Forward caches what backward needs. Backprop reuses values from the forward pass (that's why training holds activations in memory — the memory cost of training vs. inference). This is the tradeoff [[flash-attention]] and gradient-checkpointing renegotiate.
Learning rate is the master knob. Most "training won't converge" pain is η too high (diverges) or too low (stuck). Schedules (warmup + decay) exist because the right η changes during training.
Softmax+cross-entropy is a package deal. Their gradient p − y is so clean that frameworks fuse them into one numerically-stable op (log_softmax + nll_loss, or cross_entropy). Computing softmax then a separate log invites overflow/log(0) — always use the fused, max-subtracted version.
Probabilities are scores, not truths. A softmax output of 0.9 means "the model's score-pattern, normalized, lands at 0.9" — it is not a calibrated real-world probability unless explicitly calibrated. Conflating the two is a common honesty failure in deployed systems.
Float precision is part of the math. fp16/bf16 ([[quantization]]) change which numerically-stable tricks you must use. The math on paper assumes infinite precision; the math in silicon does not.

Honest caveats & open questions

Why does non-convex SGD work so well? Gradient descent only guarantees convergence to a global minimum for convex problems. Deep nets are wildly non-convex, yet SGD reliably finds good solutions. Why (loss-landscape geometry, implicit regularization, overparameterization effects) is still an active research question, not settled theory.
The gradient is local. It tells you the steepest direction right here; it knows nothing about the global landscape. This is why initialization, learning-rate schedules, and momentum matter so much — they're heuristics patching a fundamentally local signal.
This article is the *foundation*, not the *whole stack*. Real training adds normalization (LayerNorm/RMSNorm), residual connections, optimizers with second-moment estimates (Adam), and mixed-precision — each with its own math. The four parts here are necessary but not sufficient; they're the literacy that lets you read the rest.
Softmax calibration & the "p−y" elegance can mislead. The clean gradient is for categorical cross-entropy with one-hot labels. Label smoothing, soft targets (distillation), and RL objectives ([[rlhf-and-alignment]]) change the gradient — don't assume p − y holds universally.
Notation is not standardized. Row-vector vs. column-vector conventions, Wx vs xW, and whether the batch axis comes first all vary by source/framework. Shapes that "should" line up sometimes don't because two references disagree on convention. Always check the actual tensor shapes.

How it connects to OpenAlice

This is foundational literacy for anyone building or extending the [[microgpt-build-an-llm-from-scratch]] / [[llm-from-scratch]] / [[neural-network-from-scratch]] tracks in the OpenAlice Academy — those builds are these equations turned into running code, so this article is the prerequisite ramp.

Concretely within the ecosystem:

Attention internals. Alice's transformer-based reasoning rests on query·key dot products → scaled → softmax → weighted-sum (expectation) of values. Understanding [[attention-and-transformers]] and [[flash-attention]] requires the dot-product, softmax, and stability math above.
Embeddings & retrieval. Semantic search across the lab (Atlas's pgvector index, [[embeddings]], [[graphrag]]) ranks by cosine similarity — a length-normalized dot product. The "alignment" intuition from Part 1 is exactly what makes vector search work.
Routing & councils. [[model-routing]] and [[mixture-of-agents]] / [[fusion-and-llm-councils]] combine model outputs; softmax/expectation are the natural tools for weighting and aggregating those probability distributions.
Efficiency work. [[lora-and-peft]] (low-rank matrix updates) and [[quantization]] (low-bit matmul) are direct applications of linear-algebra structure — they only make sense once you see a layer as y = Wx.

If you can derive ∂L/∂z = p − y by hand and explain why a matmul is "a table of dot products," you are ready to read essentially every other article in this library.

Math for ML: Linear Algebra, Calculus & Probability (noob → deep)