Convolutional Neural Networks (CNNs)

What it is (intuition first)

A convolutional neural network is a neural network built for data that has a grid layout — most famously images, which are grids of pixels. Its one big idea is simple: instead of looking at the whole image at once, slide a small pattern-detector across it, applying the same detector everywhere.

Picture a tiny 3×3 stamp — an "edge detector." You place it on the top-left corner of the photo, it produces one number (how strongly an edge appears there). You slide it one pixel right, get another number. You sweep it over the entire image. The result is a new grid (a "feature map") that lights up wherever that edge appears. That sliding-and-stamping operation is the convolution. A CNN learns hundreds of these stamps automatically, stacks them in layers, and the deeper layers combine simple stamps (edges) into richer ones (corners → textures → eyes → faces).

Two properties make this work, and they are the whole reason a CNN beats a plain fully-connected network ([[neural-network-from-scratch]]) on images:

Weight sharing (translation equivariance). The same stamp is reused at every position. A cat in the top-left is detected by the exact same weights as a cat in the bottom-right. You don't have to re-learn "cat" once per location.
Locality. Each output looks at only a small neighborhood, not all million pixels. Nearby pixels are related; far-apart ones usually aren't — convolution bakes that prior in.

The payoff: a fully-connected layer on a 224×224×3 image would need ~150,000 weights per neuron; a 3×3 convolution filter needs 9 weights (×channels), reused everywhere. Same expressive reach over the image, a tiny fraction of the parameters, and the right inductive bias.

Why it matters

CNNs are the foundation of modern computer vision. They turned image recognition from a hand-crafted-features craft into a learned, end-to-end discipline, and the architecture lineage below set off the deep-learning era:

LeNet-5 (1998) — LeCun et al. trained a CNN by backpropagation to read handwritten digits and cheques. It proved you could learn the convolution kernels directly from pixels and beat hand-engineered methods on handwriting.
AlexNet (2012) — Krizhevsky, Sutskever & Hinton scaled CNNs to ImageNet (1000 classes, 1.2M images) on GPUs and cut top-5 error dramatically. This is the moment deep learning "won" mainstream attention.
ResNet (2015) — He et al. made networks 152 layers deep trainable via residual connections, winning ILSVRC-2015 at 3.57% top-5 error — below human-level on that benchmark.

Even in today's transformer-dominated world, CNNs remain the default for many on-device, real-time, and small-data vision tasks (they're cheaper and need less data than Vision Transformers), and their conv stems still feed many [[multimodal-llms]]. The intuitions here — receptive fields, hierarchical features, residual paths — carry directly into the [[attention-and-transformers]] world.

How it works (real mechanics)

The convolution operation

For a 2D input (one channel) $I$ and a kernel $K$ of size $k \times k$, the (cross-correlation, which is what frameworks actually compute and call "convolution") output at position $(x,y)$ is:

$$ S(x,y) = b + \sum_{i=0}^{k-1}\sum_{j=0}^{k-1} I(x+i,\; y+j)\; K(i,j) $$

With multiple input channels $C_{in}$ (e.g. RGB = 3), each filter is a $k \times k \times C_{in}$ cube; you sum over channels too. A conv layer has $C_{out}$ such filters, producing a $C_{out}$-channel output. The learnable parameter count of one conv layer is:

$$ (k \cdot k \cdot C_{in} + 1)\; \cdot\; C_{out} \quad\text{(the +1 is the bias per filter)} $$

Note this is independent of the image's spatial size — that's weight sharing made concrete.

Output size: stride, padding, dilation

The single formula you'll use constantly (Dumoulin & Visin's "convolution arithmetic"), for one spatial dimension with input size $i$, kernel $k$, padding $p$, stride $s$:

$$ o = \left\lfloor \frac{i + 2p - k}{s} \right\rfloor + 1 $$

Stride $s$ = how many pixels the stamp jumps each step. $s=2$ roughly halves the output (downsampling).
Padding $p$ = zeros added around the border. $p = \lfloor k/2 \rfloor$ with $s=1$ keeps size unchanged ("same" padding); $p=0$ shrinks it ("valid").
Dilation $d$ spreads the kernel taps apart (effective kernel $k' = k + (k-1)(d-1)$), enlarging the receptive field without more parameters.

The same formula governs pooling layers.

Pooling

Pooling downsamples a feature map by summarizing each small window into one number — max pooling takes the maximum in each window, average pooling the mean. A 2×2 max pool with stride 2 quarters the spatial area. Purposes: shrink computation, build a little translation invariance (a feature shifting one pixel within the window doesn't change the output), and grow the receptive field. AlexNet used overlapping max pooling (window 3, stride 2). Many modern nets replace pooling with strided convolutions, and ResNet ends with global average pooling (collapse each whole channel to one number) instead of giant fully-connected layers.

Nonlinearity: ReLU

After each convolution, an elementwise nonlinearity. The workhorse is ReLU: $f(x) = \max(0, x)$. AlexNet's adoption of ReLU over saturating tanh/sigmoid was a key reason it trained fast enough to be practical — non-saturating gradients mean no vanishing in the active region.

Receptive fields — why depth matters

The receptive field of a unit is the region of the input image that can influence it. A single 3×3 conv sees 3×3 input pixels. Stack two 3×3 convs and a unit sees 5×5; stack three, 7×7. Add stride/pooling and the receptive field grows multiplicatively. This is the crux: early layers have small receptive fields and learn local patterns (edges, color blobs); deep layers have large receptive fields and learn global, semantic concepts (object parts, whole objects). The hierarchy of features is a direct consequence of stacking local operations.

A useful design fact (VGG's insight): two stacked 3×3 convs have the same 5×5 receptive field as one 5×5 conv but use fewer parameters ($2 \cdot 9 = 18$ vs $25$ per channel pair) and add an extra nonlinearity. Small filters, stacked deep, won.

The architectural arc

LeNet-5 (1998) — the template still in use today: conv → subsample(pool) → conv → subsample → fully-connected → output. ~60k parameters, learned end-to-end by backprop, beat hand-crafted methods on handwritten digits (MNIST-era).

AlexNet (2012) — same skeleton, scaled up and modernized:

5 convolutional layers + 3 fully-connected layers, ~60 million parameters, ~650k neurons.
ReLU nonlinearities (fast training), dropout in the FC layers (regularization against overfitting), local response normalization, overlapping max pooling.
Trained on two GPUs (the model was split across them — see Krizhevsky 2014 for the parallelization trick).
Heavy data augmentation (crops, flips, PCA color jitter).
Result: top-1 / top-5 error of 39.7% / 18.9% on ILSVRC-2010 — a huge leap over the prior art, launching the deep-learning vision era.

ResNet (2015) — solved the problem that blocked going deeper. Naively stacking more layers made networks worse even on training data — the degradation problem (not overfitting; an optimization failure). He et al.'s fix: let each block learn a residual relative to its input rather than a fresh mapping. A block computes

$$ \mathbf{y} = \mathcal{F}(\mathbf{x}, \{W_i\}) + \mathbf{x} $$

where the $+\mathbf{x}$ is an identity shortcut (skip connection). If the optimal function is close to identity, the network just drives $\mathcal{F}\to 0$ — easy to learn. The shortcut also gives gradients a clean highway backward, taming vanishing gradients. ResNets came in 18/34/50/101/152 layers; the deep ones use bottleneck blocks (1×1 reduce → 3×3 → 1×1 expand) to stay cheap. ResNet-152 is 8× deeper than VGG yet lower complexity, and the ensemble hit 3.57% top-5 error, winning ILSVRC-2015 plus COCO detection (+28% relative). The residual idea then propagated everywhere — including the residual streams of [[attention-and-transformers]].

Pseudocode: a forward pass

# One conv layer (naive; real libs use im2col / Winograd / FFT)
def conv2d(x, W, b, stride=1, pad=0):
    # x: (C_in, H, W_in)   W: (C_out, C_in, k, k)   b: (C_out,)
    x = zero_pad(x, pad)
    C_out, C_in, k, _ = W.shape
    H_out = (x.H - k) // stride + 1
    W_out = (x.W - k) // stride + 1
    out = zeros(C_out, H_out, W_out)
    for o in range(C_out):
        for i in range(H_out):
            for j in range(W_out):
                r, c = i*stride, j*stride
                patch = x[:, r:r+k, c:c+k]          # (C_in, k, k)
                out[o, i, j] = (patch * W[o]).sum() + b[o]
    return out

def relu(x):       return maximum(0, x)
def maxpool2(x):   return block_max(x, window=2, stride=2)

# A tiny LeNet-style stack
def cnn_forward(img):
    h = relu(conv2d(img, W1, b1, pad=1));  h = maxpool2(h)
    h = relu(conv2d(h,   W2, b2, pad=1));  h = maxpool2(h)
    h = global_avg_pool(h)                 # (C,) — collapse spatial dims
    return softmax(h @ W_fc + b_fc)        # class scores

# A ResNet block: output = F(x) + x
def res_block(x):
    f = relu(batchnorm(conv2d(x, Wa, ba, pad=1)))
    f =      batchnorm(conv2d(f, Wb, bb, pad=1))
    return relu(f + x)                      # the identity shortcut

Training is ordinary backprop / gradient descent: the convolution's gradient w.r.t. its kernel is itself a convolution, so the same sliding machinery runs in reverse. The from-scratch math (chain rule, gradient flow) is exactly the engine covered in [[neural-network-from-scratch]].

Key ideas & tradeoffs

Weight sharing + locality → far fewer parameters and a strong, correct prior for images. This is the reason conv beats a plain MLP on vision: an MLP must independently learn every feature at every pixel location from limited data; a CNN learns it once and reuses it, so it generalizes from less data.
Translation equivariance vs. invariance. Convolution is equivariant (shift the input → shift the output). Pooling and global pooling add a degree of invariance (the answer stops caring exactly where). Useful, but it also means vanilla CNNs are not rotation- or scale-invariant — hence heavy data augmentation.
Depth via residuals. Skip connections decouple "make it deeper" from "make it un-trainable." Nearly every deep net since (CNN or transformer) uses residual paths.
Small filters, stacked deep beat large filters (more nonlinearity, fewer params).
Receptive field is a budget. You must engineer enough depth/stride/dilation for deep units to "see" the whole object — too shallow and the network is physically blind to global structure.
Cost profile. CNNs are cheaper and more data-efficient than Vision Transformers on small/medium datasets and on-device; ViTs tend to win at very large scale with lots of data.

Honest caveats

"Convolution" is a lie of convenience. Frameworks compute cross-correlation (no kernel flip). It doesn't matter because the kernel is learned, but it confuses anyone who knows the signal-processing definition.
CNNs are not magically invariant. They're translation-equivariant, but brittle to rotation, scale, and adversarial perturbations. Robustness comes largely from augmentation and scale, not the architecture alone.
Texture bias. ImageNet-trained CNNs often classify by texture more than shape, which makes them fragile to distribution shift — a documented, repeatedly-confirmed failure mode, not a quirk.
AlexNet's specifics aged. Local response normalization was later found to add little and is essentially abandoned (BatchNorm and others replaced it). The two-GPU split was an engineering workaround for 2012 memory limits, not a principled design.
Pooling is contested. Some argue max pooling discards useful spatial information; strided convs and attention are common replacements. There's no single "correct" downsampling.
The transformer caveat. Vision Transformers and hybrid conv-attention models now match or beat pure CNNs on many large-scale benchmarks. CNNs are foundational and often the right practical choice, but they are no longer the unconditional state of the art across the board.
Interpretability is partial. We can visualize early-layer edge detectors cleanly; deep-layer features are far murkier — see [[mechanistic-interpretability]] for the honest state of "what is this neuron doing."

How it connects to OpenAlice + the Academy ladder

CNNs are the vision foundation rung of the Academy ladder. The recommended path:

[[neural-network-from-scratch]] — backprop, gradients, an MLP by hand. CNNs are "MLPs with weight sharing + locality," so this is the prerequisite engine.
CNNs (this article) — the convolution/pooling primitives, receptive fields, and the LeNet → AlexNet → ResNet arc. The residual connection you meet here reappears everywhere downstream.
[[attention-and-transformers]] — the architecture that generalized beyond grids. Transformers replace the fixed local receptive field with learned global attention, but keep CNN-era ideas: residual streams, layer stacking, hierarchical features. Reading CNNs first makes the trade ("local hard-coded prior" → "global learned mixing") legible.
[[multimodal-llms]] — where it all meets: many vision-language models still use a conv stem or a ViT (which itself borrows the patch-embedding idea from convolutions) to turn pixels into tokens an LLM can read. The [[embeddings]] those vision encoders produce are the bridge between images and language.

Within the OpenAlice ecosystem, CNN intuitions inform how Alice perceives and reasons over visual/grid-structured inputs, and the receptive-field / hierarchical-feature mental model is reused when reasoning about how any deep stack builds abstraction layer by layer. Related generative vision lives in [[diffusion-models]], which share the conv U-Net backbone as their workhorse denoiser. For Atlas curators, this article is the canonical entry point for "how does a CNN actually work" — link here rather than re-deriving convolution arithmetic.