Autoencoders

What it is (intuition first)

An autoencoder is a neural network trained to copy its input to its output — but through a deliberate constraint that makes the copy hard. You force the data through a narrow middle layer (a bottleneck), so the network cannot just memorize and pass values through; it has to learn a compressed code that captures what actually matters about the data, then rebuild the original from that code.

Picture squeezing a 784-pixel handwritten digit down to 32 numbers and back. To reconstruct the "7" from only 32 numbers, the network must discover that the data lives on a much smaller manifold than the raw pixel count suggests — strokes, slants, loops — not arbitrary noise. The 32-number code is a learned representation. The reconstruction is just the training signal that pressures the code into being good. Once trained, you usually throw the decoder away and keep the encoder as a feature extractor.

Three things make autoencoders interesting:

They are self-supervised. No labels needed — the input is the target. This is the same trick that powers BERT and GPT pretraining: predict part of the data from the rest.
The bottleneck is a regularizer. Compression is what forces abstraction. Remove it (let the code be as wide as the input) and a linear autoencoder just learns the identity — useless.
They are a foundational lineage, not a dead-end. The same encoder→latent→decoder skeleton, with different constraints, becomes the denoising AE, the sparse AE (now central to LLM interpretability), the masked autoencoder (MAE) that powers modern vision pretraining, and the variational autoencoder (VAE) whose latent space underpins Stable Diffusion. See vaes (when present) and [[diffusion-models]].

Why it matters

Before autoencoders, "unsupervised feature learning" mostly meant PCA. A linear autoencoder with a squared-error loss provably recovers the same subspace as PCA — but autoencoders generalize that idea to nonlinear manifolds, stacked depth, and arbitrary corruption processes. Vincent et al. (2008, 2010) showed that stacked denoising autoencoders learned representations that, fed to a downstream SVM, closed and sometimes beat the gap against Deep Belief Networks — and that the learned filters were Gabor-like edge detectors on natural images and stroke detectors on digits. That was an early, concrete demonstration that self-supervised reconstruction discovers the same primitives as the visual cortex, with no labels.

The deeper reason they matter: autoencoders are the cleanest setting for the central question of deep learning — what makes a good representation? Bengio, Courville & Vincent's Representation Learning review (2012) frames the whole field around this, and autoencoders are its laboratory animal. Every modern self-supervised method (MAE, JEPA in [[world-models]], contextual [[embeddings]]) is a descendant of "corrupt the input, predict the original, keep the encoder."

And the lineage is alive in production:

MAE is a default vision pretraining recipe and a backbone choice for [[multimodal-llms]].
Sparse autoencoders are the leading tool in [[mechanistic-interpretability]] for decomposing LLM activations into interpretable features.
VAE latents are the compressed space inside which latent [[diffusion-models]] actually run — the autoencoder is what makes high-res image generation tractable.

How it works (real mechanics)

The basic objective

An autoencoder is two maps. An encoder $f_\theta$ takes input $x \in \mathbb{R}^d$ to a code $z \in \mathbb{R}^k$, and a decoder $g_\phi$ maps $z$ back to a reconstruction $\hat{x} \in \mathbb{R}^d$:

$$z = f_\theta(x), \qquad \hat{x} = g_\phi(z).$$

Train both jointly to minimize a reconstruction loss over the dataset:

$$\mathcal{L}(\theta,\phi) = \frac{1}{N}\sum_{i=1}^{N} \ell\big(x^{(i)}, g_\phi(f_\theta(x^{(i)}))\big).$$

For real-valued inputs $\ell$ is squared error $\lVert x - \hat{x}\rVert_2^2$; for binary/pixel-probability inputs it's binary cross-entropy. The undercomplete case ($k < d$) is the classic one: the bottleneck width $k$ controls how much compression — and therefore how much abstraction — is forced.

Minimal pseudocode:

# Undercomplete autoencoder, one training step
z      = encoder(x)            # x:(B,d) -> z:(B,k),  k < d
x_hat  = decoder(z)            # z:(B,k) -> x_hat:(B,d)
loss   = mse(x_hat, x)         # or bce for [0,1] pixels
loss.backward(); opt.step()
# At inference: keep `z = encoder(x)` as the feature; discard decoder.

The linear/PCA connection. If $f$ and $g$ are linear and $\ell$ is squared error, the optimal autoencoder spans the same subspace as the top-$k$ principal components. So "autoencoder" $\supset$ "PCA" — the nonlinearity and depth are what buy you more.

Denoising autoencoder (DAE)

A wider code can trivially learn the identity (just copy), defeating the point. The denoising AE (Vincent et al. 2008) fixes this with a different constraint: corrupt the input, reconstruct the clean version.

$$\tilde{x} \sim C(\tilde{x}\mid x), \qquad z = f_\theta(\tilde{x}), \qquad \hat{x} = g_\phi(z), \qquad \mathcal{L} = \ell(x, \hat{x}).$$

The corruption $C$ is masking (zero out random inputs), additive Gaussian noise, or salt-and-pepper. The key insight: to undo corruption, the network must learn the structure of the data distribution — it has to know that a partially-erased "7" maps back to a full "7," which means it has learned what 7s look like. Geometrically, the DAE learns to push corrupted points back toward the high-density data manifold; the reconstruction is a nearby, higher-density point than the noisy input. Stacking DAEs layer-by-layer (greedy, then fine-tune) gave the stacked denoising autoencoder, an early deep-pretraining workhorse.

x_tilde = x * (rand_like(x) > p_drop)   # masking-style corruption
x_hat   = decoder(encoder(x_tilde))
loss    = mse(x_hat, x)                  # target is the CLEAN x

Sparse autoencoder

Instead of a narrow bottleneck, allow a wide (overcomplete) code but penalize how many units fire. Add an L1 penalty (or a KL penalty toward a low target activation $\rho$):

$$\mathcal{L} = \lVert x - \hat{x}\rVert_2^2 \;+\; \lambda \lVert z \rVert_1.$$

This yields a dictionary: each code unit becomes a reusable, often interpretable "feature," and any input is a sparse combination of a few of them. This is exactly the recipe now used in [[mechanistic-interpretability]] to decompose dense, polysemantic LLM activations into many sparse, monosemantic features — you train a sparse AE on a layer's activations, and each dictionary atom tends to correspond to one human-legible concept.

Masked autoencoder (MAE) — the modern scale-up

He et al. (2021) took the denoising/masking idea to vision Transformers and made it scale. The recipe (CVPR 2022):

Split an image into patches; mask a very high ratio — 75% of patches at random.
An asymmetric encoder/decoder: the encoder (a ViT) sees only the ~25% visible patches — no mask tokens — so it processes a quarter of the sequence and trains far faster. A lightweight decoder then takes the encoded visible patches plus mask tokens and reconstructs the raw pixels of the missing patches.
Reported results: >3× faster pretraining than alternatives, and a vanilla ViT-Huge reaching 87.8% top-1 on ImageNet-1K using only ImageNet-1K data, with strong transfer that exceeds supervised pretraining.

The high mask ratio is the crucial design choice: images are spatially redundant, so unless you remove most of the signal, "reconstruct the rest" is too easy to force good representations — a direct echo of why the denoising AE corrupts aggressively. MAE is, almost exactly, BERT for pixels: mask a large fraction, predict the missing part, keep the encoder.

The VAE branch (where reconstruction becomes generation)

The variational autoencoder (Kingma & Welling, 2014) reinterprets the autoencoder probabilistically. The encoder becomes a recognition model $q_\phi(z\mid x)$ outputting a distribution (a mean and variance), the decoder becomes a generative model $p_\theta(x\mid z)$, and training maximizes the evidence lower bound (ELBO):

$$\mathcal{L}_{\text{ELBO}} = \underbrace{\mathbb{E}_{q_\phi(z\mid x)}[\log p_\theta(x\mid z)]}_{\text{reconstruction}} \;-\; \underbrace{D_{\mathrm{KL}}\big(q_\phi(z\mid x)\,\Vert\,p(z)\big)}_{\text{regularize toward prior } p(z)=\mathcal{N}(0,I)}.$$

The first term is a probabilistic reconstruction loss; the second pulls each input's latent toward a standard normal so the latent space is smooth and samplable. The trick that makes it differentiable is reparameterization: instead of sampling $z \sim \mathcal{N}(\mu,\sigma^2)$ (non-differentiable), write $z = \mu + \sigma \odot \epsilon$ with $\epsilon \sim \mathcal{N}(0,I)$, so gradients flow through $\mu,\sigma$. The difference from a plain autoencoder in one line: a plain AE maps $x\to z$ to a single point with no structure on $z$, so you can't meaningfully sample new data; a VAE maps $x$ to a distribution and regularizes the latent space, so you can sample $z\sim p(z)$ and generate. That sampleable latent is what latent [[diffusion-models]] like Stable Diffusion diffuse inside. Full treatment lives in the sibling vaes article (write it as prose if not yet present).

Key ideas & tradeoffs

The constraint *is* the method. A bare reconstruction loss learns nothing useful unless something forces abstraction: a narrow bottleneck (undercomplete), corruption (denoising/masked), a sparsity penalty (sparse), or a prior + KL (variational). Choosing the constraint is the design decision.
Bottleneck width $k$ is a knob, not a free parameter. Too wide → identity function, no learning. Too narrow → reconstruction blurs and discards real structure. There is no universal $k$; it trades fidelity against abstraction.
Reconstruction objective shapes the failure mode. Pixel/L2 reconstruction is notoriously blurry — it averages plausible completions. This is why VAEs produce soft images and why later generative work moved to adversarial or diffusion objectives. MAE sidesteps it by caring about the encoder, not reconstruction quality.
Asymmetry is an efficiency lever. MAE's "encoder sees only visible patches" trick is general: when the decoder is throwaway, make it small and make the encoder cheap. The encoder is the asset.
AE vs. PCA: autoencoders generalize PCA to nonlinear manifolds and depth — but PCA is convex, deterministic, and has no local minima; an autoencoder can underperform PCA if poorly trained.

Honest caveats

A trained autoencoder's reconstruction quality tells you little about representation quality. Low reconstruction loss can coexist with useless features (the network learned to copy), and a good downstream representation can come with mediocre reconstructions (MAE). Always evaluate the code on a downstream task, not by eyeballing $\hat{x}$.
Plain autoencoders are largely obsolete as generative models. For generation, VAEs, [[diffusion-models]], and autoregressive models dominate; the plain AE survives mainly for compression, denoising, anomaly detection, and as a representation-learning teaching device.
Sparse-AE interpretability is promising but not solved. Dictionary features are more interpretable, not fully interpretable; reconstruction error, dead/dead-on-arrival latents, feature splitting, and how to choose the dictionary size are open problems. Treat sparse-AE "features" as hypotheses, not ground truth — see the caveats in [[mechanistic-interpretability]].
Masking/corruption hyperparameters are domain-specific. MAE's 75% works for images because images are redundant; the right ratio for text (BERT ≈ 15%) is very different, because language is denser. Don't transplant a mask ratio across modalities.
"Autoencoder" is an overloaded word. A denoising AE, a sparse AE, an MAE, and a VAE share a skeleton but optimize genuinely different objectives with different uses. Naming one "the autoencoder" hides those differences — be specific.

How it connects to OpenAlice + the Academy ladder

Where it sits in the ecosystem. Autoencoders are the conceptual root of two things OpenAlice uses directly. First, [[embeddings]]: an encoder that maps inputs to a compact, semantically structured vector is exactly what powers retrieval/RAG across the stack — the autoencoder is the "why" behind learned vector spaces, and the [[embeddings]] article is the "how we use them." Second, interpretability tooling: sparse autoencoders are the mechanism behind the feature-decomposition work catalogued in [[mechanistic-interpretability]], relevant whenever we need to audit what a model is representing rather than just what it outputs.

Academy ladder placement. This is a foundations-tier article — it sits beside [[embeddings]] and below the generative-models rung. A natural learning path:

Math + nets first. Ground the gradient/backprop machinery via the from-scratch articles ([[neural-network-from-scratch]], [[micrograd]], [[math-for-ml-foundations]]).
This article — the cleanest setting to internalize representation learning and self-supervision with no labels and a single loss.
Branch up the lineage: - generative branch → vaes (prose if absent) → [[diffusion-models]] (the VAE latent is where diffusion lives) → [[world-models]] (predict-in-representation-space, JEPA); - representation branch → [[embeddings]] → retrieval/[[graphrag]]; - architecture branch → [[attention-and-transformers]] (MAE's encoder is a ViT; masked prediction is the self-supervised idea behind BERT/GPT); - interpretability branch → [[mechanistic-interpretability]] (sparse AEs on activations).

The single most transferable takeaway for a beginner: self-supervised pretraining = corrupt the data, predict the original, keep the encoder. Autoencoders are where that idea is at its purest, and almost everything modern — masked language modeling, MAE, JEPA — is a variation on it.