Mechanistic Interpretability

A trained neural network is a 100-billion-number spreadsheet that nobody wrote and nobody can read. Mechanistic interpretability ("mech interp") is the attempt to reverse-engineer that spreadsheet back into algorithms — to open the black box and say, in human terms, this is the circuit that copies the previous token, this is the direction in space that means "Golden Gate Bridge," and here is the exact path of arithmetic the model ran to answer "what is 36+59?". It treats a network like a compiled binary and tries to recover the source code.

What it is (intuition first)

Most of ML interpretability asks "which input mattered?" — saliency maps, attention heatmaps, "the model looked at the word not when it predicted bad." That tells you where the model paid attention, not what computation it ran. Mechanistic interpretability is more ambitious and more literal: it asks "what is the actual algorithm implemented by these weights, and can I point to the specific parameters that implement it?"

The grounding analogy from the field's founders is a decompiler. A compiler turns readable code into an opaque binary; gradient descent turns a readable task ("predict the next token") into an opaque pile of weights. Mech interp is the decompiler that runs in reverse — recovering loops, lookup tables, and copy operations from the raw floating-point.

Three load-bearing hypotheses make this even plausible (the "circuits↗" research program):

1. Features. The network's fundamental unit isn't the neuron — it's a feature: a direction in activation space that corresponds to a human-meaningful concept (e.g. "this text is in Arabic," "we are inside a for loop," "the speaker is being deceptive"). A feature is a vector; the activation along that vector is how much of that concept is present.
2. Circuits. Features are connected by the weights into circuits — sub-graphs of the network that compute something specific. An induction head ("if I saw AB earlier and just saw A, predict B") is a circuit. The "copy the previous token" operation is a circuit.
3. Universality. The same features and circuits tend to re-form across different models and architectures (curve detectors in vision nets; induction heads in nearly every transformer). If true, interpretability findings transfer — you don't re-decompile every model from scratch.

If you understand attention and transformers↗ and embeddings↗, you already have the substrate; mech interp is the science of reading meaning back out of those vectors and matrices.

Why it matters

• Safety & alignment. A model that is helpful on the eval set might be deceptive, sycophantic, or backdoored in ways no behavioral test catches. If you can find the deception feature or trace the circuit that decides to lie, you can audit, monitor, and intervene at the mechanism level instead of guessing from outputs. Anthropic's scaling work explicitly surfaced features for deception, sycophancy, dangerous content, and "unsafe code." This is the deepest level of RLHF and alignment↗ — not shaping behavior, but reading the cause of behavior.
• Debugging & capability. "Why did the model get this arithmetic wrong / hallucinate this citation / refuse this benign request?" becomes answerable by tracing the actual computation, not vibes.
• Science of intelligence. Transformers spontaneously invent algorithms (in-context learning, planning ahead when writing poetry, multi-hop reasoning). Mech interp is the empirical microscope — Anthropic calls it "the biology of a large language model"↗, studying a trained model the way a biologist studies a cell.
• Steering. Once you have a feature, you can clamp it — turn the "Golden Gate Bridge" feature to 10× and the model becomes obsessed with the bridge. Feature steering is a surgical, interpretable alternative to fine-tuning or prompt-hacking (see "Golden Gate Claude"↗).

How it works (real mechanics)

1. The residual stream is the bus

The starting move (A Mathematical Framework for Transformer Circuits↗) is to view a transformer not as a stack of layers but as a residual stream: a shared vector channel that every component reads from and writes to additively.

x_0 = embed(tokens)
for each layer:
    x = x + attn(x)      # head writes its output into the stream
    x = x + mlp(x)       # MLP writes its output into the stream
logits = unembed(x_final)

Because everything is added into the same stream, you can decompose the final logits into a sum of contributions from every head and every MLP across every layer. Each component is a term you can study in isolation. This linearity-by-addition is what makes circuit analysis tractable at all.

2. Attention heads factor into QK and OV circuits

A single attention head is usually drawn as four matrices W_Q, W_K, W_V, W_O. The framework's key insight: they collapse into two independent low-rank circuits.

• QK circuit — where to attend. The attention pattern depends only on the bilinear form

  attention_scores  ∝  x_iᵀ (W_Qᵀ W_K) x_j

So the entire "which token attends to which" behavior is governed by one matrix W_Q^T W_K. You can read this matrix directly.

• OV circuit — what to write once you've attended. Conditional on the attention pattern, the head copies information through

  output_contribution  =  (attention_weights) · x · (W_V W_O)

governed by the single matrix W_O W_V.

So a head is "the QK circuit decides where to look; the OV circuit decides what to move there." Crucially, if you freeze the attention pattern, the head becomes a linear function of its input — the whole nonlinearity lives in the softmax that produced the pattern. Freezing attention is the trick that makes most downstream circuit analysis linear and therefore decomposable.

The framework builds up by depth:

• 0-layer: pure bigram statistics — W_E W_U is literally a readable next-token lookup table.
• 1-layer: bigrams + skip-trigrams (patterns like A … B → C).
• 2-layer: heads can compose (Q-, K-, and V-composition), and this is the minimum depth at which induction heads appear. Composition produces "virtual attention heads" — effective heads formed by two real heads working in series.

3. Induction heads — the canonical circuit

The cleanest fully-understood circuit (Olsson et al., 2022↗). An induction head implements: "I am about to predict the token after `A`. Let me find the previous place `A` appeared, look at the token `B` that followed it, and predict `B`." The [A][B] … [A] → [B] pattern.

It's a two-head composition:

1. A previous-token head in an early layer writes, into each position, a copy of what the previous token was.
2. The induction head in a later layer uses its QK circuit to match "the token before me" against those stored previous-tokens (prefix matching), attends back to the matching position, and its OV circuit copies that position's token forward (copying).

The paper marshals six lines of evidence that induction heads drive the bulk of in-context learning (a model getting better at predicting later tokens within a single sequence):

1. Causal ablation in attention-only models removes ICL.
2. Correlational presence in larger MLP-containing models.
3. Co-timing: induction heads form at exactly the training step where ICL ability appears.
4. A visible bump/phase change in the training loss curve at that moment — a discrete, almost biological "organ" forming.
5. Per-token loss analysis showing the improvement is concentrated at later sequence positions (the signature of ICL).
6. Mechanistic plausibility from the small-model circuits generalizing up.

This is the field's proof-of-concept that a non-trivial capability maps onto an identifiable mechanism.

4. Superposition — why neurons are a lie

The obstacle to "just read the neurons" is polysemanticity: a single neuron fires for unrelated concepts (academic citations and HTTP requests and Korean text). The explanation is superposition (Toy Models of Superposition↗):

A network wants to represent more features than it has dimensions. It does so by storing features as non-orthogonal directions in activation space, tolerating a little interference. Neurons end up as arbitrary projections of this packed-together feature set — so they look polysemantic.

The toy model: take sparse, independent features x, squash them through a bottleneck W and try to reconstruct:

h = W x                       # project m features down to n < m dimensions
x̂ = ReLU(Wᵀ h + b)           # try to reconstruct the features
loss = Σ importanceᵢ · (xᵢ − x̂ᵢ)²

What gradient descent does with W depends critically on sparsity (how often features co-occur):

• Dense features → the network gives the top-n features their own orthogonal dimension and throws away the rest. No superposition.
• Sparse features → because two rare features rarely fire at once, the network can pack them into the same dimensions and accept rare collisions. Sparsity enables superposition. This is the central result.

The geometry it discovers is startlingly clean — features arrange into antipodal pairs (two features sharing one line, pointing opposite ways) and higher structures (digons, triangles, pentagons, tegum products of these). There are phase transitions: as you sweep sparsity, the network discretely jumps between "dedicated dimension" and "superposed" regimes.

Honest scope: this is proven in a toy model and offers a compelling hypothesis for real networks — it does not prove real LLMs use exactly this scheme. But it justifies the next tool.

5. Sparse autoencoders (SAEs) — pulling features back out

If concepts are superposed across neurons, you need to un-mix them. The tool is dictionary learning via a sparse autoencoder (Towards Monosemanticity, 2023↗).

Idea: take the model's activation vector a (e.g. the residual stream at one layer) and re-express it as a sparse combination of a large overcomplete dictionary of directions. Most dictionary atoms are off; the few that fire are your monosemantic features.

# Encoder: activations  ->  many sparse feature activations  (f >> d)
z = ReLU(W_enc (a − b_dec) + b_enc)     # z ∈ R^f, the feature activations
# Decoder: reconstruct activations as a sparse weighted sum of dictionary atoms
â = W_dec z + b_dec                       # columns of W_dec are the features

# Loss: reconstruct well, AND keep z sparse
L = ‖a − â‖²            (reconstruction / fidelity)
  + λ · ‖z‖₁           (L1 sparsity penalty → few features active at once)

The dictionary is overcomplete (more features f than activation dimensions d, e.g. 8×–256× expansion) — that's the point, since superposition packed more concepts than dimensions. The ‖z‖₁ term forces the model to explain each activation with only a handful of atoms, and sparsity is what buys interpretability: a feature that only fires in narrow, consistent contexts is monosemantic.

Empirically (2023 result↗): trained on a 1-layer transformer's MLP activations, SAEs pulled out clean features (Arabic script, DNA sequences, specific HTTP/code contexts) where human raters found ~70% cleanly mapped to a single human concept — a sharp improvement over raw neurons.

Then it scaled (Scaling Monosemanticity, 2024↗): SAEs with up to 34 million features trained on the middle-layer residual stream of Claude 3 Sonnet — a production model at the time of publication (the Claude family is now at 4.x). Findings:

• Features for famous entities (the now-iconic Golden Gate Bridge feature 34M/31164353), but also abstract ones: sarcasm, code errors, gender bias, sycophancy, deception, and security vulnerabilities.
• Features are multilingual (one "bridge" feature fires across languages) and multimodal (fires on images despite text-only SAE training).
• Features respond to both concrete instances and abstract discussion of a concept.

#### SAE training pathologies and the fixes

L1 has two well-known sins: dead features (atoms that never fire and are wasted) and shrinkage (the L1 term penalizes magnitude, so even correctly-identified features get their activation values pulled toward zero, hurting reconstruction). The 2024–2025 literature fixed both:

• Gated SAEs / [JumpReLU SAEs](https://arxiv.org/abs/2407.14435) (DeepMind) — separate the decision "is this feature on?" from the estimate "how strong is it?", using a learned per-feature threshold. This kills shrinkage and gives state-of-the-art reconstruction at a given sparsity on Gemma 2 9B, with a single forward/backward pass.
• TopK SAEs (OpenAI↗) — drop the L1 penalty entirely; just keep the k largest activations per token. This directly sets the sparsity (L0 = k) and sidesteps shrinkage, at the cost of a partial sort.
• All three (Gated/TopK/JumpReLU) yield roughly equally interpretable features; they trade off training cost vs. reconstruction-vs-sparsity frontier.

6. Feature steering — interpretability you can act on

A feature is a direction d in activation space. You can clamp it: at inference, overwrite the activation along d to a chosen value and let the rest of the forward pass proceed.

a_steered = a − (a·d̂) d̂ + value · d̂      # remove current component, inject target

• Clamp the Golden Gate feature to ~10× its max → "Golden Gate Claude," which steers nearly every reply toward the bridge and even self-identifies as it.
• Clamp a "transit infrastructure" feature to 5× → the model mentions a bridge when it otherwise wouldn't.

This is causal validation (it proves the feature does something, not just correlates) and a practical control knob. It's a cleaner cousin of "activation steering / control vectors," but grounded in an interpretable dictionary atom rather than a hand-built difference vector. It connects directly to RLHF and alignment↗: steering a "sycophancy" or "harmful" feature down is alignment at the mechanism level.

7. The 2025 frontier — circuit tracing & attribution graphs

SAEs explain what features exist; they don't natively show how features cause each other across layers. The 2025 work (Circuit Tracing↗, "On the Biology of an LLM"↗) closes that gap with cross-layer transcoders (CLTs) and attribution graphs.

• A transcoder replaces an MLP with an interpretable sparse-feature layer (read in, sparse features, write out) — like an SAE but it substitutes the computation, not just observes activations.
• A cross-layer transcoder lets a feature at layer ℓ write into every later layer ℓ…L, with separate decoder weights per output layer, trained jointly. CLTs beat per-layer transcoders on the reconstruction-vs-sparsity frontier and produce much shorter causal paths (avg ~2.3 vs ~3.7 steps).
• Swap the model's MLPs for CLT features → a local replacement model. Freeze attention patterns and layernorm denominators from the real forward pass, add error nodes to absorb what the CLTs miss, and the whole thing becomes linear → you can compute exact edge weights between features. That graph is the attribution graph: token inputs → intermediate features → output logits, with the salient pathway pruned out.

With this, Anthropic read off real algorithms in Claude 3.5 Haiku: planning ahead when writing rhyming poetry (picking the rhyme word first, then writing backward toward it), genuine multi-step reasoning, and parallel arithmetic circuits.

Key ideas & tradeoffs

Cross-cutting tensions:

• Faithfulness vs. interpretability. The more you simplify a model to read it (freeze attention, swap in SAEs, prune the graph), the more your explanation is of a different model than the one you're trying to understand.
• Completeness vs. clarity. Bigger dictionaries / lower sparsity explain more but become unreadable; smaller ones are clean but leave "dark matter."
• Automation vs. trust. Auto-labeling millions of features (via an LLM judge) is the only way to scale, but every label is itself a fallible model output.

Honest caveats & open questions

• It explains a fraction, not the whole. Even the best CLT attribution graphs leave large error nodes — "dark matter," computation the interpretable features don't capture. Anthropic states plainly: "we only explain a portion of model computation, and much remains hidden." Claims of "we understand the model" are overclaims.
• Attention is largely unexplained. Attribution graphs freeze attention patterns and only capture frozen OV effects — they do not explain how the attention pattern itself was formed (the QK computation). On tasks that are fundamentally about attention (induction!), the method fails completely.
• Superposition is a toy-model theory. It's an elegant, well-evidenced hypothesis, not a proven property of GPT-4-scale models. The "linear representation hypothesis" (features are directions, concepts compose linearly) is itself contested at the edges.
• SAEs are a lossy, somewhat arbitrary lens. Different SAEs on the same model find different dictionaries; feature splitting means one "concept" fractures into many features as you grow the dictionary (is "the bird feature" one feature or fifty?). There's no ground truth for "the right features," and reconstruction is never perfect.
• The interpretability illusion / faithfulness gap. A feature label that looks clean on cherry-picked examples may misfire elsewhere; a replacement model may reach the right answer by a different mechanism. Perturbations show only ~0.8 cosine fidelity one layer downstream, compounding with depth.
• Inhibition & absence are hard. Methods explain active features well but struggle with suppression circuits and with "this feature being off is the interesting part."
• Scaling cost & the moving target. Training 34M-feature SAEs is expensive; meanwhile models keep getting bigger and change every few months. Interpretability runs perpetually behind capability.
• Open frontiers: attention/QK circuits at scale, automated circuit discovery (not just feature discovery), measuring faithfulness rigorously, weight-space (not just activation-space) interpretability, and whether universality truly holds across architectures like mixture-of-experts↗ and state-space models↗.

How it connects to OpenAlice

OpenAlice runs and orchestrates LLMs; it does not (today) train SAEs or trace circuits. But the mindset of mech interp maps onto several live concerns:

• Alice's "soul" and personality preservation. The repo treats Alice's identity/personality blocks as load-bearing and demands byte-identical prompt composition and behavioral-envelope (not cosine-distance) non-regression checks. That instinct — "don't perturb the mechanism, verify behavior at the entry point" — is the prompt-level analogue of mech interp's faithfulness discipline. The honest framing here is the same one this article preaches: outputs looking right ≠ the internal cause being unchanged.
• Steering & control, the practical version. OpenAlice steers behavior through prompts, agent-memory systems↗, and mode axes (Companion/Worker, personality moods) rather than feature clamps — but feature steering is the principled endpoint of the same goal: change behavior by changing an interpretable internal quantity. If Alice ever needs hard safety guarantees (a "never produce this" knob that prompts can't jailbreak), SAE-based feature steering is the technique to watch.
• Safety gates & HITL. The ecosystem's emphasis on approval protocols, denylists, and kill switches is behavioral safety. Mech interp is the mechanistic layer beneath it — the difference between "we filtered the output" and "we found and suppressed the feature that wanted to produce it." Worth tracking as the alignment story matures.
• Atlas as interpretability-by-analogy. Atlas↗ reverse-engineers the codebase into a queryable graph of symbols, callers, and circuits-of-calls. It's the same epistemic move — recover structure and causality from an opaque artifact — applied to source instead of weights. The "candidate-grade, filter-by-confidence" honesty in Atlas's call-graph mirrors mech interp's "dark matter / error node" honesty exactly.

Where to go next in this library

• Attention & the Transformer↗ — the architecture being decompiled (residual stream, QK/OV live here).
• Embeddings↗ — features are directions in the same vector spaces embeddings inhabit.
• Positional encoding↗ — another readable circuit component.
• RLHF and alignment↗ — the behavioral alignment that mech interp aims to make mechanistic.
• Scaling laws↗ — why features/circuits get more numerous and worth decompiling as models grow.
• Mixture-of-experts↗ · State-space models↗ — architectures where "does universality hold?" is an open interpretability question.