Representation Engineering & Activation Steering

One-line summary. Inside a running language model there are directions in activation space that mean human things — "this is honest," "I'm about to refuse," "the sentiment is positive." Representation engineering (RepE) is the program of finding those directions (mostly by contrasting activations on opposite prompts), and activation steering is adding or erasing them mid-forward-pass to change the model's behavior at inference time, with no fine-tuning and no gradient updates. It's a control knob you discover from a few hundred examples and apply with a single vector addition. The catch: the same trick that makes a model more honest also, run in reverse, surgically removes its safety — which is why this is simultaneously an alignment tool and an attack.

What it is (intuition first)

The famous king − man + woman ≈ queen arithmetic from word2vec showed that meaning can live in directions you can add and subtract. Representation engineering is that idea, grown up and pointed at the internal activations of a deep transformer instead of static word vectors.

Here is the whole picture in one move. Run the model on a sentence that is clearly honest and record the residual-stream vector at some layer. Run it on a clearly dishonest sentence and record that vector too. Do this for a few hundred contrast pairs and average the differences. What you get is a single vector — call it the honesty direction d. Now, at inference time on a new prompt, just add α·d to the residual stream as the model generates. Positive α pushes the model toward honesty; negative α pushes it toward lying. You changed behavior without touching a single weight, without a training run, with one line of arithmetic.

That is the entire field in miniature. Two halves:

• Reading (RepE-Reading / probing). Where in the network does a concept live, and along which direction? Found by contrasting activations or training a linear probe.
• Controlling (RepE-Control / steering). Inject that direction back in to causally move the behavior. The injection is also the proof the direction was real.

The intellectual lineage is top-down, deliberately the opposite of [[mechanistic-interpretability]]. Where mech-interp works bottom-up — neuron by neuron, circuit by circuit, decompiling the spreadsheet — RepE (Zou et al. 2023) takes its cue from cognitive neuroscience: don't trace every synapse, look at population-level representations of high-level phenomena (honesty, harmfulness, power-seeking, emotion) and manipulate them directly. It trades mechanistic completeness for leverage you can actually use today.

It rests on the linear representation hypothesis: many human-meaningful concepts are encoded as roughly linear directions in activation space, so they can be read by a linear probe and steered by vector addition. This is older than LLMs — Alain & Bengio (2016) showed linear-classifier "probes" on intermediate layers, and that linear separability of features increases with depth. RepE is what happens when you make that hypothesis causal.

Why it matters

• Control without retraining. Fine-tuning is expensive, slow, and global; prompting is fragile and jailbreakable. Steering is a cheap, targeted, reversible third option: derive a vector from a few hundred examples, dial it up or down per-request, and stack it on top of fine-tuning and system prompts (CAA showed it works additively with both).
• A control knob, not a coin flip. Unlike a system prompt ("please be honest"), a steering coefficient is a continuous dial — you can push 2× honest or 0.5× honest. That graded, monotone control is the practical appeal.
• Safety, both edges. Steering a "refusal," "harmfulness," or "sycophancy" direction down is alignment at the representation level. But Arditi et al. (2024) showed the same machinery is a white-box jailbreak: erase the refusal direction and a safety-tuned model complies with harmful requests. RepE is dual-use by construction — this is the honest, uncomfortable core of the topic. See [[ai-safety-and-jailbreaks]].
• Evidence about model cognition. ITI's result — that you can raise truthfulness by intervening on activations — implies the model often internally represents that something is false while saying it anyway. The "honest direction" exists even when the output is a confident lie. That's a real, somewhat unsettling finding about the gap between a model's representations and its behavior.
• It's the practical cousin of mech-interp. SAEs ([[mechanistic-interpretability]]) find an unsupervised dictionary of thousands of features; RepE finds one supervised direction for the concept you care about and steers it today. Different tradeoff on the same linear-representation substrate.

How it works (the real mechanics)

1. Finding a direction — difference-of-means

The workhorse is embarrassingly simple and shockingly effective. Build contrast pairs: positive examples P of a behavior, negative examples N. Collect residual-stream activations at a chosen layer ℓ, average each group, subtract:

d_ℓ  =  mean_{p∈P} h_ℓ(p)  −  mean_{n∈N} h_ℓ(n)        # "difference of means"
d̂_ℓ =  d_ℓ / ‖d_ℓ‖                                     # unit steering direction

That d̂ is the concept direction. Variants for finding it:

• Difference-of-means (mean-diff) — as above. Cheap, robust, and the method behind both CAA and the refusal-direction work. Often beats fancier alternatives.
• PCA on paired differences — RepE's "RepReading": take per-pair difference vectors and grab the top principal component, which captures the dominant axis of variation between the two classes.
• Linear probe — train a logistic-regression classifier w to separate P from N; the weight vector w is the direction. The Alain–Bengio probing tradition.

2. Choosing where — layer and component

Steering is applied at a specific layer (and sometimes a specific component). Empirically:

• Middle layers of the residual stream are usually the sweet spot — concepts are abstract there, and the injection has many remaining layers to propagate through. Too early = not yet formed; too late = no layers left to act on it.
• ITI (Li et al. 2023) goes finer: it probes individual attention heads, finds the top-K heads whose activations best separate true/false, and intervenes only on those heads — "minimally invasive" steering. On Alpaca, ITI raised TruthfulQA truthfulness from a reported 32.5% → 65.1% using only a few hundred examples (vendor/author-reported single-benchmark number; treat as a demonstration, not a universal guarantee).

3. Applying it — the three operators

Once you have d̂ and a layer, there are three things you can do to the activation h:

# (a) Activation addition  — push toward / away from the behavior
h' = h + α · d̂                       #  α>0 amplify,  α<0 suppress

# (b) Directional ablation  — surgically remove the concept entirely
h' = h − (h · d̂) d̂                   #  project the component out at EVERY layer

# (c) Affine / clamp        — set the component to a target value
h' = h − (h · d̂) d̂ + value · d̂

• Activation addition is the graded dial (CAA, ActAdd, RepE-Control). Coefficient α (and its sign) is the knob.
• Directional ablation is the scalpel: zero out the component along d̂ at every layer and the model cannot express that behavior. This is exactly how Arditi et al. disable refusal.

4. The named methods (what to actually cite)

The convergence is the story: ActAdd, CAA, and the refusal work all land on essentially difference-of-means as the right primitive. The simplest method won.

5. The refusal direction — the load-bearing case study

Arditi et al. (2024) is the cleanest demonstration of the whole paradigm and worth knowing in detail. They take harmful instructions vs harmless ones, compute the mean-difference direction in the residual stream, and show across 13 open chat models up to 72B:

• Ablate d̂ (project it out at every layer) → the model stops refusing harmful instructions — a surgical white-box jailbreak with minimal damage to other capabilities.
• Add d̂ → the model refuses even benign requests.
• They further show that adversarial suffix jailbreaks (the optimized gibberish strings from GCG-style attacks) work in part by suppressing the propagation of this same refusal direction — connecting hand-crafted steering to discovered adversarial attacks.
• The ablation can be baked into the weights ("weight orthogonalization"), producing a permanently-jailbroken model with no inference-time hook.

Honest framing: later work — including a 2025/2026 follow-up literally titled "There Is More to Refusal… than a Single Direction" — argues refusal is not purely one-dimensional, and that the single-direction picture is a strong first-order approximation rather than the full truth. The geometry is richer than one line. Cite the single-direction result as a powerful, partial model.

Key ideas & tradeoffs

• Top-down vs bottom-up. RepE/steering buys you usable control today by working at the population/representation level; [[mechanistic-interpretability]] (SAEs, circuits) buys you mechanistic completeness but is slower and explains only a fraction. They share the linear-representation substrate; they differ on supervision (RepE supervises with contrast pairs; SAEs are unsupervised dictionaries) and on goal (control vs understanding).
• Supervised & cheap vs unsupervised & broad. A steering vector needs you to name the behavior and supply contrast pairs — but then it's one direction, derived in minutes. SAEs need no labels but produce millions of features you must then interpret. Pick by whether you already know what you want to control.
• Reversible, graded, stackable. Steering is the only behavioral-control method that is simultaneously inference-time, continuously dial-able, and additive on top of fine-tuning and prompts. That combination is its moat.
• Causality is built in. Unlike a probe (which only correlates), injecting the direction and observing the behavior change is a causal intervention — the steering is the validation that the direction was real.
• The dual-use floor. Every "make it more X" steering vector is a "make it less X" vector with a sign flip. A safety direction is a jailbreak direction reversed. There is no version of this technique that is control-only.

Honest caveats & open questions

• The linear hypothesis is an approximation, not a law. Steering works because concepts are roughly linear — but "roughly." Some behaviors are not captured by one direction (the refusal-is-more-than-one-direction critique), and non-linear ITI variants exist precisely because the linear version leaves performance on the table. Treat "concept = direction" as a useful first-order model.
• Off-target damage & the capability tax. Add too much α and the model degrades — repeats itself, breaks fluency, loses unrelated skills ("Golden-Gate-Claude"-style obsession is the friendly version). CAA and ActAdd report minimal off-target loss, but that's measured on chosen benchmarks; aggressive steering always costs something. Tune the coefficient; there is no free lunch.
• Generalization & transfer are uneven. A direction found on one dataset, layer, or model may not transfer cleanly. Vector quality depends on the contrast-pair design — bad pairs give a direction that encodes a confound (formatting, length) instead of the concept.
• Vendor/single-benchmark numbers. Figures like ITI's 32.5%→65.1% on TruthfulQA are author-reported on one benchmark. They demonstrate the effect, not a guaranteed delta on your task or model. TruthfulQA itself is a contested, gameable benchmark. Don't quote these as if they were universal.
• Steering ≠ understanding. A working steering vector tells you a direction correlated-and-causal-for a behavior exists; it does not tell you the circuit that computes it, nor that the direction is "the" representation rather than one of several. It's control without a full mechanistic account.
• Safety brittleness is the headline. The refusal-direction result means current RLHF safety can be surgically removed from open weights by anyone with activation access — and baked permanently into the weights. This is a structural fact about how safety-tuning sits in the representation, not a bug to be patched. See [[ai-safety-and-jailbreaks]] and [[rlhf-and-alignment]].

How it connects to OpenAlice

OpenAlice orchestrates frontier models via API rather than training or running white-box hooks, so today it uses the philosophy of representation engineering more than the literal activation surgery:

• Steering, the prompt-level version. Alice's behavior is steered through prompts, [[agent-memory-systems]], and the three mode-axes (Companion/Worker interaction, personality moods, chat/topic mode). Those are control knobs over high-level behavior — the prompt-space analogue of a steering coefficient. The principled endpoint of "dial Alice's honesty/warmth/caution" is exactly an activation steering vector; it's the technique to watch if/when OpenAlice runs an open-weights model it can hook.
• Why we don't naively trust system prompts for safety. The refusal-direction result is the clean argument for why the repo's safety story is layered (approval protocols, denylist floors, kill switches, HITL) rather than relying on the model's own learned refusal: representation-level safety is removable and prompt-level safety is jailbreakable. Defense-in-depth at the system boundary is the rational response to RepE's dual-use floor.
• Personality-envelope, not cosine-distance. The repo's insistence on behavioral-envelope non-regression (not embedding-cosine) for Alice's personality echoes a RepE lesson: a direction looking preserved in representation space is not proof the behavior is preserved. Validate by causal behavior, the way steering validates a direction by injecting it.
• Education. For OpenAlice Academy this sits beside [[mechanistic-interpretability]] as the control-and-top-down half of interpretability — the cheap, usable, dual-use cousin of SAEs and circuits.

See also

• [[mechanistic-interpretability]] — the bottom-up sibling: SAEs, circuits, superposition; shares the linear-representation substrate, differs on supervision and goal.
• [[ai-safety-and-jailbreaks]] — the refusal direction is a white-box jailbreak; steering is dual-use by construction.
• [[rlhf-and-alignment]] — the behavioral alignment that steering makes mechanistic (and can surgically undo).
• [[embeddings]] — king−man+woman and the direction-arithmetic intuition steering generalizes to live activations.
• [[lora-and-peft]] — the fine-tuning alternative steering stacks on top of and competes with for cheap control.
• [[llm-inference-internals]] — where the residual-stream hook that applies a steering vector actually lives.
• [[test-time-compute-reasoning]] · [[agent-memory-systems]] — other inference-time levers for shaping behavior without retraining.