Attention & the Transformer

The one architecture that ate machine learning. Every modern LLM — GPT, Claude, Llama, DeepSeek — is a transformer at its core. This is the engine. Understand this page and most of the rest of the field stops being magic.

What it is (intuition first)

Imagine you're reading the sentence:

"The animal didn't cross the street because **it** was too tired."

When you hit the word "it", your brain instantly knows it refers to "animal", not "street". You did this by looking back at the other words and deciding which ones are relevant to understanding "it". That act — each word looking at every other word and pulling in the information it needs — is attention. (Illustrated Transformer↗)

Before 2017, the dominant way to process sequences (sentences, audio, time-series) was the RNN (recurrent neural network): read word 1, update a memory, read word 2, update the memory, and so on — strictly left-to-right, one step at a time. This had two fatal problems:

1. It's sequential — you can't process word 50 until you've processed words 1–49. That kills parallelism on GPUs.
2. Long-range memory leaks — by the time the RNN reaches word 50, the signal from word 1 has been squeezed through 49 update steps and is mostly gone. "it ↔ animal" across a long sentence is hard.

The 2017 paper "Attention Is All You Need" (Vaswani et al., arXiv:1706.03762↗) made a radical bet: throw out recurrence entirely. No step-by-step memory. Instead, let every word directly look at every other word, all at once, in parallel. The title is the thesis — attention is the only sequence-mixing mechanism you need. The architecture they built is the Transformer.

The payoff: every position is computed simultaneously (great for GPUs), and any word can reach any other word in one step regardless of distance (no long-range decay). That combination is why transformers scaled to the trillion-parameter models we have today.

A useful mental model is a soft, content-based dictionary lookup. Each word emits:

• a Query ("here's what I'm looking for"),
• a Key ("here's what I contain / what I match on"),
• a Value ("here's the actual information I'll hand over").

Every query is compared against every key to get a relevance score, those scores become weights, and each word's output is a weighted blend of all the Values — heavily weighting the words it found relevant. "it" sends out a query that matches the key of "animal", so "it" pulls in "animal"'s value. That's the whole trick. The rest of this page makes it precise.

Sibling pages worth reading alongside this one: [[embeddings]] (how words become vectors in the first place), [[positional-encoding]] (how the model knows word order), [[tokenization]] (what a "word" even is to the model), and [[llm-from-scratch]] / [[microgpt-build-an-llm-from-scratch]] (build the whole thing yourself).

Why it matters

• It's the substrate of modern AI. Essentially every large language model, most state-of-the-art vision models (ViT), speech models, protein models (AlphaFold uses attention), and multimodal systems are transformers or transformer variants. Learning this is learning the lingua franca of the field.
• It unlocked scale. Because attention is fully parallelizable, you can throw enormous compute at it. The transformer is the architecture that made the [[scaling-laws]] story possible — bigger model + more data + more compute → predictably better, no architectural ceiling hit yet.
• It's a general "set/sequence mixer." Attention isn't really about language. It's a permutation-aware way to let a set of vectors exchange information based on content rather than fixed position. That generality is why it transferred to so many domains.
• It's the thing everything else modifies. [[flash-attention]], [[mixture-of-experts]], [[lora-and-peft]], [[quantization]], [[deepseek-architecture]], long-context tricks — almost every advance in the LLM stack is a tweak to the transformer block. You can't understand them without understanding the base.

How it works (the real mechanics)

We'll build it bottom-up: one attention head → multi-head → the full block → the full encoder-decoder. Notation follows the paper.

Step 0: from tokens to vectors

Text is split into tokens ([[tokenization]]) and each token is mapped to a learned vector of dimension d_model = 512 (in the base paper) via an embedding table ([[embeddings]]). A sequence of n tokens is therefore a matrix X ∈ ℝ^(n × d_model).

Because attention itself is order-blind (it's a weighted sum — shuffle the inputs and you get shuffled outputs, no notion of "before/after"), the paper adds a positional encoding to each embedding so the model can tell position 3 from position 30. The original uses fixed sinusoids (Vaswani et al.↗):

PE(pos, 2i)   = sin( pos / 10000^(2i / d_model) )
PE(pos, 2i+1) = cos( pos / 10000^(2i / d_model) )

where pos is the token position and i indexes the dimension. Different dimensions oscillate at different wavelengths (a geometric progression from 2π to ~10000·2π), giving each position a unique, smoothly-varying fingerprint. Full treatment in [[positional-encoding]] (modern models mostly use RoPE or learned encodings instead).

Step 1: Query, Key, Value projections

From the input X, three learned weight matrices project each token into three roles (Raschka↗):

Q = X · W_Q      (queries)
K = X · W_K      (keys)
V = X · W_V      (values)

W_Q, W_K, W_V are trainable parameters — they are the entire knowledge of an attention layer. In self-attention, Q, K, V all come from the same sequence (words attend to each other). Later we'll see cross-attention, where Q comes from one sequence and K, V from another.

Step 2: scaled dot-product attention (the core equation)

This is the heart of the entire architecture. The exact formula from the paper:

                    ⎛  Q · Kᵀ  ⎞
Attention(Q,K,V) = softmax ⎜ ───────── ⎟ · V
                    ⎝   √d_k   ⎠

Read it in four moves:

1. Scores: `Q · Kᵀ`. Take the dot product of every query with every key. The dot product is a similarity measure — a big value means "this query and this key point the same way → they're relevant to each other." For n tokens this produces an n × n matrix: entry (i, j) = how much token i should attend to token j. (This is the omega / "unnormalized attention weights" in Raschka↗: ω_ij = q^(i)ᵀ k^(j).)

1. Scale: `/ √d_k`. Divide every score by the square root of the key dimension d_k. (why — see below, it's a subtle and important detail.)

1. Softmax (row-wise). Turn each row of raw scores into a probability distribution — all positive, summing to 1. Now each token has a set of attention weights saying "I'm 70% about word A, 20% about word B, 10% everything else."

1. Weighted sum of Values: `... · V`. Multiply those weights by the Value vectors and sum. Each token's output is a content-weighted blend of all the values — its new, context-aware representation. (z^(i) = Σ_j α_ij · v^(j).)

That's it. Everything else in the transformer is plumbing around this one operation.

#### Why divide by √d_k? (the variance argument)

This looks like a magic constant but there's a clean reason (Vaswani et al.↗, cross-checked across multiple explainers↗):

Suppose the components of q and k are independent random variables with mean 0 and variance 1. The dot product q · k = Σ_{m=1}^{d_k} q_m k_m is a sum of d_k independent terms each with variance ≈ 1. So the dot product has:

mean ≈ 0,   variance ≈ d_k

The variance grows linearly with `d_k`. With d_k = 64, raw scores have standard deviation ≈ 8 — large. Feeding large-magnitude values into softmax pushes it into a saturated regime: one entry gets nearly all the mass, the rest go to ~0, and the softmax becomes a near-hard argmax. In that regime the gradient is almost zero (vanishing gradients), so the layer barely learns.

Dividing by √d_k rescales the variance back to ≈ 1 (since dividing a variable by √d_k divides its variance by d_k), keeping scores in a sane range where softmax stays soft and gradients flow. The square root is exactly right because variance scales with d_k, so standard deviation scales with √d_k. (Why √dₖ explainers↗)

#### Masking

For autoregressive generation (decoder), a token must not "see the future" — predicting word 5 while peeking at word 6 is cheating. So before the softmax, the upper-triangular entries of the score matrix are set to −∞ (a causal mask). After softmax those become 0 weight. This single trick is what makes a transformer a left-to-right language model. Padding tokens are masked the same way.

Step 3: multi-head attention

One attention operation gives the model one way to relate words. But "it" might need to track both what it refers to (animal) and its state (tired) at the same time. A single softmax-weighted average can't easily do both. The fix: run several attention operations in parallel, each with its own W_Q, W_K, W_V, and let each specialize. (Illustrated Transformer↗)

MultiHead(Q,K,V) = Concat(head_1, ..., head_h) · W_O
where head_i = Attention(Q·W_Q^i, K·W_K^i, V·W_V^i)

Base-model numbers from the paper:

• h = 8 heads
• d_model = 512
• d_k = d_v = d_model / h = 64 per head

Crucially, each head works in a smaller 64-dim subspace, so 8 heads cost roughly the same total compute as one full-width head. Run all 8, concatenate their 64-dim outputs back to 512, then mix them with a final learned matrix W_O. Different heads empirically learn different relationships — some track syntax, some track coreference, some attend to the previous/next token. (Modern models often use far more heads and variants like Grouped-Query Attention to save memory; see [[deepseek-architecture]].)

Step 4: the full transformer block

A single attention layer isn't a transformer. The block wraps it with three more critical pieces. Each sub-layer is wrapped in a residual connection followed by layer normalization (Vaswani et al.↗):

output = LayerNorm( x + Sublayer(x) )

The two sub-layers in an encoder block are:

(a) Multi-head self-attention — tokens exchange information (the "communication" step).

(b) Position-wise feed-forward network (FFN) — applied to each token independently (the "computation" step). It's a two-layer MLP with a ReLU:

FFN(x) = max(0, x·W_1 + b_1) · W_2 + b_2

with inner dimension d_ff = 2048 (4× wider than d_model). This is where a large fraction of the model's parameters and stored "knowledge" live — attention moves information around, the FFN transforms it. (Karpathy's framing: attention = communication between tokens, FFN = per-token thinking.)

Two ideas that make deep stacks trainable:

• Residual connection (`x + Sublayer(x)`): add the input back to the output. This gives gradients a "highway" to flow through dozens of layers without vanishing, and lets each layer learn a refinement rather than a from-scratch transformation. Without residuals, deep transformers don't train.
• Layer normalization: re-center and re-scale each token's vector to keep activations stable across layers. (The paper uses post-norm: LayerNorm(x + Sublayer(x)). Most modern LLMs use pre-norm: x + Sublayer(LayerNorm(x)), which trains more stably at depth.)

Step 5: stacking and the encoder-decoder

The original transformer was built for translation, so it has two stacks (Vaswani et al.↗):

• Encoder: N = 6 identical blocks. Reads the source sentence; every token attends to every other token (bidirectional, no mask). Output: a context-rich representation of the input.
• Decoder: N = 6 identical blocks, but each has three sub-layers instead of two.

This gives the famous three uses of attention:

1. Encoder self-attention — source tokens attend to all source tokens. Bidirectional.
2. Decoder masked self-attention — output tokens attend to earlier output tokens only (causal mask), preserving autoregression.
3. Encoder-decoder (cross) attention — the decoder's queries come from the decoder, but the keys and values come from the encoder's output. This is how the translation reaches back into the source sentence to decide what to generate next.

Modern LLMs (GPT, Claude, Llama) are decoder-only — they drop the encoder and cross-attention entirely, keeping just the masked-self-attention + FFN block, stacked many times (e.g. dozens to 100+ layers). The final layer's output for each position is projected through a vocabulary-sized matrix + softmax to predict the next token. That decoder-only stack is a modern LLM.

Putting it together (pseudocode in prose)

To process a sequence: embed the tokens and add positional encodings → feed into the first block. In each block: (1) compute Q, K, V for all tokens, (2) for each head, do scaled dot-product attention with the causal mask, (3) concat heads and project with W_O, (4) add the residual and layer-norm, (5) pass through the position-wise FFN, (6) add the residual and layer-norm again. Repeat for all N blocks. At the top, project each position to vocabulary logits and softmax to get next-token probabilities. Training is just next-token prediction via cross-entropy, with [[positional-encoding]], dropout (P_drop = 0.1) and label smoothing (ε = 0.1) as regularization.

Key ideas & tradeoffs

The single most consequential tradeoff is the quadratic cost. Because every token attends to every token, both compute and the attention-score memory scale as O(n²) in sequence length (complexity analysis↗). Double the context, quadruple the cost. This is the central bottleneck behind every "long-context" effort in the field — and the reason for [[flash-attention]], sliding-window attention, and entirely different architectures like [[state-space-models]] (Mamba) that trade attention's exactness for linear scaling.

Honest caveats & open questions

• Quadratic scaling is fundamental, not just an implementation wart. FlashAttention↗ (Dao et al., arXiv:2205.14135) makes attention dramatically faster and more memory-efficient by being IO-aware — tiling the computation to minimize slow GPU-HBM ↔ fast-SRAM transfers — but it computes exact attention; it does not lower the asymptotic O(n²) compute. In fact there's evidence the quadratic time is unavoidable for exact attention unless the Strong Exponential Time Hypothesis is false (arXiv:2209.04881↗). Genuinely sub-quadratic exact attention may not exist. See [[flash-attention]].
• The √d_k argument is an approximation. It assumes query/key components are independent, zero-mean, unit-variance. After training, learned Q/K distributions violate these assumptions, so √d_k is a sensible heuristic that works well in practice — not a proven optimum. Alternatives have been studied (arXiv:2311.09406↗).
• Interpretability of heads is overstated. It's tempting to say "this head does coreference, that head does syntax." Sometimes true, often not — many heads are diffuse, redundant, or do nothing identifiable, and a large fraction can be pruned with little loss. Attention weights are not a faithful explanation of model behavior; that's an active debate.
• Positional encoding is unsolved-feeling. Sinusoids were an elegant first answer, but the field has churned through learned, relative, ALiBi, and RoPE encodings — and length-extrapolation (using more context than trained on) remains finicky. See [[positional-encoding]].
• Post-norm vs pre-norm. The original paper's post-norm is harder to train deep; nearly all modern LLMs switched to pre-norm (plus RMSNorm). This is a real architectural correction, not a cosmetic one — worth knowing the paper's exact form differs from what you'll see in production code.
• "Attention is all you need" was almost too literal. The FFN sub-layer does enormous heavy lifting (most parameters, most knowledge). Recent work argues the combination of attention + wide FFN matters; attention alone is a weak learner. The title is a great slogan, an incomplete account of where the capability lives.
• It may not be the final architecture. [[state-space-models]] (Mamba), [[mixture-of-experts]], and hybrid designs ([[deepseek-architecture]]) all chip at the pure-transformer baseline. The transformer dominates 2026, but "is attention truly all you need at the limit?" is genuinely open.

How it connects to OpenAlice

• Alice's reasoning core runs on transformer LLMs. Alice's default provider is a transformer-based model (GPT-5.x family on mvp; see provider config). Every prompt Alice composes is processed by exactly the masked-self-attention + FFN decoder stack described above — the context window limit you hit in practice is the O(n²) attention cost made concrete. When a long conversation or document blows the budget, that's this page's central tradeoff biting in production.
• Memory & retrieval exist to dodge the quadratic wall. Because you cannot feed unbounded history through O(n²) attention, OpenAlice's [[agent-memory-systems]], [[mempalace]], and [[graphrag]] retrieve only the relevant slice of context to put in the prompt. Attention's cost profile is the direct economic reason those systems exist — they keep n small while preserving recall.
• Embeddings used across the ecosystem are transformer outputs. Atlas's semantic search (pgvector) and concept-level code/knowledge lookup rely on [[embeddings]] produced by transformer encoders — the same attention machinery, used to encode rather than generate.
• Multi-model orchestration is "attention at the agent level." OpenAlice's [[model-routing]], [[fusion-and-llm-councils]], and [[mixture-of-agents]] route a query to the right model(s) and blend their outputs — conceptually a coarse-grained echo of attention's content-based routing, one layer up the stack.
• Build-from-scratch tracks. For the OpenAlice Academy ramp, this page is the conceptual prerequisite for [[microgpt-build-an-llm-from-scratch]], [[llm-from-scratch]], and the math in [[math-for-ml-foundations]] — implement scaled dot-product attention by hand once and the rest of the LLM stack opens up.

Sources

1. Vaswani et al., "Attention Is All You Need", arXiv:1706.03762 — the original paper. abstract↗ · full HTML↗
2. Jay Alammar, "The Illustrated Transformer" — canonical visual intuition. https://jalammar.github.io/illustrated-transformer/
3. Sebastian Raschka, "Understanding and Coding Self-Attention from Scratch" — step-by-step Q/K/V mechanics. https://sebastianraschka.com/blog/2023/self-attention-from-scratch.html
4. Dao et al., "FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness", arXiv:2205.14135 — the IO-aware speedup. https://arxiv.org/abs/2205.14135
5. Keles et al., "On the Computational Complexity of Self-Attention", arXiv:2209.04881 — quadratic lower bound. https://arxiv.org/abs/2209.04881