Positional Encoding (RoPE, ALiBi)

For NAO + anyone who wants to actually understand how a transformer knows word order. This is the missing-but-essential plumbing of every LLM. The attention mechanism (see [[attention-and-transformers]]) is, at its core, order-blind — it sees a bag of tokens, not a sequence. Positional encoding is the trick that puts the order back in. We start with the intuition a beginner needs, then go all the way down to the real rotation matrices and the context-extension hacks (RoPE scaling, YaRN) that let a model trained on 4K tokens read 128K.

The one-sentence idea

Self-attention has no built-in notion of "first vs. second vs. third token", so we deliberately inject position information — either by adding a position signal to the embeddings (sinusoidal, learned) or by baking position directly into the attention score computation (RoPE rotates the vectors; ALiBi adds a distance penalty).

What it is (intuition first)

Imagine you hand someone the words {ate, the, cat, mouse, the} in a shuffled pile and ask "who ate whom?" They can't tell — the words alone don't say. "The cat ate the mouse" and "The mouse ate the cat" use the exact same five tokens. The only difference is order.

A transformer's attention layer has precisely this problem. Attention computes, for every pair of tokens, a score q·k measuring how relevant token j is to token i, then mixes their values accordingly. But that computation is permutation-equivariant: shuffle the input tokens and the outputs just shuffle the same way — the content of each token's output is unchanged. The math literally cannot see that "cat" came before "ate". (Contrast this with an RNN, which reads left-to-right and therefore encodes order in its very control flow. Transformers traded that sequentiality for parallelism — and lost order as a side effect.)

So we add position back in on purpose. There are two philosophies:

1. Add a position signal to the token vectors before attention sees them. Each position gets a distinct fingerprint vector; "cat at position 2" ends up with a different vector than "cat at position 5". This is the absolute family: sinusoidal (Vaswani et al. 2017↗) and learned position embeddings (GPT/BERT).
2. Modify the attention scores themselves as a function of how far apart two tokens are. The model never sees an absolute position number; it only ever feels relative distance. This is the relative family, and it's where the two stars of this article live: RoPE (Su et al. 2021↗) and ALiBi (Press et al. 2021↗).

If you only remember one thing: modern LLMs (LLaMA, Mistral, Qwen, GPT-NeoX, DeepSeek) almost all use RoPE. ALiBi is the elegant minimalist alternative. Absolute sinusoidal/learned encodings are the historically-important originals that the field has largely moved past for big autoregressive models.

Why it matters

• Without it, the model is grammatically blind. Word order carries enormous meaning. No position info → the model cannot distinguish subject from object, cannot do counting/ordering tasks, cannot handle syntax. Position encoding is not optional polish; it's load-bearing.
• It's the lever for context length. The single hottest applied-LLM topic of 2023–2024 — "how do I make my 4K-context model read 128K tokens?" — is almost entirely a positional-encoding problem. The way you encode position decides whether a model extrapolates gracefully to longer inputs or collapses into garbage. RoPE's frequency structure is exactly what techniques like Position Interpolation and YaRN manipulate.
• It's cheap but consequential. RoPE adds only ~1–3% compute overhead (EleutherAI blog↗) yet changed the default architecture of the entire field. Small mechanism, huge leverage.

How it works (the real mechanics)

We'll build up in the historical order, because each method is a reaction to the previous one's weaknesses.

1. Absolute sinusoidal (the original, 2017)

The Transformer paper defines, for position pos and dimension index i in a model of width d:

PE(pos, 2i)   = sin( pos / 10000^(2i/d) )
PE(pos, 2i+1) = cos( pos / 10000^(2i/d) )

You compute this fixed d-dimensional vector for each position and add it to the token embedding before layer 1. Each dimension-pair is a sinusoid; pair i=0 oscillates fast (wavelength 2π), and as i grows the wavelength stretches geometrically out to 10000·2π. So a position is encoded as a bank of clocks ticking at many frequencies — like binary counting, but with smooth sines instead of crisp bits.

Why sinusoids specifically? Vaswani et al. hypothesized it would let the model attend by relative position, "since for any fixed offset k, PE(pos+k) can be represented as a linear function of PE(pos)." (A shift in position is a rotation of each sine/cosine pair — and rotation is linear.) They also report trying learned positional embeddings and getting nearly identical results, choosing sinusoids partly because they might extrapolate to longer sequences.

Learned absolute embeddings (used by BERT, original GPT) drop the formula entirely: position 0…N−1 each gets its own trainable vector, looked up and added just like a token embedding. Maximum flexibility, zero inductive bias — and a hard wall at the max training length N. Position N+1 was never trained and has no vector. This is the reason GPT-2 cannot natively read beyond its 1024-token window. (apxml comparison↗)

Honest note: the "sinusoids extrapolate" hope turned out weak in practice. Press et al. showed sinusoidal models also degrade sharply beyond train length. Extrapolation needed a different idea — which is ALiBi and (with scaling) RoPE.

2. Relative position embeddings (the bridge idea)

Shaw et al. (2018) and Transformer-XL introduced a key reframing: what attention should care about is not "token at absolute position 5" but "token 3 steps to my left." So they injected learnable biases/keys indexed by the relative offset i − j into the attention score. This is conceptually right (translation-invariant — a phrase means the same thing at the start or middle of a doc) but the early implementations were slow and incompatible with efficient attention kernels. RoPE and ALiBi are the two clean, fast ways to get the relative property.

3. RoPE — Rotary Position Embedding (the modern default)

The idea in one line: instead of adding a position vector, rotate each query and key vector by an angle proportional to its absolute position. Then when attention takes a dot product q·k, the absolute rotations cancel and what survives depends only on the relative angle — i.e. on m − n, the distance between positions m and n.

The 2D building block. Take two consecutive components of a query vector, (q₁, q₂), and treat them as a point in a plane. To encode position m, rotate that point by angle m·θ:

[ q₁' ]   [ cos(mθ)  −sin(mθ) ] [ q₁ ]
[ q₂' ] = [ sin(mθ)   cos(mθ) ] [ q₂ ]

Do the same to the key at position n with angle n·θ. The dot product of two rotated 2D vectors depends only on the angle between them, (m − n)·θ — a standard fact of rotations. That's the whole magic: absolute in, relative out.

Scaling to full dimension. A real head has d dimensions, not 2. RoPE splits the vector into d/2 consecutive pairs and gives each pair its own rotation frequency, exactly mirroring the sinusoidal frequency ladder:

θ_i = 10000^(−2i/d) ,   i = 0 … d/2 − 1

So pair 0 spins fast (captures fine, local position differences) and the last pair spins glacially slowly (captures long-range, coarse position). The full operation is a block-diagonal rotation matrix R_m (one 2×2 rotation block per pair) applied as R_m · (W_q x). Because each block is its own rotation, the relative-only property holds per pair, and the full dot product becomes a sum of relative-position terms (EleutherAI↗).

The complex-number view (same thing, slicker). Pack each pair as a complex number q₁ + i·q₂. Then rotating by m·θ is just multiplying by e^{i·m·θ}:

f(q, m) = q · e^{i·m·θ}

and the attention inner product ⟨f(q,m), f(k,n)⟩ carries the factor e^{i·(m−n)·θ} — manifestly a function of m − n only.

Efficient implementation (what's actually in the code). You never build the big matrix. You precompute cos and sin tables and use a rotate_half trick:

rotate_half(x) = (−x₂, x₁)   # second half negated, swapped with first
RoPE(x, m)     = x ⊙ cos(mθ) + rotate_half(x) ⊙ sin(mθ)

This is two elementwise multiplies and an add — hence the ~1–3% overhead. RoPE is applied to q and k at every layer, not added once at the input (a crucial difference from sinusoidal). Values v are not rotated.

Key properties RoPE buys you:

• Relative position baked into attention, for free, with no extra parameters.
• Long-term decay: because high-frequency pairs dephase quickly, the inner product between far-apart tokens tends to shrink — a sensible prior that distant tokens matter less (Su et al.↗).
• Compatible with linear attention — RoPE multiplies q/k, so it slots into kernelized/linear-attention formulations, unlike additive relative biases.
• It plays nicely with [[flash-attention]] (it's applied to q,k before the kernel).

4. ALiBi — Attention with Linear Biases (the minimalist)

The idea in one line: don't touch the embeddings or rotate anything. Just, right before softmax, subtract a penalty from each attention score proportional to the distance between the two tokens.

The attention score for query i attending to key j becomes:

score(i, j) = (q_i · k_j) − m · |i − j|      (causal: j ≤ i, so − m·(i − j))

m is a fixed slope — a per-head constant, not learned. Nearby tokens get almost no penalty; tokens far in the past get heavily down-weighted, so attention naturally biases toward recency.

Per-head slopes. Different heads should look at different ranges, so ALiBi assigns slopes as a geometric sequence. For n heads, slopes start at 2^(−8/n) and ratio 2^(−8/n) — e.g. for 8 heads: 1/2, 1/4, 1/8, … , 1/256. Steep-slope heads attend locally; shallow-slope heads see far. No parameters added; the bias matrix is static and precomputable.

The headline result. ALiBi was designed for extrapolation. Press et al. trained a 1.3B model on length-1024 sequences and ran inference at length 2048, matching the perplexity of a sinusoidal model trained directly at 2048 — while training 11% faster and using 11% less memory. Because the bias is purely a function of distance with no max-length parameter, longer sequences "just work" (up to the recency-bias ceiling).

Key ideas & tradeoffs

The central tradeoffs:

• Inductive bias vs. flexibility. Learned embeddings have none (learn anything, generalize to nothing past N). Sinusoidal/RoPE/ALiBi bake in a relative, distance-decaying prior — less flexible, but it's the right prior for language, and it's what enables length generalization.
• RoPE vs. ALiBi. RoPE is more expressive (full rotational encoding across all frequency bands, no inherent recency-only bias) and dominates frontier models. ALiBi is simpler, extrapolates more cleanly out of the box, but its hard recency bias can hurt tasks needing genuine long-range retrieval (it structurally down-weights distant tokens). RoPE + a scaling method usually wins on long-context retrieval; ALiBi wins on simplicity.
• Context extension is a RoPE-frequency game. This deserves its own section.

Extending context: PI, NTK-aware, YaRN

The applied magic of 2023–2024. A RoPE model trained at length L breaks past L because the slow-frequency rotation pairs hit angles never seen in training. Three families of fixes, all manipulating RoPE's frequencies:

• Position Interpolation (PI) — Chen et al., Meta 2023↗. Brutally simple: to go from L to s·L, divide every position by `s` so the new max maps back into the trained range. "Squeeze" 32K positions into the 4K the model knows. Extends LLaMA to 32K with light fine-tuning. Downside: uniform squeezing also crushes the high-frequency dims that encoded fine local order, hurting short-range precision.
• NTK-aware interpolation (community / "NTK-by-parts"). Don't stretch all frequencies equally. High-frequency (short-wavelength) dims already generalize — leave them alone; only interpolate the low-frequency (long-wavelength) dims that actually exceed the trained range. Often works with no fine-tuning. Derived from the insight that MLPs struggle to learn high-frequency content, so don't disturb it.
• YaRN — Peng et al. 2023↗. The current go-to. Combines NTK-by-parts (per-band selective interpolation: untouched high-freq, interpolated low-freq, blended in between) with an attention temperature scaling factor on the logits that compensates for the entropy shift at long context. Extends context with <0.1% of the original pretraining tokens and far fewer steps than PI. Dynamic-YaRN/Dynamic-NTK adjust the scale per-forward-pass, giving >2× extension without fine-tuning and graceful behavior across input lengths.

The progression PI → NTK-aware → YaRN is a textbook case of "uniform hack → frequency-aware hack → frequency-aware + calibration." All three exist only because RoPE exposes a clean, manipulable frequency structure — something ALiBi and learned embeddings don't offer.

Honest caveats & open questions

• "Extrapolation" is slippery. A model not crashing on long inputs ≠ using them well. ALiBi extrapolates perplexity but its recency bias can mean it isn't truly reading far-back tokens. RoPE+YaRN extends the window but long-context retrieval (needle-in-a-haystack) often still degrades. Don't conflate "accepts N tokens" with "reasons over N tokens."
• RoPE's long-term decay is a double-edged prior. Great as a default, but it can bias against legitimately important distant tokens; part of why long-context fine-tuning and scaling tricks are needed at all.
• The base `10000` is a tunable hyperparameter, not a law. Many long-context models simply increase the RoPE base (e.g. to 500K or 1M) so frequencies are slow enough to cover the target length. It's effective and widely used, but somewhat empirical — the theory of why a given base works is still underdeveloped.
• No consensus on "best." RoPE dominates by adoption, not by a settled proof of optimality. There's active research on alternatives (no-positional-encoding decoders that implicitly learn position via the causal mask; new schemes for multimodal/2D position). The field is not closed.
• 2D/structured positions are harder. Sinusoidal/RoPE/ALiBi are 1D-sequence ideas. Images, video, tables, and graphs need 2D/structured position encodings (e.g. 2D-RoPE variants) — an open and messy area.
• Sources note: the arXiv abstract pages for the RoPE, ALiBi, and YaRN papers don't expose the in-paper equations; the precise formulas above are cross-checked against the EleutherAI RoPE blog and apxml/secondary references, and against the abstracts. For a derivation you trust line-by-line, read the PDFs directly.

How it connects to OpenAlice

Be precise and honest: OpenAlice (Alice) is a consumer of these mechanisms via the LLM providers it routes to (Codex/GPT, and any RoPE-based open models in the [[model-routing]] cost ladder), not an implementor of custom positional encodings. The honest, concrete ties:

• Context-length budgeting is a positional-encoding problem in disguise. Alice's prompt-composer and memory systems ([[mempalace]], [[agent-memory-systems]]) constantly decide what fits in the window. Knowing that a model's effective long-context quality (not just its advertised window) is gated by RoPE-scaling artifacts is exactly the kind of caveat that should inform "do we trust this model with a 100K-token transcript, or do we retrieve+compress first?"
• Model routing. When the [[model-routing]] cost ladder swaps in different open models, their positional scheme (RoPE base, YaRN vs. native long-context) is part of why their long-context behavior differs — useful when choosing which model handles long documents vs. short chats.
• If/when OpenAlice trains or fine-tunes anything in-house (the from-scratch educational track: [[microgpt-build-an-llm-from-scratch]], [[llm-from-scratch]]), RoPE is the position encoding you'd implement — this article is the spec for that ~15 lines of code.

No deeper claim than that — Alice doesn't currently ship custom position-encoding code, and pretending otherwise would be dishonest.

See also

• [[attention-and-transformers]] — the order-blind mechanism this article fixes.
• [[embeddings]] / [[tokenization]] — what gets positioned in the first place.
• [[flash-attention]] — the kernel RoPE is applied around.
• [[deepseek-architecture]] — a modern RoPE-using (and decoupled-RoPE-innovating) stack.
• [[state-space-models]] — an alternative that encodes order recurrently, sidestepping explicit position encoding.
• [[microgpt-build-an-llm-from-scratch]] / [[llm-from-scratch]] — where you'd implement RoPE yourself.

References

1. Vaswani et al., Attention Is All You Need (2017) — https://arxiv.org/abs/1706.03762 (sinusoidal + learned PE)
2. Su et al., RoFormer: Enhanced Transformer with Rotary Position Embedding (2021) — https://arxiv.org/abs/2104.09864
3. Press et al., Train Short, Test Long: Attention with Linear Biases (ALiBi) (2021) — https://arxiv.org/abs/2108.12409
4. EleutherAI, Rotary Embeddings: A Relative Revolution — https://blog.eleuther.ai/rotary-embeddings/
5. Peng et al., YaRN: Efficient Context Window Extension (2023) — https://arxiv.org/abs/2309.00071
6. Chen et al., Extending Context Window via Position Interpolation (2023) — https://arxiv.org/abs/2306.15595