Atlas
kb://library/linear-attention-and-deltanet2026-06-16

Linear Attention & DeltaNet: The Softmax Alternative Shipping in Frontier Models

architecturelinear-attentiondeltanetgated-deltanetglasequence-modelstransformer-alternativelong-contextfast-weightsfrontier-models

Linear Attention & DeltaNet

TL;DR. Softmax [[attention-and-transformers|attention]] is brilliant but quadratic: every token attends to every other token, so cost grows with N² and inference carries a KV-cache that grows without bound. Linear attention drops the softmax so the same computation collapses into a recurrence — a fixed-size matrix "memory" S that you update one token at a time, giving O(N) training and O(1)-per-token inference with constant memory. The price is a weaker memory: a plain linear-attention layer only ever adds to S, so it smears associations together and is bad at exact recall. DeltaNet fixes the update rule — borrowing the 1960s delta rule (Widrow–Hoff) to erase the old value for a key before writing the new one — and Gated DeltaNet adds a learned forget gate so the model can also decay stale memory globally. The result decouples two operations softmax never separated: erase/decay (gating) vs. targeted write (delta rule). This is no longer academic — as of 2025–2026 it ships in frontier models: Qwen3-Next / Qwen3.5, Kimi Linear, and MiniMax/Ling all stack mostly Gated-DeltaNet-style layers with a few full-attention layers sprinkled in. The honest catch: pure linear attention still loses to softmax on hard content-based retrieval, which is exactly why every shipping model keeps it a hybrid, not a replacement.

What it is (intuition first)

Start from one fact about [[attention-and-transformers|self-attention]]: to produce the output for token t, softmax attention compares the query q_t against the keys of all previous tokens, normalizes those scores with a softmax, and takes a weighted sum of their values. That comparison-against-everything is what makes attention so good — and what makes it cost O(N²) and force you to keep every past key/value around (the KV-cache that bloats long-context inference; see [[long-context]]).

Now ask a slightly heretical question: what if we didn't need to keep all the past keys and values around?

Here's the trick. Softmax attention is:

out_t = Σ_{i≤t}  softmax(q_t · k_i) · v_i

The softmax is what couples everything — you can't simplify it because the normalizer depends on all the scores. But if you remove the softmax and instead apply a feature map φ(·) to the queries and keys, the score becomes a plain dot product φ(q_t) · φ(k_i), and a beautiful thing happens: by associativity of matrix multiplication you can re-group the sum so the past collapses into a single running quantity:

out_t = φ(q_t) · Σ_{i≤t} φ(k_i) v_iᵀ   =   φ(q_t) · S_t

That S_t = Σ_{i≤t} φ(k_i) v_iᵀ is a fixed-size matrix (d×d, where d is the head dimension) — it does not grow with sequence length. So you can compute it recurrently:

S_t = S_{t-1} + φ(k_t) v_tᵀ        # update memory
out_t = φ(q_t) · S_t               # read memory

This is the central insight of Katharopoulos et al. (2020), "Transformers are RNNs": a linear-attention transformer is an RNN with a matrix-valued hidden state S. Training can still be done in parallel (it's a sum), but generation becomes a constant-time-per-token recurrence with a constant-size state — the paper reported up to ~4000× speedups on long autoregressive generation.

The intuition for S that pays off later: think of it as an associative memory or a tiny "fast-weight" matrix. Writing the pair (key k, value v) means S += k vᵀ; reading with query q means q · S, which returns a blend of the values whose keys point in the same direction as q. It's a key→value lookup table baked into one matrix.

Where it breaks: that lookup table only knows how to add. If two different keys aren't orthogonal, their entries interfere; if you write to the same key twice, the old value never leaves — it just gets a new one piled on top. Memory smears. This is precisely why plain linear attention is famously bad at exact retrieval ("what was the password I told you 3000 tokens ago?"). Everything below is the field's answer to "how do we get a better update rule for `S` without giving up the cheap recurrence?"

Why it matters

Three reasons this is one of the most active frontier-architecture stories of 2025–2026:

  1. The KV-cache is the long-context tax, and linear attention doesn't pay it. Softmax inference memory grows linearly with context (every token's K and V are retained); linear-attention inference memory is constant — one fixed-size S per head. At 100K–1M token contexts that is the difference between feasible and not. This is the same motivation behind [[state-space-models|state-space models (Mamba)]], and the two families have largely converged (see below).
  1. It composes with, rather than replaces, the rest of the stack. [[flash-attention|FlashAttention]] made softmax attention IO-efficient but stayed O(N²) in FLOPs. Linear attention attacks the algorithmic complexity instead — and the GLA paper's FlashLinearAttention kernel shows you can get a chunk-parallel, hardware-efficient implementation that beats FlashAttention-2 even at 1K tokens. The two are complementary kernel-engineering stories.
  1. Frontier models actually ship it now. This is the part that moved linear attention from "interesting alternative" to "production reality." Qwen3-Next (2025) was the first near-flagship to go hybrid — three Gated DeltaNet layers for every one full-attention layer (3:1) — and Qwen promoted that design into its main Qwen3.5 flagship line. Kimi Linear ships a refinement (Kimi Delta Attention), and the same 3:1 hybrid pattern appears in Ling 2.5 and related models. When multiple independent frontier labs converge on the same recipe in the same year, it's a real architectural shift, not a fad.

How it works (real mechanics)

1. Plain linear attention (the additive baseline)

Per head, with feature map φ (early work used elu(x)+1; many modern variants use identity or a small MLP):

S_t   = S_{t-1} + φ(k_t) v_tᵀ          # S is d_k × d_v, the matrix memory
out_t = φ(q_t)ᵀ S_t  /  (φ(q_t)ᵀ z_t)  # z_t = z_{t-1} + φ(k_t) is the normalizer

O(N·d²) time, O(d²) state. Good. But the only verb is add — no forgetting, no overwrite. Memory capacity is finite (the rank/conditioning of S), so it saturates and smears.

2. Gated Linear Attention — GLA (add a forget gate)

Yang et al. (2023) introduce a data-dependent decay gate G_t so the state can decay old memory before adding new:

S_t   = G_t ⊙ S_{t-1} + φ(k_t) v_tᵀ
out_t = q_tᵀ S_t

G_t ∈ (0,1) is computed from the input (it's selective, like Mamba's gating) and multiplies the old state down. This is the linear-attention analogue of an LSTM forget gate: the model learns when to let memory fade. In GLA the gate is typically a per-feature (channel-wise) vector, which is more expressive than a single scalar.

The non-obvious contribution of the GLA paper is `FlashLinearAttention`: a chunk-parallel training algorithm. You split the sequence into chunks; compute an exact intra-chunk attention quadratically within each small chunk (cheap), carry the running state S across chunks via the recurrence, and fuse it all into a tiled GPU kernel that trades memory movement for parallelism. This is what makes gated linear attention trainable at scale and faster than FlashAttention-2 even on short sequences — the recurrence alone would be too sequential for a GPU.

3. DeltaNet — the delta rule (fix the write, not just the decay)

GLA can forget globally (scale the whole state down) but it still writes additively — it can't surgically replace the value for one specific key. Yang et al. (2024), DeltaNet, borrows the delta rule (a.k.a. Widrow–Hoff / LMS, the 1960s online least-squares learning rule). The idea: before writing v_t for key k_t, first retrieve what `S` currently predicts for `k_t`, compute the error, and correct it.

The update, with a learned write-strength β_t ∈ (0,1):

S_t = S_{t-1} − β_t (S_{t-1} k_t − v_t) k_tᵀ

Read that middle term: S_{t-1} k_t is the current prediction for key k_t; v_t is the target; their difference is the error. The update nudges S to reduce that error — exactly gradient descent on ½‖S k_t − v_t‖². Expand it and the "erase-then-write" structure becomes visible:

S_t = S_{t-1}(I − β_t k_t k_tᵀ)  +  β_t v_t k_tᵀ
      └─── erase old value at k_t ───┘  └── write new value ──┘

The first factor (I − β_t k_t k_tᵀ) is a Householder-like projection that removes the component of S aligned with k_t (scaled by β_t) — i.e. it clears the slot for that key. The second term writes the fresh association. When β_t → 1 it's a full overwrite; smaller β_t blends old and new. This is the thing plain linear attention and GLA cannot do: targeted, key-specific replacement that minimizes retrieval error instead of just piling on. It's why DeltaNet beats Mamba and GLA on associative-recall and in-context retrieval.

Parallelizing it was the hard part. That (I − β_t k_t k_tᵀ) factor makes the recurrence multiplicative and sequence-dependent — naively it's sequential. The DeltaNet paper's trick is to recognize that a chunk's worth of these Householder updates can be folded into a WY representation (the same compact form used to accumulate products of Householder reflections in numerical linear algebra). That makes the chunk update one matmul-friendly block, so DeltaNet trains chunk-parallel like GLA. They scaled a 1.3B model on 100B tokens and beat Mamba/GLA baselines; hybrids of DeltaNet + sliding-window/global attention beat plain Transformers.

4. Gated DeltaNet — decouple erase-vs-write (the part shipping in frontier models)

Yang, Kautz & Hatamizadeh (2024) make the obvious-in-hindsight move: combine GLA's gate with DeltaNet's delta rule. Their one-line thesis: "gating enables rapid memory erasure while the delta rule facilitates targeted updates." Two genuinely different operations:

  • Gate `α_t` → global decay: "forget everything a bit" (handles topic shifts, sequence boundaries, stale context).
  • Delta rule `β_t` → local, key-specific write: "replace the value at this key precisely" (handles updates to a specific fact).

The combined update (schematically):

S_t = α_t S_{t-1} (I − β_t k_t k_tᵀ)  +  β_t v_t k_tᵀ
      └─ gated decay ─┘└─ delta erase ─┘     └─ write ─┘

Now the model has two decoupled knobs: how fast to forget everything (α) and how hard to overwrite one thing (β). Softmax attention never separates these — it just re-attends to the whole past every step. Gated DeltaNet "consistently surpasses Mamba2 and DeltaNet" across language modeling, retrieval, length extrapolation, and long-context.

5. The state-space connection (it's the same family)

Gated DeltaNet is explicitly framed as "improving Mamba2 with the delta rule." [[state-space-models|Mamba-2]]'s State-Space Duality showed selective SSMs and gated linear attention are two views of the same S_t = (decay) ⊙ S_{t-1} + (input) recurrence. Gated DeltaNet keeps Mamba-style gating but swaps Mamba's additive state-space write for DeltaNet's delta-rule write. So "linear attention," "gated linear attention," and "modern SSMs" are not rival families — they're points on one design space of linear-recurrent sequence mixers with a matrix memory. The interesting axis is the write rule: additive (linear attn / Mamba) vs. error-correcting (DeltaNet).

Minimal pseudocode (recurrent reference form)

# One linear-attention head, autoregressive. Recurrent (inference) form.
# variant ∈ {"linear", "gla", "deltanet", "gated_deltanet"}
S = zeros(d_k, d_v)                      # the entire memory: one matrix
for t in range(N):
    q, k, v = W_q@x[t], W_k@x[t], W_v@x[t]
    q, k = phi(q), phi(k)                # feature map (identity in many modern variants)

    if variant == "linear":
        S = S + outer(k, v)                                  # add-only
    elif variant == "gla":
        g = sigmoid(W_g @ x[t])                              # per-channel forget gate (0,1)
        S = g[:, None] * S + outer(k, v)                     # decay, then add
    elif variant == "deltanet":
        beta = sigmoid(W_b @ x[t])                           # write strength (0,1)
        pred = S.T @ k                                       # what S currently recalls for k
        S = S + beta * outer(k, (v - pred))                 # erase-then-write (delta rule)
    elif variant == "gated_deltanet":
        alpha = sigmoid(W_a @ x[t])                          # global decay gate
        beta  = sigmoid(W_b @ x[t])                          # local write strength
        pred  = S.T @ k
        S = alpha * S + beta * outer(k, (v - pred))         # decouple decay vs targeted write

    out[t] = S.T @ q                     # read: O(d²), no growing cache
# Training uses the CHUNK-PARALLEL form (FlashLinearAttention / WY-Householder),
# NOT this sequential loop — the loop is the semantic reference, not the fast path.

What frontier hybrids actually do (2025–2026)

  • Qwen3-Next / Qwen3.5 — 3:1 layout: three Gated DeltaNet layers per one Gated (full softmax) Attention layer. Qwen3-Next was the first near-flagship to ship this; Qwen3.5 promoted it to the flagship line.
  • Kimi Linear (Kimi Delta Attention, KDA) — refines Gated DeltaNet by replacing Qwen3-Next's scalar per-head gate with channel-wise (per-feature) gating, for finer control of the decay. Same 3:1 hybrid skeleton.
  • MiniMax / Ling 2.5 — same hybrid philosophy (mostly-linear, a few full-attention layers).
  • Why keep *any* softmax layers? Because DeltaNet is still less exact at content-based retrieval than full attention. The full-attention layers are the "precise lookup" backstop; the linear layers carry the long-context efficiency. Every shipping design is a hybrid for exactly this reason — see the same hybrid-not-replacement conclusion in [[state-space-models]].

Key ideas & tradeoffs

  • The verb matters more than the gate. Plain linear attention and Mamba add; DeltaNet corrects error. Error-correcting writes (delta rule) are the single biggest jump in retrieval quality among linear variants, because they stop the memory from smearing non-orthogonal keys together.
  • Decoupling erase from write is the Gated-DeltaNet thesis. Global decay (gate α) and targeted overwrite (delta β) are different needs; giving the model both, separately, beats giving it either alone.
  • Constant inference memory is the headline win. No KV-cache growth → cheap, flat-memory long-context decoding. This is the whole commercial reason frontier labs care.
  • Chunk-parallelism is non-negotiable. The recurrence is sequential; without FlashLinearAttention (GLA) / WY-Householder chunking (DeltaNet) these models would be untrainable at scale. The clever kernel is the contribution as much as the math.
  • Hybrid > pure. Nobody ships pure linear attention. The winning recipe is ~75–90% linear layers + a few full-attention layers. You buy efficiency without sacrificing the exact-recall that softmax still does best.
  • Fixed state = bounded memory. A d×d matrix can only hold so many cleanly-separable associations. Gating and the delta rule manage that budget far better than addition, but they don't make it infinite — the capacity ceiling is structural.

Honest caveats & open questions

  • Pure linear attention still loses on hard retrieval — full stop. The entire hybrid trend is an admission that a fixed-size state cannot match softmax's "look at literally everything" on needle-in-a-haystack and exact-copy tasks. The Mamba "free lunch?" line of work (see [[state-space-models]]) showed these limits formally. Linear attention is a complement, not a Transformer killer — be suspicious of any claim otherwise.
  • The architecture is moving fast and not settled. Scalar vs. channel-wise gating (Qwen3-Next vs. Kimi), exact hybrid ratio (3:1 is common but not derived from first principles), where to place the full-attention layers, which feature map φ — these are empirical choices, currently in flux. As Raschka's own framing puts it: nobody fully agrees on attention anymore. Treat specific ratios as the current meta, not settled law.
  • Apples-to-apples scaling at true flagship size is still thin. Most head-to-head numbers are ≤~3B dense or modest-MoE. The hybrids are now in flagship-class models, but rigorous, controlled "same data, same budget, pure-softmax vs hybrid" ablations at frontier scale remain limited in the open literature.
  • Kernel maturity lags softmax. FlashLinearAttention and DeltaNet's chunked kernels are good but younger and less battle-tested than the [[flash-attention|FlashAttention]] ecosystem. Numerical stability of the (I − βkkᵀ)/WY path, and behavior under low precision, are areas where the tooling is still hardening.
  • "Delta rule" is a 1960s idea wearing new clothes. Worth stating plainly so nobody over-mystifies it: it's Widrow–Hoff/LMS error-correction reused as a state-update rule. The novelty is the parallelization (WY-Householder) and the combination with gating, not the learning rule itself.
  • Interpretability is open. We understand softmax attention heads reasonably well (induction heads, etc.; see [[mechanistic-interpretability]]). What the fast-weight matrix S actually stores, and how gated-delta dynamics implement in-context learning, is much less mapped.

How it connects to OpenAlice

  • Long-context cost is Alice's real bottleneck. Alice runs long, persistent conversations with growing memory; the softmax KV-cache is exactly the tax that hurts at scale. Whatever provider model Alice routes to, its attention architecture (increasingly a Gated-DeltaNet hybrid in the Qwen/Kimi line) directly sets Alice's long-context latency and cost envelope — relevant to [[model-routing]] and [[long-context]] decisions.
  • Fast-weights ≈ working memory, and that's an architectural mirror. The S matrix is literally an in-weights associative memory updated per token with erase-then-write semantics. That's a clean low-level analogue to OpenAlice's application-level memory systems (file-first memory, the org KB) — the delta rule's "retrieve, compute error, overwrite the specific slot" is the same discipline a good memory layer needs: don't just append, update the right entry.
  • The hybrid lesson generalizes to system design. "Cheap linear layers for the bulk + a few exact-attention layers for precision" rhymes with how OpenAlice builds: cheap/fast retrieval (semantic search over Atlas, embeddings) for the common path, exact lookup (grep, direct file reads, the call-graph) for when precision matters. Same precision/efficiency tradeoff, one layer up.
  • Sibling reading. This page sits next to [[attention-and-transformers]] (the softmax baseline this replaces), [[flash-attention]] (making softmax IO-efficient — the orthogonal kernel story), [[state-space-models]] (the converged cousin family; Gated DeltaNet is "Mamba2 + delta rule"), [[positional-encoding]], and [[long-context]] (the problem all of this is trying to solve cheaply).

Sources

  1. Katharopoulos, Vyas, Pappas, Fleuret — Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention (2020). https://arxiv.org/abs/2006.16236 — the linear-attention foundation; S_t = S_{t-1} + φ(k_t)v_tᵀ.
  2. Yang, Wang, Shen, Panda, Kim — Gated Linear Attention Transformers with Hardware-Efficient Training (GLA, 2023). https://arxiv.org/abs/2312.06635 — data-dependent decay gate + FlashLinearAttention chunk-parallel kernel.
  3. Yang, Wang, Zhang, Shen, Kim — Parallelizing Linear Transformers with the Delta Rule over Sequence Length (DeltaNet, 2024). https://arxiv.org/abs/2406.06484 — delta-rule state update + WY/Householder chunk-parallel training.
  4. Yang, Kautz, Hatamizadeh — Gated Delta Networks: Improving Mamba2 with Delta Rule (Gated DeltaNet, 2024). https://arxiv.org/abs/2412.06464 — decouples gated decay from delta-rule write.
  5. Songlin Yang — DeltaNet / delta-rule blog. https://sustcsonglin.github.io/blog/2024/deltanet-1/ — derivation of the erase-then-write form and its link to Widrow–Hoff.
  6. Sebastian Raschka — A Visual Guide to Attention Variants in Modern LLMs. https://magazine.sebastianraschka.com/p/visual-attention-variants — Qwen3-Next / Qwen3.5 / Kimi Linear hybrid layouts, scalar vs. channel-wise gating, 3:1 ratio.