RNNs, LSTMs & GRUs

TL;DR. A recurrent neural network (RNN) reads a sequence one token at a time, carrying a small fixed-size "memory" vector (the hidden state) forward from step to step. It's elegant and constant-memory, but training it by backpropagation through time (BPTT) multiplies many copies of the same weight matrix together — so gradients either shrink to nothing (vanishing) or blow up (exploding) over long sequences, and the network forgets anything more than a handful of steps back. The LSTM (1997) fixed the vanishing half with a gated cell state — an additive "conveyor belt" of memory protected by learned forget/input/output gates — letting gradients flow across hundreds of steps. The GRU (2014) is a leaner two-gate cousin that often matches it. For a decade these were sequence modeling. Then attention (2014) and the Transformer (2017) removed the sequential bottleneck entirely, and recurrence faded from the frontier — only to quietly return, re-engineered, as modern [[state-space-models]] and [[linear-attention-and-deltanet]].

What it is (intuition first)

Imagine reading a sentence aloud while keeping a single running thought in your head. After each word you update that thought a little, then move on. You never re-read; all you carry forward is the updated thought. That running thought is the hidden state h, and the fixed rule you apply to update it after each word is a recurrent neural network.

Formally, an RNN is one small neural network applied over and over, once per time step, feeding its own output back into itself:

h_t = f(x_t, h_{t-1})      # new memory = f(this token, old memory)
y_t = g(h_t)               # optional output at this step

The same function f (the same weights) runs at every step. That weight-sharing is what makes it "recurrent" — and, as we'll see, it's also exactly what makes it hard to train. The network's whole knowledge of the past is squeezed into the fixed-size vector h_{t-1}; there is no looking back at earlier tokens directly. Contrast this with a Transformer, which keeps every past token and re-reads them all via [[attention-and-transformers]]. The RNN's bet is the opposite: compress everything into one small evolving summary.

This is the same intuition the [[makemore]] project teaches hands-on. makemore builds a character-level name generator first as a bigram/MLP, then as an RNN/GRU, so you can feel the difference between "look at a fixed window" and "carry a state forward." And [[state-space-models]] (Mamba) is best understood as a modern re-derivation of this exact RNN picture, with the math chosen so it can also train in parallel.

Why it matters

For roughly 2014–2017, LSTMs and GRUs were the state of the art for almost everything sequential: machine translation, speech recognition, handwriting, language modeling, time series. Google Translate's 2016 neural system was a stack of LSTMs. The encoder–decoder ("seq2seq") pattern — one RNN reads the input into a vector, another RNN generates the output — was born here (Cho et al. 2014).

They matter today for four reasons:

They are the conceptual ancestor of everything. The encoder–decoder structure, the idea of a learned "memory," and — crucially — attention itself were all invented to patch RNN limitations. Bahdanau et al. 2014 added attention precisely because cramming a whole sentence into one fixed RNN vector was a bottleneck. The Transformer (Vaswani et al. 2017) then kept the attention and threw away the recurrence. You cannot understand "Attention Is All You Need" without knowing what attention was replacing.
The vanishing-gradient story is foundational ML literacy. It explains why depth (in time or layers) is hard, why residual/additive connections matter, and why gating recurs everywhere from LSTMs to highway networks to the gates inside Mamba.
Recurrence is back. [[state-space-models]] and [[linear-attention-and-deltanet]] are, mathematically, RNNs with a linear recurrence chosen so they parallelize. The constant-memory inference that makes Mamba attractive is the same property that made RNNs attractive in 1997.
Constant inference memory. Unlike a Transformer's KV-cache that grows with every token, an RNN's state is fixed-size. For very long streams and edge deployment, that's a real advantage — which is why the idea never fully died.

How it works (real mechanics)

1. The vanilla RNN forward pass

At each step, combine the current input with the previous hidden state, squash through a nonlinearity (classically tanh):

h_t = tanh(W_x x_t + W_h h_{t-1} + b)
y_t = W_y h_t + b_y

W_h is the recurrent weight matrix — the same matrix at every step. Hold that thought.

2. BPTT: training by unrolling time

To train, you unroll the network across the sequence — picture T copies of the cell laid out left to right, the hidden state threading between them — and run ordinary backprop over that unrolled graph. This is backpropagation through time (BPTT). The loss at the end must send gradient back through every step to update the shared weights.

3. Why gradients vanish or explode

Here is the heart of it. The gradient of a late loss with respect to an early hidden state requires the chain rule across every intermediate step, which produces a product of Jacobians:

∂h_t/∂h_k = Π_{i=k+1..t}  ∂h_i/∂h_{i-1}

Each factor ∂h_i/∂h_{i-1} involves the recurrent weight matrix W_h (times the derivative of tanh). Multiplying (t − k) of these together is roughly like raising W_h to a power. Pascanu, Mikolov & Bengio (2013) made this precise: the behavior is governed by the largest singular value (spectral radius) of `W_h`:

If that value is < 1, the product shrinks exponentially → vanishing gradients. The network literally cannot learn dependencies more than ~5–10 steps apart; the learning signal from far back arrives as numerical zero.
If it is > 1, the product grows exponentially → exploding gradients. Updates become NaN; training diverges.

The exploding half has a cheap fix that paper popularized: gradient clipping — if the gradient's norm exceeds a threshold, rescale it down to that threshold. Crude, but it works and is still standard practice. The vanishing half is the deep problem, and clipping does nothing for it. That required architecture, not a training trick.

4. The LSTM: a protected, additive memory

Hochreiter & Schmidhuber (1997) introduced Long Short-Term Memory to solve vanishing gradients structurally. The core insight is the constant error carousel (CEC): alongside the hidden state, carry a separate cell state C_t whose default update is additive and linear — C_t ≈ C_{t-1} — so error can flow backward through it without being repeatedly multiplied by a weight matrix. In the absence of interference, gradient through the cell state neither decays nor explodes; it stays constant. The error rides the carousel.

Of course you don't want literally constant memory — you need to write to it, read from it, and clear it. So three multiplicative gates (each a sigmoid producing values in (0,1)) guard the carousel. The modern equations (matching Dive into Deep Learning; σ = sigmoid, ⊙ = elementwise/Hadamard product):

# Gates: each looks at the current input and previous hidden state
I_t = σ( X_t W_xi + H_{t-1} W_hi + b_i )        # input gate   — how much new info to write
F_t = σ( X_t W_xf + H_{t-1} W_hf + b_f )        # forget gate  — how much old memory to keep
O_t = σ( X_t W_xo + H_{t-1} W_ho + b_o )        # output gate  — how much memory to read out

# Candidate memory: the new content we *might* write (tanh, range (-1,1))
C̃_t = tanh( X_t W_xc + H_{t-1} W_hc + b_c )

# THE conveyor belt: forget some old, add some new
C_t = F_t ⊙ C_{t-1}  +  I_t ⊙ C̃_t

# Hidden state / output: a gated, squashed view of the cell
H_t = O_t ⊙ tanh(C_t)

The load-bearing line is C_t = F_t ⊙ C_{t-1} + I_t ⊙ C̃_t. Because the path from C_{t-1} to C_t is addition gated by `F_t` (not a dense matrix multiply through a tanh), the gradient ∂C_t/∂C_{t-1} ≈ F_t. When the forget gate stays near 1, gradients pass across many steps essentially undiminished. That is the constant error carousel, expressed as a derivative. It is the same trick residual connections use in deep Transformers — an additive identity path that gradients can ride. (Empirically the forget gate is the most important component: Greff et al. 2015 ablated eight LSTM variants across ~5,400 runs and found the forget gate and the output activation matter most, while no variant beat the standard LSTM significantly. The original 1997 LSTM actually had no forget gate — it was added by Gers et al. in 2000 and proved essential.)

5. The GRU: same idea, two gates

Cho et al. (2014) introduced the Gated Recurrent Unit, which merges the cell and hidden state into one and uses only two gates. It's lighter (fewer parameters) and often performs comparably (equations cross-checked against d2l):

Z_t = σ( X_t W_xz + H_{t-1} W_hz + b_z )        # update gate — interpolation knob
R_t = σ( X_t W_xr + H_{t-1} W_hr + b_r )        # reset gate  — how much past to ignore when proposing

H̃_t = tanh( X_t W_xh + (R_t ⊙ H_{t-1}) W_hh + b_h )   # candidate state

H_t = Z_t ⊙ H_{t-1}  +  (1 − Z_t) ⊙ H̃_t        # blend old state and candidate

Note the final line is a convex combination: when Z_t → 1 the unit copies the old state verbatim (the additive skip that preserves gradient); when Z_t → 0 it overwrites with the fresh candidate. One gate does the job the LSTM splits between input and forget. No separate cell state, no output gate. GRU vs LSTM is one of ML's enduring "it depends" questions — GRU is cheaper and faster, LSTM has slightly more expressive memory; benchmarks split roughly evenly.

6. Minimal pseudocode (LSTM cell)

def lstm_step(x, h_prev, c_prev, P):
    # P holds all weight matrices and biases
    i = sigmoid(x @ P.Wxi + h_prev @ P.Whi + P.bi)
    f = sigmoid(x @ P.Wxf + h_prev @ P.Whf + P.bf)
    o = sigmoid(x @ P.Wxo + h_prev @ P.Who + P.bo)
    c_tilde = tanh(x @ P.Wxc + h_prev @ P.Whc + P.bc)
    c = f * c_prev + i * c_tilde      # the conveyor belt
    h = o * tanh(c)
    return h, c                        # carry BOTH forward

# A sequence is just this cell in a Python loop:
h, c = zeros(), zeros()
for x_t in sequence:
    h, c = lstm_step(x_t, h, c, P)     # h, c thread through time
# loss.backward() unrolls this loop = BPTT

If that loss.backward() over a Python loop feels mysterious, it's exactly what the [[micrograd]] and [[nn-zero-to-hero]] materials build from scratch: BPTT is just autograd over an unrolled graph, nothing more.

Key ideas & tradeoffs

Constant memory, sequential compute. O(L) work, O(1) state size — but inherently sequential: step t needs step t−1. You cannot parallelize an RNN over the time axis during training, which is its fatal weakness on modern GPUs (and the exact thing [[state-space-models]] and the [[attention-and-transformers]] family were designed to fix).
Gating = learned, additive memory control. The reusable lesson, far beyond RNNs: an additive identity path protected by a multiplicative sigmoid gate lets gradients flow and lets the model choose what to keep. This shows up in ResNets, highway networks, GLU/SwiGLU FFNs, and the selective gates inside Mamba.
Forget gate is the soul of the LSTM. Per Greff et al., remove it and performance collapses; everything else (peepholes, coupling input/forget) is marginal.
GRU vs LSTM: fewer gates, fewer params, often-equal accuracy. Default to GRU when compute-bound; reach for LSTM when you suspect you need finer-grained long memory.
Clipping for explosion, gating for vanishing. Two different problems, two different fixes. Don't conflate them.

Honest caveats

The fixed-size state is a hard ceiling. Everything the model knows about the past is one vector. This is fine for local structure, brutal for tasks needing precise recall of something 500 tokens back (e.g. "what was the third name mentioned?"). Attention sidesteps this by keeping every token addressable. This same bottleneck limits modern SSMs too — it's why the field landed on [[hybrid-attention-ssm-architectures]] rather than pure recurrence.
LSTMs mitigate, not eliminate, vanishing gradients. The carousel keeps gradients alive when the forget gate stays open. Over truly long horizons, or when the model genuinely needs to forget, gradients still attenuate. "Bridges 1000 steps" was a best-case claim, not a guarantee.
They don't parallelize over time — full stop. This, not accuracy, is why the frontier abandoned them. The Transformer trains every position simultaneously; an RNN cannot. On GPU clusters that gap is decisive, and no clever gating closes it.
GRU vs LSTM has no universal winner. Anyone claiming one always beats the other is overselling. It's task-, data-, and tuning-dependent.
The "constant error carousel" is an idealization. The clean ∂C_t/∂C_{t-1} = F_t story ignores that F_t, I_t, and C̃_t themselves depend on H_{t-1}, which depends on C_{t-1}. The real Jacobian has extra coupling terms; the carousel is the dominant path, not the only one.
Recurrence is not obsolete, but classic LSTMs largely are. New frontier work uses linear recurrences ([[linear-attention-and-deltanet]], [[state-space-models]]) specifically because the tanh nonlinearity in the recurrence is what blocks parallel training. The 1997 LSTM as-is is rarely the right choice for a new large model in 2026.

How it connects to OpenAlice + the Academy ladder

Academy ladder. RNNs/LSTMs sit at a hinge point in the curriculum — the bridge between "neural net basics" and "the Transformer era":

Foundations — [[micrograd]] and [[nn-zero-to-hero]] teach autograd; BPTT is just that autograd over an unrolled loop. [[neural-network-from-scratch]] and [[math-for-ml-foundations]] give the linear-algebra/Jacobian vocabulary the vanishing-gradient argument needs.
First sequence models — [[makemore]] builds a character-level generator and explicitly walks bigram → MLP → RNN → GRU, so learners feel recurrence and its limits firsthand before any attention.
The pivot — this article. Understand why recurrence struggles (BPTT, vanishing gradients) and how gating patches it. Then read [[attention-and-transformers]] to see what replaced it, and [[positional-encoding]] to see how Transformers recover the order information that recurrence got "for free."
The return of recurrence — [[state-space-models]] (Mamba), [[linear-attention-and-deltanet]], and [[hybrid-attention-ssm-architectures]] re-derive the RNN with a linear recurrence so it parallelizes. Reading those after this article, the lineage is unmistakable: gates, hidden state, constant-memory inference — all back, re-engineered.

OpenAlice relevance. Alice's stack is overwhelmingly Transformer-based, so LSTMs/GRUs aren't in the hot path. Their value here is twofold: (1) conceptual literacy — the gating/additive-path lesson underpins residual connections and the selective gates in the SSM hybrids the lab tracks for [[long-context]] efficiency; and (2) the constant-memory inference property is exactly the lever long-running streaming and edge scenarios care about, which is why the lab follows [[state-space-models]] as a Transformer alternative. The historical line — RNN → attention-on-RNN → Transformer → linear-recurrence comeback — is itself the map for reasoning about where sequence architectures go next.

Sources

Hochreiter & Schmidhuber (1997), Long Short-Term Memory — original LSTM + constant error carousel: https://www.bioinf.jku.at/publications/older/2604.pdf
Pascanu, Mikolov & Bengio (2013), On the difficulty of training RNNs — vanishing/exploding gradients, clipping: https://arxiv.org/abs/1211.5063
Cho et al. (2014), RNN Encoder-Decoder (GRU): https://arxiv.org/abs/1406.1078
Greff et al. (2015), LSTM: A Search Space Odyssey — gate ablations: https://arxiv.org/abs/1503.04069
Bahdanau, Cho & Bengio (2014), Neural MT by Jointly Learning to Align and Translate — attention for the RNN bottleneck: https://arxiv.org/abs/1409.0473
Vaswani et al. (2017), Attention Is All You Need — what replaced recurrence: https://arxiv.org/abs/1706.03762
Dive into Deep Learning, LSTM & GRU chapters (equation cross-check): https://d2l.ai/chapter_recurrent-modern/lstm.html · https://d2l.ai/chapter_recurrent-modern/gru.html
Olah (2015), Understanding LSTM Networks (canonical intuition): https://colah.github.io/posts/2015-08-Understanding-LSTMs/