Latent reasoning — thinking in continuous space instead of tokens

For NAO + anyone wondering "what comes after chain-of-thought?" The one-sentence version: instead of forcing a model to write out its reasoning as words ("First, 3 apples plus 2 apples..."), let it reason inside its own hidden state — passing a continuous vector from step to step, never decoding it to text — and only emit words for the final answer. This is the frontier past explicit chains of thought. It is genuinely promising and genuinely unsettled: it buys huge representational bandwidth and a kind of parallel search, but it costs interpretability and is hard to train. This article is intuition-first, then gives the real mechanics (Coconut's feedback loop, the recurrent-depth loop, the looped-transformer theory), then is honest about what is frontier and uncertain — including a 2025 paper that questions whether the latent tokens are "thinking" at all.

This is the spatial-of-the-model companion to [[test-time-compute-reasoning]] (o1 / R1 — same goal of "think longer at inference," but those models think in tokens; this article thinks in vectors). It leans hard on [[attention-and-transformers]] (you need to understand a hidden state and an input embedding to follow any of it) and is a cousin of [[state-space-models]] (recurrence as an alternative to attention) and [[long-context]] (both are about where computation "lives"). Related rewards machinery: [[process-outcome-rewards]], [[grpo]], [[rlvr]].

What it is (the intuition)

When o1 or DeepSeek-R1 "reason," they do it by generating tokens — a long visible (or hidden) monologue. Each reasoning step is a word, and each word is chosen by sampling one entry from a ~100,000-word vocabulary. That has two under-appreciated costs:

1. A token is a *tiny* communication channel. Picking one of ~50k tokens carries about log2(50000) ≈ 15.6 bits of information per step. Everything the model "figured out" on this step has to be squeezed through that 15-bit straw before the next step can use it.
2. Reasoning is forced to be serial *and* committal. To explore two possibilities, a token-CoT model must literally write out both branches, one after the other.

Latent reasoning asks: why decode to words at all? The transformer already computes a rich hidden state — a continuous vector of dimension d (2,048–8,192 for modern models) — at the last layer of each step. That vector is the model's actual "thought." Latent reasoning feeds that vector straight back in as the next step's input, skipping the "pick a word" bottleneck entirely.

The survey's headline comparison makes the stakes concrete: an explicit reasoning step transmits one discrete token (~15 bits), while a latent step passes a full 2,560-dim FP16 hidden state (~40,960 bits) — roughly a 2,700× bandwidth gap between the two channels (Zhu et al. 2025, A Survey on Latent Reasoning). Latent reasoning is, in one line, spending that 2,700× instead of throwing it away.

Ramp for beginners. Token CoT = the model thinks out loud and you read its diary. Latent reasoning = the model thinks silently and only tells you the conclusion. The diary is interpretable but low-bandwidth and serial; the silent thought is high-bandwidth and can hold several half-formed ideas at once — but you can't read it, and that's the whole tension.

Why it matters

• Bandwidth. As above: a hidden state can carry orders of magnitude more information per reasoning step than a token. Some reasoning is, in the authors' phrasing, "not easily represented in words" (Geiping et al. 2025) — latent reasoning doesn't force it to be.
• Implicit parallel search (superposition). A continuous thought isn't committed to a single next step — it can encode a distribution over next steps. Coconut's authors show this behaves like a breadth-first search: the model keeps several candidate reasoning paths alive in superposition and prunes later, instead of greedily committing to one chain and backtracking in text. This is exactly where token-CoT struggles (planning-heavy logical problems).
• Test-time compute that scales depth, not length. [[test-time-compute-reasoning]] scales by emitting more tokens. Latent reasoning can instead loop the same block more times — scaling effective depth with no extra context length and no extra reasoning data. Huginn (a 3.5B-param recurrent-depth model) reaches the reasoning performance of a model equivalent to ~50B parameters purely by iterating its recurrent block more times at test time (Geiping et al. 2025).
• Depth is what reasoning actually needs. Saunshi et al. (2025) argue many reasoning problems (addition, p-hop induction, graph connectivity) need depth, not parameters — a k-layer block looped L times nearly matches a kL-layer model on these, at a fraction of the weights.

How it works (the real mechanics)

There are two main families. The survey (Zhu et al. 2025) splits them as activation-based / vertical (recurrent-depth) vs horizontal / token-recurrent, with a separate weight-based internalization branch. The two you must understand are Coconut (horizontal feedback) and recurrent-depth / looped transformers (vertical).

Family 1 — Coconut: chain of continuous thought

Coconut ("Continuous Concatenuted... ", Hao et al., Meta FAIR 2024) is the canonical horizontal method. A standard LM in token mode does, per step:

x_t  →  embed →  e_t            (token id → embedding)
e_1..e_t  →  Transformer  →  H_t = last-layer hidden states
h_t  →  unembed/softmax  →  sample next token x_{t+1}

Coconut adds a latent mode, fenced by two special tokens <bot> (begin thought) and <eot> (end thought). Inside the fence, it skips the unembed-and-sample step and feeds the hidden state straight back as the next input embedding:

LATENT MODE (between <bot> and <eot>):
    e_{t+1}  =  h_t            # last hidden state IS the next input embedding
                               # no softmax, no token, no 15-bit bottleneck

So a "continuous thought" is just h_t re-injected as e_{t+1}. The model runs c such latent steps, then emits <eot> and switches back to normal token generation for the answer. The thought vectors are never decoded to words.

Training curriculum (the hard part). You can't just turn this on — there's no ground truth for what a latent vector "should" be. Coconut uses a multi-stage curriculum that gradually swaps language reasoning for latent thoughts. Given a worked solution with N language reasoning steps:

• Stage 0: normal CoT — all N steps are text.
• Stage k: replace the first k language steps with k·c continuous thoughts; the remaining N−k steps stay as text.
• Increase k until (ideally) all reasoning is latent.

Loss: standard next-token cross-entropy, but computed only on the remaining text tokens — never on the latent vectors themselves. The latent thoughts get no direct supervision; they are shaped purely by their downstream usefulness in predicting the surviving text (and the answer). Backprop flows through the fed-back hidden states, which makes training a long latent chain expensive (the whole sequence of forward passes must be differentiated through).

Results (directional — Coconut paper). On GSM8K (grade-school math), ProntoQA and ProsQA (logical reasoning with search), Coconut matches or beats CoT while using fewer forward passes during generation, and the gap is largest on the planning-heavy ProsQA — consistent with the BFS / superposition story. (Treat exact percentages as model- and setup-specific; the robust claim is "competitive accuracy at better accuracy-vs-compute trade-off, biggest on search-heavy tasks.")

Family 2 — Recurrent depth & looped transformers (vertical)

Here you don't pass thoughts across token positions; you loop a block of layers in place, deepening the computation for the same position.

Huginn / recurrent-depth (Geiping et al. 2025). The architecture is a prelude → recurrent core block → coda. The core block is iterated r times, the number of iterations chosen at test time:

s_0 = embed(x)                       # injected state
for i in 1..r:                       # r recurrences, set at INFERENCE time
    s_i = CoreBlock(s_{i-1}, e)      # same weights every iteration; e = input embed re-injected
y   = Coda(s_r)                      # decode to logits

Crucially r is not fixed by training — you can spend more test-time compute on a hard prompt by looping more. A 3.5B model trained on 800B tokens scales its reasoning to ~50B-param-equivalent compute purely via larger r. No CoT data, no long context window required.

Looped-transformer theory (Saunshi et al. 2025). This is the "why it works" result. Key claims:

• A k-layer transformer looped `L` times nearly matches a `kL`-layer non-looped model on reasoning — "reasoning needs depth, not parameters."
• A looped transformer with depth T can simulate `T` steps of CoT under mild assumptions — loops and CoT steps are, formally, two ways to buy the same effective depth.
• With depth Θ(log n) a looped transformer solves problems (regular-language recognition, graph connectivity) that are intractable for an LLM limited to logarithmically-many CoT steps.
• Looped and non-looped models show scaling behavior governed by effective depth — the inference-time scaling of CoT and the loop-count scaling of latent reasoning are the same phenomenon viewed two ways. (See [[scaling-laws]].)

The unifying idea: token-CoT, looping a block, and Coconut's fed-back vectors are three knobs on the same machine — they all buy more sequential computation / effective depth. Latent methods buy it without paying the 15-bit token tax and without lengthening the context.

Key ideas & tradeoffs

The dominant tradeoff is the bandwidth ⇄ interpretability swap. The same property that makes latent reasoning powerful (it isn't bottlenecked through words) makes it opaque (it isn't expressed in words). For a system that values auditability — see [[mechanistic-interpretability]] and OpenAlice's safety-gate posture below — that is not a free lunch.

Honest caveats & open questions

This is a frontier topic. Be skeptical, including of the framing above.

• Do the latent tokens actually "think"? A 2025 causal-and-adversarial study, Do Latent Tokens Think? (arXiv:2512.21711), probes chain-of-continuous-thought and finds reasons to doubt that the continuous thoughts perform the rich search/superposition we'd like to attribute to them. The BFS interpretation is suggestive, not settled — some apparent gains may be the latent tokens acting more like extra compute / extra "pause" capacity than as genuine alternative reasoning paths. Hold the superposition story loosely.
• Training is the real blocker. No supervision exists for the latent vectors, so Coconut leans on a fiddly curriculum and backprop through a chain of forward passes. It does not obviously scale to long latent chains, and the survey lists optimization difficulty as a top open problem. Recurrent-depth sidesteps the curriculum but bakes the recurrence into pretraining (expensive, and you can't bolt it onto an existing frozen model).
• Evaluation is unstandardized. There's no agreed way to measure "latent reasoning quality." Most published wins are on relatively narrow / synthetic reasoning suites (GSM8K, Prosit/ProntoQA/ProsQA, p-hop, connectivity). General agentic / open-ended performance at scale is far less proven than for token-CoT reasoners, which already ship in production. See [[llm-evaluation]] and [[agent-evaluation]].
• You lose the audit trail. Token CoT can be read, scored by a process-reward model ([[process-outcome-rewards]]), and inspected after an incident. Latent reasoning gives up that visible trace — a real cost for safety-critical or regulated deployments, not just an aesthetic one.
• No frontier production reasoner is (publicly) latent yet. As of mid-2026 the shipping reasoning models (o-series, R1-lineage, etc.) reason in tokens. Latent reasoning is the research frontier, not yet the default. Do not assume a given production model uses it.

Net: latent reasoning is one of the most credible answers to "what's past chain-of-thought," with real theory (looped transformers) and real systems (Huginn, Coconut) behind it — but it trades away interpretability, is harder to train, and at least one careful study questions how much "reasoning" is really happening in the latents. Frontier, not finished.

How it connects to OpenAlice

OpenAlice / Alice today reasons the token way — via provider reasoning models (gpt-5.x, R1-lineage) and explicit chains, scored and routed by the machinery in [[test-time-compute-reasoning]], [[process-outcome-rewards]], and [[model-routing]]. Latent reasoning is relevant to us as a forward-looking lens, not a current dependency:

• The safety-gate tension is ours directly. OpenAlice's posture is HITL approvals, kill-switch, and auditability (Alice's [[agentic-loops]] are designed to be inspectable). Latent reasoning's loss of a readable trace is exactly the property our gate/standing-approval architecture relies on. If we ever adopt latent-reasoning backends, we'd need a parallel observability story (probes, forced periodic token check-ins) — an explicit research item, not a freebie.
• Test-time depth as a routing knob. The recurrent-depth idea — "spend more loops on a hard prompt" — maps cleanly onto how [[model-routing]] and Alice's effort/critic machinery already decide how hard to think. Even with token models, the conceptual frame ("buy effective depth on demand") is the same.
• Atlas / [[autoresearch]] tracking. This is a fast-moving frontier; the five sources here (Coconut, Huginn/recurrent-depth, looped-transformer theory, the survey, and the skeptical Do Latent Tokens Think? paper) are the spine to re-check when curating. Status here is deliberately frontier.

Sources

• Hao et al., Coconut: Training Large Language Models to Reason in a Continuous Latent Space (Meta FAIR, 2024) — arXiv:2412.06769. The canonical continuous-thought / hidden-state-feedback method + BFS interpretation.
• Geiping et al., Scaling up Test-Time Compute with Latent Reasoning: A Recurrent Depth Approach (2025) — arXiv:2502.05171. The Huginn recurrent-depth model; 3.5B → ~50B-equivalent via test-time loops.
• Saunshi et al., Reasoning with Latent Thoughts: On the Power of Looped Transformers (2025) — arXiv:2502.17416. The theory: loops ≈ depth ≈ CoT steps.
• Zhu et al., A Survey on Latent Reasoning (2025) — arXiv:2507.06203. Taxonomy (vertical/horizontal, activation/weight-based) + the bandwidth argument.
• Do Latent Tokens Think? A Causal and Adversarial Analysis of Chain-of-Continuous-Thought (2025) — arXiv:2512.21711. The skeptical counterweight.