DeepSeek architecture (V2 / V3 / R1) — doing more with less, from MLA to GRPO

For NAO + anyone trying to understand why a Chinese lab trained a GPT-4-class model for ~$5.6M and then open-sourced a reasoning model that rivalled OpenAI o1. DeepSeek is not one trick — it is a stack of efficiency ideas that compound: a cheaper attention (MLA), a cheaper FFN (DeepSeekMoE), a cheaper training signal (Multi-Token Prediction), cheaper arithmetic (FP8), and a cheaper way to teach reasoning (GRPO RL with no human labels). This article reads each one intuition-first, then gives you the real equations and the honest caveats. Sourced from the DeepSeek-V2, V3, R1 and DeepSeekMath arXiv papers plus two excellent technical walkthroughs.

This is the deep architecture companion to [[mixture-of-experts]], [[attention-and-transformers]], [[quantization]], [[rlhf-and-alignment]] and [[test-time-compute-reasoning]]. If those are the concepts, DeepSeek is the system that wires them all together at frontier scale.

What it is (intuition first)

Every modern LLM is a tall stack of two repeating blocks: attention (tokens look at each other) and a feed-forward network / FFN (each token thinks on its own). Making models smarter has mostly meant making both bigger. DeepSeek's whole research program is a different bet: keep the model huge in *capacity* but tiny in *what you actually pay for per token*. Three observations drive it:

1. The KV cache is the inference tax. During generation, a transformer must remember a Key and Value vector for every past token, every head, every layer. At long context this dominates memory and bandwidth. DeepSeek's Multi-head Latent Attention (MLA) compresses that memory by ~40–60×.

1. Dense FFNs waste compute. A 671B-parameter dense model would multiply all 671B weights for every token. DeepSeekMoE (a Mixture-of-Experts FFN) routes each token to a few small expert sub-networks, so DeepSeek-V3 has 671B total parameters but activates only 37B per token — GPT-4-class capacity at a fraction of the FLOPs.

1. Reasoning can be *learned*, not *labelled*. Instead of paying humans to write step-by-step solutions, DeepSeek-R1 rewards the model only on whether the final answer is correct, and lets reinforcement learning grow the chain-of-thought by itself. The model spontaneously learns to "think longer," backtrack, and double-check — the famous "aha moment."

So the lineage is: V2 introduced MLA + DeepSeekMoE (236B/21B). V3 scaled it to 671B/37B and added Multi-Token Prediction + FP8 training, hitting GPT-4-class quality for ~2.79M H800 GPU-hours (≈ $5.6M in rented compute). R1 took the V3 base and turned it into an o1-class reasoning model purely through RL (arXiv:2412.19437↗, arXiv:2501.12948↗).

Why it matters

• It broke the "only hyperscalers can build frontier models" assumption. The V3 report claims 2.788M H800 GPU-hours total — pre-training on 14.8T tokens cost ~2.66M of those, and no irrecoverable loss spike or rollback happened across the whole run (arXiv:2412.19437↗). That number, plus full open weights, is what caused the "DeepSeek moment" in January 2025.

• The ideas are reusable, not one-off hacks. MLA is now a standard option for KV-cache reduction; auxiliary-loss-free MoE balancing and GRPO have been adopted widely. DeepSeekMath's GRPO (arXiv:2402.03300↗) is now one of the most common RL algorithms for reasoning models because it deletes the critic network that PPO needs.

• It open-sourced a working recipe for reasoning models — R1-Zero showed you can get strong reasoning from a base model with pure RL and no SFT at all, then R1 showed how to make that usable, then distillation showed you can pour the reasoning into small dense models that run on a laptop.

How it works (the real mechanics)

1. Multi-head Latent Attention (MLA) — compress the KV cache

The problem. In standard Multi-Head Attention (MHA) you cache, per token, a Key and Value of size n_h × d_h for each layer. In DeepSeek-V3 that is 128 heads × 128 dims × 2 (K and V) = 32,768 floats per token per layer (McCormick↗). At 128K context that explodes.

The idea. Don't cache the big K and V. Cache a small shared latent vector and reconstruct K and V from it on the fly. Concretely, for hidden state h_t (dimension d = 7168 in V3):

c_KV_t = h_t · W_DKV         # down-project to latent, d_c = 512   (this is what we cache)
k_C_t  = c_KV_t · W_UK       # up-project to per-head keys   (at inference, "absorbed" into W_Q)
v_C_t  = c_KV_t · W_UV       # up-project to per-head values (absorbed into the output proj)

W_DKV is 7168 × 512. The cached object is just c_KV_t — 512 floats per token per layer instead of 32,768 (McCormick↗). The query is compressed the same way (down to d_c' = 1536) to save activation memory during training.

The RoPE wrinkle (decoupled RoPE). Rotary position embeddings rotate K and Q by a position-dependent angle. But if you absorb W_UK into the query weights to avoid materialising K, RoPE can no longer be applied cleanly (it doesn't commute with the absorbed matmul). DeepSeek's fix: split each key/query into two pieces — a content part computed from the latent (no RoPE), and a small decoupled RoPE part (extra dimension d_R = 64 per head) that does carry position. You cache the latent plus one shared RoPE key vector. So the real per-token cache is d_c + d_R = 512 + 64 = 576 floats per layer — still a ~40–60× reduction, and V2 reported a 93.3% KV-cache cut and 5.76× higher max throughput vs the dense DeepSeek-67B (arXiv:2405.04434↗).

Weight absorption. At inference, because attention is linear in the latent, you can pre-multiply W_UK into W_Q and W_UV into the output projection. Then you never materialise the full per-head K/V at all — you attend directly in latent space. That is what makes MLA faster, not just smaller.

Intuition: instead of storing every head's full key and value, store one compressed "summary" per token and a tiny position tag, and let the math unfold the heads only when needed.

2. DeepSeekMoE — fine-grained experts + a shared expert

A Mixture-of-Experts replaces the single big FFN with N expert FFNs and a router that sends each token to the top-K. (See [[mixture-of-experts]] for the general idea.) DeepSeekMoE makes two specific design choices:

• Fine-grained expert segmentation. Instead of a few fat experts, use many thin ones. DeepSeek-V3 has 256 routed experts per MoE layer and activates the top-8 for each token. Slicing experts finer lets the router combine specialists more precisely — more combinations of knowledge for the same active-parameter budget.

• Shared-expert isolation. Some knowledge (grammar, common formatting) is needed by every token. Forcing the router to re-learn it in every expert wastes capacity. So DeepSeekMoE carves out 1 shared expert that always runs, alongside the 8 routed ones. The shared expert holds the common knowledge; the routed experts specialise (arXiv:2405.04434↗, V3 config from arXiv:2412.19437↗).

So DeepSeek-V3's FFN per token = 1 shared + 8 routed (of 256) experts, giving 671B total params, 37B active.

Auxiliary-loss-free load balancing (V3's headline trick). MoEs collapse if the router sends everything to a few favourite experts. The classic fix is an auxiliary loss that punishes imbalance — but that loss fights the language objective and hurts quality. DeepSeek-V3 drops it. Instead each expert gets a bias term `b_i` added only to its routing score for the top-K *selection* — not to the gate value that actually weights the output. After each step they nudge the bias:

if expert i was overloaded:   b_i ← b_i − γ
if expert i was underloaded:  b_i ← b_i + γ

where γ is a small "bias update speed." This is a control loop: overloaded experts get quietly down-ranked in selection while their contribution weights stay clean, so balance is enforced without a gradient that corrupts the loss (arXiv:2412.19437↗; good writeup at Yugen.ai↗). V2 still used device-limited routing + small balance losses; V3's auxiliary-loss-free scheme is the cleaner successor.

3. Multi-Token Prediction (MTP) — a denser training signal

Standard LLMs predict one next token per position. MTP makes the model predict the next few tokens at each position, using small sequential MTP modules that each have their own output head but share the trunk (arXiv:2412.19437↗). Two payoffs:

1. Denser learning signal — each position now supplies several prediction targets, so the model squeezes more learning out of every token of the 14.8T corpus.
2. Speculative decoding for free — the extra heads can propose the next 1–2 tokens, which a verification pass accepts or rejects, speeding inference. MTP is a training objective; the modules can be dropped at inference or reused for speculation.

4. FP8 mixed-precision training — cheaper arithmetic

DeepSeek-V3 is one of the first frontier models trained largely in FP8 (8-bit floats) instead of BF16, roughly halving memory and boosting throughput. FP8 has tiny dynamic range, so naïvely it overflows/underflows. DeepSeek's recipe:

• Fine-grained (tile/block-wise) quantization. Scale factors are computed per small tile (e.g. 1×128 activation tiles, 128×128 weight blocks) instead of per whole tensor, so one outlier can't wreck the scale of a big block.
• Keep the sensitive parts in higher precision. Embeddings, the output head, normalization, the attention softmax, and the master weights / optimizer states stay in BF16/FP32. The heavy GEMMs (matmuls) run in FP8.
• High-precision accumulation. FP8 multiply, but accumulate partial sums in higher precision to stop error build-up.

Combined with the DualPipe pipeline-parallel schedule (which overlaps computation with cross-GPU communication to hide the all-to-all cost of MoE routing), this is how the run finished in ~2.79M H800-hours with no loss spikes or rollbacks (arXiv:2412.19437↗).

5. DeepSeek-R1 — learning to reason with GRPO

Take the V3 base model and teach it to reason. DeepSeek did it with reinforcement learning, not human-written reasoning traces.

GRPO (Group Relative Policy Optimization). PPO — the usual RLHF algorithm (see [[rlhf-and-alignment]]) — needs a separate critic / value network the same size as the policy, which doubles memory. GRPO (DeepSeekMath, arXiv:2402.03300↗) deletes the critic. For each prompt it samples a group of G outputs, scores them, and uses the group's own statistics as the baseline:

Â_i = ( r_i − mean(r_1..r_G) ) / ( std(r_1..r_G) + ε )      # group-relative advantage

Every token in output i gets the same advantage Â_i. The objective is a clipped surrogate (PPO-style) with the KL penalty subtracted from the loss rather than baked into the reward:

J_GRPO(θ) = E[ (1/G) Σ_i (1/|o_i|) Σ_t
              min( ρ_{i,t}·Â_i , clip(ρ_{i,t}, 1−ε, 1+ε)·Â_i )
              − β · D_KL(π_θ ‖ π_ref) ]

where ρ_{i,t} is the new/old policy probability ratio, and the KL is the unbiased low-variance estimator exp(Δ) − Δ − 1 with Δ = log π_ref − log π_θ (Cameron Wolfe↗). The advantage is relative: "was this answer better or worse than its siblings on the same prompt?" — which is exactly the signal you want and needs no learned value model.

Rule-based rewards (RLVR). R1 mostly avoids a learned reward model (which can be reward-hacked). For verifiable tasks the reward is mechanical: accuracy reward (math: does the boxed final answer match? code: does it pass the unit tests in a sandbox?) plus a format reward (did it wrap its thinking in <think>…</think>?) (arXiv:2501.12948↗).

R1-Zero and the "aha moment." Applying GRPO directly to the base model with no SFT at all produced DeepSeek-R1-Zero, which spontaneously learned to generate longer chains of thought, re-evaluate its approach, and verify its own work — the model literally writes "wait, let me reconsider." Its AIME score climbed from ~15.6% to ~71% during RL. But R1-Zero had ugly outputs: mixed languages, poor readability (arXiv:2501.12948↗).

The full R1 pipeline (4 stages) fixes that:

1. Cold-start SFT — fine-tune the base on a few thousand high-quality, readable reasoning traces to give RL a clean starting point.
2. Reasoning-oriented RL — GRPO with accuracy + format + a language-consistency reward to stop language mixing.
3. Rejection sampling + second SFT — generate many samples from the RL model, keep only the good ones (≈600K reasoning + ≈200K general samples), and re-SFT to broaden into writing, QA, and general helpfulness.
4. Second RL — a final RL pass aligning for helpfulness and harmlessness across all task types.

Distillation. They then SFT small dense models (Qwen and Llama, 1.5B→70B) on R1's outputs. The result: distilled-Qwen-32B beats much larger models on reasoning benchmarks — strong reasoning transfers into small models cheaply, and the paper finds distillation-from-a-strong-reasoner beats running RL directly on the small model (arXiv:2501.12948↗).

Key ideas & tradeoffs

The throughline: each trick attacks a different cost axis (memory, FLOPs, training signal, bit-width, RL infrastructure, human labels), and because they're roughly independent they stack multiplicatively. That compounding is the real "DeepSeek architecture," not any single equation.

Honest caveats & open questions

• The "$5.6M" is the *final training run*, not total cost. It excludes prior R&D, failed runs, data, salaries, and the hardware itself. It is a real and impressive number, but it is not "anyone can build GPT-4 for $6M."

• MLA's win is context-dependent. The dramatic KV-cache reduction matters most at long context and high batch size. The extra projections add compute, and the McCormick walkthrough notes the latent is read twice, so bandwidth savings are roughly half the memory savings (McCormick↗).

• Serving a 671B MoE is not cheap. "37B active" is the compute story; you still need enough GPUs to hold all 671B parameters in memory. MoE shifts cost from FLOPs to memory + interconnect, which is why expert-parallelism and load-balancers (DeepSeek's own EPLB) exist.

• Rule-based RL has a hard boundary. R1-Zero's magic needs a checkable answer. For open-ended generation (essays, dialogue, taste) you're back to learned reward models or human preference — and those can be reward-hacked. R1's later stages reintroduce SFT and general alignment for exactly this reason.

• R1-Zero's raw outputs were not deployable (language mixing, poor readability). The clean R1 product still depends on cold-start SFT + multiple alignment passes — "pure RL" is a research result, not the shipped recipe.

• Reproducibility of FP8 is genuinely hard. Several independent groups reproduced parts of R1's RL recipe quickly, but the FP8 + DualPipe pre-training stack is far harder to replicate exactly.

• The MHC angle (from the linked video) is a *later, separate* line of work. The transcript (Kalebrights Code, video z9LRAxVRXOk) discusses DeepSeek's "Manifold-Constrained HyperConnections" (MHC) — a 2026 paper, not a model release — which constrains ByteDance-style "hyperconnections" (richer residual plumbing than ResNet's single skip) onto a doubly-stochastic Birkhoff polytope via the Sinkhorn-Knopp algorithm to stop the residual stream from exploding (the video cites >3000× amplification by layer 60 in raw hyperconnections). Treat this as adjacent and provisional: the transcript has audio-transcription errors ("Burkov"→Birkhoff, "sync corn knob"→Sinkhorn-Knopp), it is one explainer not the primary paper, and it is orthogonal to the V2/V3/R1 stack above. The video's broader claim — that DeepSeek's edge is rewiring residual/normalization plumbing for stability at depth (post-LN vs pre-LN tradeoffs, representation collapse) — is a fair framing of the research culture, not a spec.

How it connects to OpenAlice

Honest, specific ties only:

• Model routing & councils. OpenAlice already routes across providers and fuses multiple models ([[model-routing]], [[fusion-and-llm-councils]], [[mixture-of-agents]]). DeepSeek-V3/R1 are open-weight, strong, and cheap to run per token — a natural candidate as a routed backend (especially R1 for verifiable/reasoning-heavy turns where its long chain-of-thought pays off). The cost-per-token math from MLA + MoE is exactly the kind of input a router weighs.

• Reasoning vs companion turns. OpenAlice's per-chat InteractionMode (Companion vs Worker) maps cleanly onto DeepSeek's split: a Worker turn that must get the answer right (e.g. a diploma, code) benefits from an R1-style reasoning model with test-time compute; a Companion turn does not. See [[test-time-compute-reasoning]] for when "think longer" actually helps.

• GRPO as a self-improvement primitive. OpenAlice's AGI/coding direction (repo→PR, measured on SWE-bench) is a domain with verifiable rewards (does the patch pass the tests?). That is precisely the setting where GRPO + rule-based rewards shine — a concrete, honest path if OpenAlice ever trains its own reasoning/coding policy rather than only prompting one.

• No deeper claim than that. OpenAlice does not currently train MoE/FP8 base models, so MLA/MTP/FP8 are reference knowledge here, not active components.

See also

• [[mixture-of-experts]] — the general MoE mechanism DeepSeekMoE refines.
• [[attention-and-transformers]] · [[flash-attention]] · [[positional-encoding]] — MLA is an attention variant; decoupled RoPE is a positional-encoding move.
• [[quantization]] — FP8 training is quantization pushed into the training loop.
• [[rlhf-and-alignment]] — GRPO is the RL family R1 belongs to; PPO is the foil.
• [[test-time-compute-reasoning]] — why R1's long chains-of-thought help.
• [[scaling-laws]] — DeepSeek is the "efficiency frontier" counterpoint to brute scale.
• [[model-routing]] · [[fusion-and-llm-councils]] · [[mixture-of-agents]] — how a cheap strong open model slots into a multi-model system.