World Models

What it is (intuition first)

Close your eyes and imagine pushing a coffee cup off the edge of a table. You don't need to do it to know what happens: the cup falls, it accelerates, it shatters, coffee splashes outward. You ran a little simulation in your head. That internal simulator — the thing that lets you predict "if I take this action, the world will become that" — is a world model.

In AI, a world model is a learned simulator of an environment. It takes the current state (often raw pixels or a video), optionally an action, and predicts the next state:

next_state  ≈  WorldModel(current_state, action)

Roll that prediction forward — feed the predicted next state back in as the new current state — and you get an imagined rollout: a dream of how the future unfolds. The two things that make this powerful:

1. It's action-conditioned. A plain video predictor answers "what happens next?" A world model answers "what happens next if I do X?" That dependence on action is the whole game, because it's what turns prediction into planning: try many imagined action sequences, keep the one whose imagined future you like, then act for real.
2. It can be learned from unlabelled experience. The world is its own supervision signal — the next frame is always available as the answer. So you can train a world model on raw internet video or raw robot logs without anyone hand-labelling "correct" outcomes.

There is a deep and genuinely contested split in how to build one, and it's worth holding both pictures in your head from the start:

• Render the world to predict it. Predict the pixels of the future. If your video looks real enough — physics, lighting, object permanence — you've implicitly captured how the world works. This is the generative-video camp: [[diffusion-models|Sora]], DeepMind's Genie. The bet: rendering reality faithfully is understanding it.
• Compress the world to understand it. Don't waste capacity painting every pixel. Predict in an abstract latent space — capture only the bits that matter for what-happens-next, throw away the rest. This is JEPA (Yann LeCun) and the model-based-RL lineage (Dreamer). The bet: a leaf's exact texture is unpredictable noise; understanding lives in the abstraction, not the pixels.

This page assumes you're comfortable with neural nets, gradients, and the transformer ([[neural-network-from-scratch]], [[attention-and-transformers]], [[math-for-ml-foundations]]). World models lean heavily on two other pillars in this library — [[diffusion-models]] (the engine of the generative camp) and [[multimodal-llms]] (the trick of turning pixels into tokens a sequence model can consume). We'll tie back to both.

Honesty up front: this is a frontier topic, mid-2026. Some of what follows is settled engineering (Dreamer, JEPA, Genie 1). Some — "video models are world simulators," "world models are the path to AGI/embodiment" — is an active, unresolved bet that serious researchers disagree about. We flag which is which throughout.

Why it matters

1. Planning and embodiment. Robots and agents that have to act in the physical (or a complex digital) world cannot afford to learn purely by trial-and-error in reality — it's slow, expensive, and dangerous. A world model lets an agent imagine the outcome of candidate actions and pick a good one before moving a single motor. This is the core loop of model-based RL and, in 2025, of robot control (V-JEPA 2-AC plans on a Franka arm; Genie 3 is pitched as a place to train embodied agents).

2. Infinite, controllable training environments. Hand-built simulators (game engines, MuJoCo) are expensive and don't cover the long tail of reality. A learned world model that can generate an endless variety of interactive environments from a prompt is, in effect, an infinite curriculum for agents. That's the explicit pitch of Genie 1 → Genie 3.

3. Sample efficiency. Because a world model squeezes signal out of every transition (predicting the next state is always a valid task), it can learn far more per unit of real-world interaction than model-free RL. DreamerV3 collected diamonds in Minecraft from scratch — long-horizon, sparse-reward — with no human data and no curriculum, something model-free methods had failed at.

4. The (contested) AGI argument. A recurring claim — LeCun's most loudly — is that an agent without a predictive model of the world cannot reason or plan in any deep sense, and that learning such a model from sensory data is the missing piece between today's LLMs and grounded intelligence. Whether scaling video generation gets you there, or whether you need JEPA-style abstraction, is one of the live debates of the field.

How it works (real mechanics)

0. The ancestor: Ha & Schmidhuber's V / M / C (2018)

The modern formulation crystallised in Ha & Schmidhuber's "World Models." Three components:

• V (Vision) — a VAE that compresses each high-dimensional frame o_t into a small latent vector z_t. (Throw away pixels, keep a compact code.)
• M (Memory) — an RNN (MDN-RNN) that models the temporal dynamics: given z_t and action a_t, predict a distribution over the next latent z_{t+1}. This is the world model proper — it knows how the world moves.
• C (Controller) — a tiny linear policy mapping [z_t, h_t] (latent + RNN hidden state) to an action.

The headline result that still defines the field: they trained the controller entirely inside M's hallucinated rollouts — "in the dream" — never touching the real environment during policy learning, then transferred the policy back to reality. Every modern world model is a variation on this skeleton: compress (V) → predict dynamics (M) → plan/act using the prediction (C).

1. The model-based-RL line: DreamerV3

DreamerV3 is the most mature, "it just works across 150+ tasks with one config" instantiation of the V/M/C idea. Its world model is the RSSM (Recurrent State-Space Model), whose key design choice is a split latent state:

• a deterministic recurrent state h_t (a GRU carries history forward), and
• a stochastic latent z_t (a discrete/categorical code sampled each step).

The RSSM components (schematically):

Recurrent:        h_t   = f(h_{t-1}, z_{t-1}, a_{t-1})        # GRU
Prior (predict):  ẑ_t   ~ p(z_t | h_t)                        # imagine next latent w/o seeing the frame
Posterior (fit):  z_t   ~ q(z_t | h_t, x_t)                   # correct it once the real frame x_t arrives
Decoders:         x̂_t   = decode(h_t, z_t)                    # reconstruct frame (training signal)
                  r̂_t   = reward(h_t, z_t)                    # predict reward
                  ĉ_t   = continue(h_t, z_t)                  # predict episode-continue flag

Training minimises a reconstruction + reward + KL loss; the KL between prior and posterior is what teaches the prior to predict the future without seeing it — that's the part that becomes the imagination engine. "Free bits" (clip KL below a threshold) and KL balancing keep this term from collapsing or exploding.

The behaviour is then learned purely in imagination: starting from real encoded states, roll the prior forward H steps using the current actor, and train an actor-critic on those imagined trajectories (no environment interaction inside the loop). The robustness tricks that made it domain-agnostic:

• symlog transform on rewards/values: symlog(x) = sign(x)·ln(1+|x|) — squashes wildly different magnitude scales so one set of hyperparameters works whether rewards are ±1 or ±10⁴.
• two-hot encoding of scalar targets into a discrete distribution, so regression becomes a stable softmax-classification.

Tradeoff: Dreamer is fantastically sample-efficient and general, but its latent is small and its rollouts are short-horizon; it's a control engine, not a photorealistic generator. You plan in latents you can't look at.

2. The generative-video line: Genie

Genie (DeepMind, 2024) is the "learned simulator you can actually play" — an 11B-parameter foundation world model trained unsupervised on ~30,000 hours of unlabelled internet 2D-platformer gameplay, with no action labels at all. Three pieces, and the middle one is the clever bit:

(a) Spatiotemporal (ST) video tokenizer. A VQ-VAE built on an ST-transformer turns video into discrete tokens. The ST-transformer interleaves spatial-attention layers (attend within a frame) and temporal-attention layers (attend across time, causally). Why interleave instead of full spatiotemporal attention? Cost. Naive attention over all patches × all frames is quadratic in (patches·frames); factorising it makes memory grow roughly linearly in the number of frames, which is what lets you model long video at all. (This factorised-attention trick is the same family of idea you'll meet in [[attention-and-transformers]] and the efficiency lineage around [[state-space-models]].) Each token, thanks to causal temporal attention, carries information from all earlier frames.

(b) Latent Action Model (LAM) — the unsupervised-action trick. Here's the problem: internet video has no action labels. Genie infers them. The LAM looks at frames x_1…x_t and the actual next frame x_{t+1}, and is forced to summarise "what changed" into a latent action `a_t` — through a VQ-VAE bottleneck whose codebook holds only 8 codes. That tiny vocabulary is the entire trick: with only 8 possible "actions," the model cannot cheat by copying the next frame; it must discover the 8 most useful, controllable axes of change (move left, jump, …) that, combined with the past, explain the future. The LAM is used only in training; at play time the human supplies one of those 8 action indices per frame.

(c) Dynamics model. A decoder-only MaskGIT transformer takes past frame-tokens + the latent action and predicts the next frame's tokens (iterative masked-token decoding, not raw left-to-right). Compose it: token current frame → pick a latent action → dynamics predicts next-frame tokens → decode to pixels → repeat. That loop is a playable, action-controllable simulator learned with zero action labels.

Pseudocode for interactive play:

tokens = tokenizer.encode(initial_frame)
loop:
    a = user_action_index            # one of the 8 learned latent actions
    tokens = dynamics.predict(tokens_history, a)   # MaskGIT next-frame tokens
    frame  = tokenizer.decode(tokens)
    display(frame)

Genie 3 (Aug 2025) scaled this into a striking frontier artifact: real-time generation at 720p, 24 fps, navigable live, with "promptable world events" (change the weather, spawn a character) and visual consistency holding for several minutes with explicit visual memory ~1 minute back. It's pitched explicitly as an environment to train embodied agents in. (Architectural details for Genie 3 are not fully disclosed — treat specifics as the company's claims, see caveats.)

3. The latent-prediction line: JEPA / V-JEPA 2

JEPA (Joint-Embedding Predictive Architecture) is the deliberate rejection of pixel prediction. I-JEPA's recipe:

• A context encoder sees a masked/partial view of an image.
• A target encoder (a slow EMA copy of the context encoder — no gradient flows into it) encodes the full target blocks into representations.
• A predictor maps context-representation → predicted target-representations.
• Loss: match predicted vs. actual target embeddings (an L2/regression loss in latent space), never reconstructing pixels.

z_ctx   = f_θ(context)                 # online encoder
z_tgt   = f_ξ(target)   ; ξ ← EMA(θ)    # target encoder, stop-grad
ẑ_tgt   = predictor(z_ctx, positions)  # predict the target's REPRESENTATION
loss    = || ẑ_tgt − stopgrad(z_tgt) ||²

Why latent, not pixels? Because a generative decoder is forced to predict unpredictable detail (every blade of grass, every reflection), spending capacity on noise. Predicting representations lets the model say "I don't know the exact texture, but semantically a tree is here" — the abstraction is the point. Collapse (all embeddings → constant) is the obvious failure mode; the EMA target + stop-gradient asymmetry (no contrastive negatives, no pixel loss) is what holds it apart.

V-JEPA 2 (Meta, Jun 2025) scales this to video as an actual world model: a 1.2B-parameter encoder + predictor self-supervised on >1 million hours of video (predict masked spatiotemporal latents). Then the action-conditioned twist — V-JEPA 2-AC: freeze the pretrained encoder and train a lightweight action-conditioned predictor on just 62 hours of unlabelled robot teleoperation data. Now the predictor answers "if the arm does this, what's the next latent?" — exactly the action-conditioned form. Planning is model-predictive control in latent space: imagine candidate action sequences forward, score each against an image-specified goal latent, execute the best. Result: zero-shot pick-and-place on a real Franka arm in unseen labs, ~65–80% on short-horizon goals — with no reward, no demonstrations of the task, no environment-specific training. Meta also shipped physical-reasoning benchmarks (IntPhys 2, MVPBench, CausalVQA) to measure whether these models actually "get" physics.

The connection to diffusion and to multimodal LLMs

• The generative camp's quality engine is diffusion — specifically diffusion transformers on spacetime patches. OpenAI's Sora report frames video generation itself as a path to "general-purpose simulators of the physical world": compress video into a spacetime latent, chop it into patch-tokens, and run a [[diffusion-models|diffusion transformer]] over them (the same patch-as-token move that powers [[multimodal-llms]]). At scale, the report claims emergent simulation — rough 3D consistency, object permanence, even controlling a Minecraft-like agent — purely from the generative objective, with nobody hard-coding physics.
• Genie's tokenize-then-sequence-model design is the multimodal-LLM pattern (turn pixels into discrete tokens, model them with a transformer) pointed at interactive video instead of captioning. If you understand how a VLM turns an image into tokens ([[multimodal-llms]]), Genie's tokenizer is the same idea with a temporal axis and an action channel bolted on.

Key ideas & tradeoffs

Other load-bearing ideas:

• The latent-action bottleneck (Genie's 8 codes) is the single most elegant trick here: forcing controllability to emerge unsupervised by starving the action vocabulary. Too many codes → it memorises frames and learns nothing controllable; too few → it can't explain the dynamics. The bottleneck is the inductive bias.
• Imagination horizon vs. compounding error. Every world model's predictions feed back into themselves, so errors compound. Short rollouts (Dreamer) stay accurate but plan myopically; long rollouts (minutes of Genie 3) drift, hallucinate, and forget. The "consistency horizon" is the field's central practical metric.
• Self-supervision is the unlock. None of these need action labels at scale: the next frame is the label (generative), or the next latent is the label (JEPA), or discovered latent actions stand in for real ones (Genie). This is why internet video — millions of hours — is the fuel.

Honest caveats & open questions

This section is the point of the article. Mid-2026, here is what is genuinely unsettled — stated plainly:

1. "Video generators are world models" is a claim, not a fact. A model can render a glass not shattering, or a hand passing through a solid object, while looking photorealistic. OpenAI's own Sora report concedes failures to model basic physical interactions and incoherent long-horizon dynamics. Fidelity ≠ a correct causal model of physics. The JEPA camp's entire thesis is that pixel-perfect rendering and understanding are different things — and the physics-violation benchmarks (IntPhys 2, CausalVQA) exist precisely because we don't yet have a clean way to tell whether a model "gets" physics or just paints convincing surfaces.

1. Consistency is short and brittle. Genie 3's worlds hold for "a few minutes" with ~1-minute visual memory; DeepMind lists limited action spaces, poor multi-agent interaction, geographic inaccuracy, and bad text rendering as open limitations. We do not have world models that maintain a coherent, persistent, physically-lawful world over long horizons. Compounding error is unsolved.

1. Closed weights, company claims. Genie 1/3 and Sora are not openly released; much of what's "known" is from blogs and demos, which are curated. Independent reproduction is thin (the strongest open Genie reimplementations are community efforts, not validated at the 11B/internet-scale of the original). Treat headline capabilities as vendor claims pending independent verification.

1. Which camp wins is undecided. Generative-video has the fidelity and the buzz; latent-predictive has the sample-efficiency and the real-robot zero-shot results. It's entirely possible the answer is hybrid (abstract planning latent + a generative decoder for grounding), and the 2026 surveys explicitly frame the field as these "two paths" still racing, not converged.

1. The path to AGI / embodied planning is a research bet, not a roadmap. "World models are the missing piece for grounded intelligence" is a hypothesis (LeCun's, among others). Real robot results exist but are short-horizon and narrow (pick-and-place, ~65–80%). Extrapolating from "65% pick-and-place" to "general embodied agency" is exactly the kind of leap that has burned the field before. Be excited; don't confuse the demo with the destination.

1. Evaluation is immature. We barely know how to measure whether a world model is "good." Pixel-FID rewards fidelity, not physics; downstream task success conflates the world model with the policy. The honest state: the benchmarks are younger than the models.

How it connects to OpenAlice

OpenAlice's Alice is a deployed agentic system — an LLM in a perceive→decide→act loop ([[agentic-loops]]), with memory, tools, and a streaming/embodied avatar surface. World models intersect that work at several concrete seams, and it's worth being precise about which are real today vs. aspirational:

• Planning by imagined rollout is the same idea as agentic look-ahead. Alice's loop already chooses tool-calls and actions; a world model is the principled, learned version of "simulate the consequence before committing." The conceptual bridge to [[test-time-compute-reasoning]] is direct — both spend extra compute imagining/searching candidate futures before acting. Today Alice plans in language/tool space, not a learned latent simulator; a learned world model would be a substantial future addition, not a current capability.
• Action-conditioned prediction = the structure of an action-and-environment substrate. If Alice ever drives an embodied or richly-simulated environment (the streaming/world-rendering surface is the obvious candidate), the predict(state, action) → next_state contract is exactly the interface a world model provides, and exactly what would let Alice try before it acts.
• The generative camp leans on machinery Alice's stack already touches. Genie/Sora are built from [[diffusion-models]] and the tokenize-pixels-into-a-sequence pattern of [[multimodal-llms]] — both already in this library and adjacent to OpenAlice's multimodal ambitions.
• The honesty discipline transfers. Just as OpenAlice insists agent behaviour be measured (not assumed) via [[agent-evaluation]] and reward signals like [[rlvr]], world models demand the same skepticism: a beautiful rollout that's physically wrong is the multimodal version of a confident hallucination. The lesson is identical — fidelity is not correctness; measure the thing you actually care about.

If you're new here, read [[diffusion-models]] and [[multimodal-llms]] first (they're the engine and the I/O of the generative camp), then come back: world models are what you get when you point those tools at interactive, action-conditioned prediction instead of one-shot generation.