Video Generation

What it is (intuition first)

Modern AI video generation is denoising done in slow motion across both space and time. You start from a block of pure random static — not a flat image, but a 3D loaf of noise: width × height × time. The model then sharpens that loaf, step by step, until it resolves into a coherent moving scene that matches a text prompt ("a corgi skateboarding through Tokyo at dusk, neon reflections on wet pavement").

If you already understand how a still-image diffusion model paints a picture out of noise (see [[diffusion-models]]), video is the same idea with one extra axis: time. The hard part is not making one good frame — it's making hundreds of frames that are individually sharp and consistent with each other: the corgi has the same fur, the same number of legs, the skateboard doesn't teleport, and the camera moves smoothly instead of jump-cutting.

The big conceptual leap that made this work at scale (Sora, CogVideoX, Movie Gen, and successors) is to stop thinking of a video as "a stack of images" and instead chop it into spacetime patches — small 3D cubes of video — and treat each cube as a token, exactly like a word-piece in a language model. Once a video is "just a sequence of tokens," you can throw a Transformer at it and let the scaling laws that powered LLMs do their work. That reframing is why people call these systems world simulators: a model that learns to predict consistent pixels over time has implicitly had to learn something about how objects, light, and motion behave.

Why it matters

It's the multimodal frontier. Text and images are largely "solved" at the foundation-model level; video is where generative modeling is currently being stress-tested, because temporal consistency exposes whether a model has any real grasp of the world or is just memorizing textures.
The world-simulator hypothesis. If a single model can render a physically plausible, controllable, infinitely-long video, you have the substrate for game engines, robotics simulators, planning agents, and embodied AI — a learned, queryable physics engine. This is the bridge from generative video to [[world-models]].
It validated the "everything is tokens + Transformer" thesis once more. DiT showed Transformers beat U-Nets for diffusion; Sora showed the same recipe scales to video. The architectural convergence with [[attention-and-transformers]] and [[multimodal-llms]] is the story.
Economic gravity. Film, advertising, game asset creation, and synthetic training data are enormous markets, which is why every major lab shipped a flagship video model.

How it works (real mechanics)

There are four moving parts in essentially every modern text-to-video system. We'll build them up.

1. Compress the video into a latent space (the spacetime VAE)

Generating in raw pixels is hopelessly expensive: a 5-second 720p clip is billions of numbers. So step one is a video autoencoder that compresses the clip into a much smaller latent tensor, runs all the heavy generation there, and decodes back to pixels at the end. This is the video analogue of latent diffusion (Stable Diffusion's trick), extended to time.

CogVideoX uses a 3D causal VAE that compresses both spatially and temporally. "Causal" means the temporal convolutions only look at past frames, so the encoder can be applied to variable-length / streaming video without leaking future information.
Movie Gen uses a Temporal Autoencoder (TAE) that compresses 8× along each of time, height, and width (so T/8 × H/8 × W/8 — a ~512× reduction in element count). They build it by "inflating" a 2D image autoencoder: after each 2D spatial conv they add a 1D temporal conv, and after each spatial attention a 1D temporal attention. This "inflation" trick lets you bootstrap a video model from a pretrained image model.

The compressed output is a 4D latent z of shape roughly [C, T', H', W'] (channels × compressed-time × compressed-height × compressed-width).

2. Patchify: turn the latent into a sequence of spacetime tokens

Now cut the latent loaf into small 3D cubes and flatten each into a vector — a spacetime patch. This is the move that makes a Transformer applicable. The Sora review (arXiv:2402.17177) describes two flavors:

Spatial-patch compression — patch each frame like a Vision Transformer (ViT), naturally supporting arbitrary resolutions/aspect ratios.
Spatial-temporal-patch compression — use 3D convolutions so each patch spans both a little region and a few frames, capturing motion directly.

Because clips have different durations and shapes, the patches are arranged into one long sequence and likely packed with a patch-n-pack (PNP)-style scheme — variable-length sequences batched together. This is precisely why Sora can train on, and generate, native resolutions, durations, and aspect ratios without forced cropping/resizing — a notable departure from older fixed-size pipelines.

# Conceptual patchify of a video latent  z: [C, T', H', W']
# patch sizes pt, ph, pw along time/height/width
def patchify(z, pt, ph, pw):
    tokens = []
    for t in range(0, z.T, pt):
        for h in range(0, z.H, ph):
            for w in range(0, z.W, pw):
                cube = z[:, t:t+pt, h:h+ph, w:w+pw]   # a spacetime patch
                tokens.append(flatten(cube))          # -> one token vector
    return stack(tokens)   # sequence the Transformer will denoise
# Each token also carries a 3D positional encoding (t, h, w);  see [[positional-encoding]]

3. The Diffusion Transformer (DiT) backbone

This is the engine. DiT (Peebles & Xie, arXiv:2212.09748) replaced the U-Net used by earlier diffusion models with a plain Transformer operating on latent patches. Its headline empirical result is a clean scaling law: as you increase forward-pass compute (Gflops) — deeper/wider model or more tokens — image quality improves monotonically (FID drops). DiT-XL/2 hit a then-SOTA FID of 2.27 on class-conditional ImageNet 256×256. Video models (Sora, CogVideoX, Movie Gen) are this same DiT, scaled up and fed spacetime tokens.

Conditioning (the timestep t and the text/class signal c) is injected through adaptive LayerNorm, specifically DiT's adaLN-zero: instead of adding conditioning as tokens, a small MLP maps (t, c) to per-block scale/shift/gate parameters that modulate each Transformer block's normalization. The "zero" means the residual gate is initialized to zero, so each block starts as the identity — a stability trick that helps deep diffusion Transformers train. CogVideoX uses an expert adaptive LayerNorm to deeply fuse the text and video modalities, and 3D full attention so every spacetime token can attend to every other (non-local motion and long-range coherence, at quadratic cost — the [[long-context]] / [[flash-attention]] tension is very real here).

The denoiser is trained either with the standard diffusion noise-prediction objective or with flow matching (Movie Gen): instead of predicting the noise, the network predicts the velocity of the sample along a straight-ish path from noise to data. Sampling then integrates that velocity field.

# Training step (flow-matching / rectified-flow style, as in Movie Gen)
z1   = tae.encode(video)                 # clean latent
z0   = randn_like(z1)                    # pure noise
tau  = rand()                            # interpolation time in [0,1]
zt   = (1 - tau) * z0 + tau * z1         # point on the noise->data path
v_target = z1 - z0                       # the straight-line velocity
v_pred   = DiT(patchify(zt), t=tau, cond=text_emb)
loss = mse(v_pred, v_target)             # learn the velocity field

# Sampling: start at noise, integrate the learned velocity (ODE solve)
z = randn(...)
for tau in linspace(0, 1, num_steps):
    z = z + (1/num_steps) * DiT(patchify(z), t=tau, cond=text_emb)
video = tae.decode(z)

4. Recaptioning: the data quality multiplier

A quieter but decisive trick (borrowed from DALL·E 3): web video captions are short and bad ("cool clip 😎"). So labs train a dedicated video captioner, run it over the whole training set to produce dense, detailed descriptions, and train the generator on those. Prompt-following is largely a function of caption quality, not just model size. At inference, a short user prompt is often expanded by an LLM into the long, detailed style the model was trained on.

Putting it together

text prompt ──► (LLM expand) ──► text encoder ──► cond c
                                                     │
video ──► [TAE/3D-VAE encode] ──► latent z ──► patchify ──► spacetime tokens
                                                     │
                          DiT (Transformer + adaLN/flow-matching), conditioned on (t, c)
                                                     │
                                       denoised latent ──► [VAE decode] ──► video

Key ideas & tradeoffs

Patches unify modalities. Spacetime patches are to video what tokens are to text and what ViT patches are to images. This is the idea that let video ride the Transformer scaling curve, and it ties video directly to [[multimodal-llms]] and [[tokenization]] (the philosophy, not BPE specifically).
Latent vs. pixel space. Generating in a compressed latent (8× per axis) is the only way this is affordable, but the VAE is a quality bottleneck — high-frequency detail and fine motion can be lost or smeared by the decoder.
Global 3D attention vs. cost. Full 3D attention gives the best temporal coherence (Sora-class consistency) but is quadratic in the number of spacetime tokens, and token counts explode with duration/resolution (Movie Gen trains at up to 73K tokens). This is the central scaling pain; it motivates factorized/windowed attention, [[state-space-models]] hybrids, and autoregressive next-frame schemes.
Diffusion vs. flow matching. Flow matching (predict velocity along straight paths) tends to give simpler, often faster sampling and stable training; classic diffusion noise-prediction is the well-trodden baseline. Both share the DiT backbone.
Native-resolution training. Training on raw aspect ratios/durations (Sora) improves composition and flexibility but complicates batching (needs PNP-style packing).
Emergence from scale. Sora's "3D consistency, object permanence, simulating Minecraft" were not explicitly programmed — they appeared as the DiT was scaled, echoing the emergence story of LLMs ([[scaling-laws]]). This is the evidential core of the world-simulator claim.

Honest caveats

The physics is faked, and it shows. These models do not contain a physics engine; they learn statistical regularities of pixels. The Sora review documents concrete failures: no bite marks after a cookie is eaten, objects morphing unnaturally, broken cause-and-effect, left/right confusion, characters and props appearing from nowhere. "World simulator" is an aspiration and a research framing, not a verified property. Treat it as a hypothesis under test, which is exactly how [[world-models]] frames it.
Length and drift. Flagship public numbers are short: CogVideoX ~10s, Movie Gen ~16s at 16fps. Long video is genuinely unsolved — error accumulates frame-over-frame ("drift"), identity and scene state degrade, and quadratic attention makes naive extension unaffordable. Tricks (autoregressive rollout, sliding windows, explicit memory/3D) help but don't fully fix consistency over minutes.
Compute is brutal. Training is multi-thousand-GPU; inference for a few seconds of HD can take many seconds-to-minutes of GPU time. None of this runs on a CPU server — relevant context for OpenAlice's GPU-less infrastructure (see below).
Benchmarks are weak and gameable. Video quality is judged largely by human preference plus shaky automatic metrics (FVD and friends). Cross-paper "wins" should be read with the same skepticism as any leaderboard — see [[llm-evaluation]] and [[benchmark-contamination]] for why.
Reverse-engineered details. Sora has no published paper — the architecture above is reconstructed by the community (arXiv:2402.17177) from the technical report, demos, and analogous open models (DiT, CogVideoX, Movie Gen). Specifics (exact patch sizes, VAE design) for the closed models are educated inference, not ground truth.
The data and consent problem. These models are trained on enormous, largely undisclosed video corpora, raising unresolved copyright, likeness, and consent issues — and they are potent deepfake engines.

How it connects to OpenAlice + the Academy ladder

OpenAlice's stance. OpenAlice runs on a GPU-less Hetzner server, so it does not train or self-host diffusion video models — generating video is fundamentally a heavy-GPU diffusion workload, the opposite of Alice's CPU/Rust + hosted-LLM substrate. Where video shows up in the ecosystem it is via hosted generation (and the project's AI-media path historically leans on hosted services — e.g. Lyria for music). The value of this article is therefore conceptual literacy, not an implementation guide: it lets Alice and the agents reason about video models, route requests sensibly, and understand the world-simulator direction that adjacent products (streaming personas, world-modeling) gesture at.

Where it plugs into the library. Video generation is the capstone that makes the diffusion and multimodal strands concrete:

Prereqs: [[diffusion-models]] (the denoising core), [[attention-and-transformers]] (the DiT backbone), [[positional-encoding]] (3D spacetime positions), [[multimodal-llms]] (text↔visual conditioning, captioning).
Siblings / contrasts: [[diffusion-language-models]] (the same iterative-denoising idea applied to text instead of pixels — a clean cross-modal mirror), [[world-models]] (the simulator framing and where this is headed), [[scaling-laws]] (why scale produced emergence).
Tensions it surfaces: [[long-context]] and [[flash-attention]] (the quadratic-attention wall for long clips), [[state-space-models]] (a candidate escape route for sub-quadratic temporal modeling), [[llm-evaluation]] / [[benchmark-contamination]] (how not to trust the leaderboards).

Academy ladder placement. This sits near the top of the generative-modeling rung. A learner climbs: (1) what a [[diffusion-models|diffusion model]] is → (2) how [[attention-and-transformers|Transformers]] replaced U-Nets via DiT → (3) how adding a time axis + spacetime patches yields video → (4) the open research frontier of [[world-models|world models]]. Reaching this article should leave a beginner able to read the Sora report critically: to see why the architecture works, where the "world simulator" framing is real versus marketing, and what (length, consistency, physics) remains genuinely unsolved.