Multimodal LLMs (Vision-Language Models)

One sentence: A multimodal LLM is a language model that has been taught to read pictures (and sometimes audio/video) by turning those pixels into the same kind of token-vectors the model already understands — so that "a photo of a cat" and the word "cat" end up living in the same representational space and the transformer can reason across both.

If you understand [[attention-and-transformers]] and [[embeddings]], you are 80% of the way to understanding a VLM. The remaining 20% is one question: how do you get a continuous grid of pixels into a transformer that was built to consume a sequence of token-vectors? This page answers that question precisely, from the intuition down to the projection equation.

What it is (intuition first)

A normal LLM eats a sequence of token-vectors. Text gets there easily: [[tokenization]] chops the string into subwords, and an embedding table looks each one up as a vector (see [[embeddings]]). The transformer then mixes those vectors with [[attention-and-transformers|self-attention]] and predicts the next token.

An image is not a sequence of symbols. It is a continuous 2D grid of pixel intensities — there is no "vocabulary of pixels" to look up. So the entire game of multimodal LLMs is building a bridge from "grid of pixels" to "sequence of vectors the transformer accepts." There are two fundamentally different ways to build that bridge, and almost every VLM is one of these two shapes:

1. The adapter approach (most common in 2023–2025). Keep a powerful, separately-trained vision encoder (usually a Vision Transformer, ViT↗, often the one inside CLIP↗). It turns the image into a handful of dense feature vectors. Then bolt a small connector onto a frozen text LLM that translates those vision vectors into "fake word embeddings" the LLM can read. This is the LLaVA↗ recipe. The image becomes a few hundred soft visual tokens that you literally splice into the prompt, as if you'd typed them.

1. The native / early-fusion approach (Chameleon, GPT-4o-style). Don't bolt anything on. Instead, tokenize the pixels into discrete tokens with a learned image codebook (like a VQ-VAE), extend the model's vocabulary to include those image tokens, and train one transformer from scratch on an interleaved soup of text-tokens and image-tokens. Now "image" and "text" are not two systems glued together — they are two regions of a single token vocabulary. This is Chameleon↗.

A useful mental image: in approach (1) the LLM is a monolingual person with a real-time interpreter sitting next to them (the connector) whispering translations of what the picture says. In approach (2) the model is genuinely bilingual from birth — it never separates "seeing" from "reading."

Both approaches end up at the same place that makes the whole field work: a shared representation space where a picture of a dog and the word "dog" are near each other. That alignment is the load-bearing miracle, and it's worth understanding where it comes from first.

Why it matters

• It's how the model "sees." Every product that lets you upload a screenshot, a chart, a meme, a whiteboard photo, or a PDF page and ask questions about it is a VLM. Document understanding, UI agents, OCR-free reading, accessibility ("describe this image"), and visual reasoning all live here.
• Language became the universal supervision signal. CLIP↗'s deep insight was that you don't need hand-labeled categories — the caption is the label. Train on 400 million noisy (image, text) pairs scraped from the web and you get a vision model that does zero-shot classification: name any new category in plain English and it can recognize it, no fine-tuning. That broke the decades-old "fixed list of 1000 ImageNet classes" ceiling.
• It's the substrate for agents that act on the world. A coding/computer-use agent that has to read a screen, a robot that has to look at a scene — these need vision fused into reasoning, not a separate captioning microservice.
• It connects directly to OpenAlice's roadmap (screen-reading agents, the streaming-avatar's awareness of what's on screen) — see the last section.

How it works (real mechanics)

We'll build it in four layers: (A) how a transformer eats an image at all (ViT), (B) how vision and text get aligned (CLIP), (C) how you graft vision onto an existing LLM (LLaVA + connectors), and (D) the native alternative (Chameleon).

A. ViT — turning a picture into a sequence of tokens

A Vision Transformer↗ does the obvious thing: chop the image into a grid of fixed-size square patches and treat each patch as a "word."

Given an image of height H, width W, channels C, and patch size P (commonly 14 or 16):

number of patches:   N = (H · W) / P²

For a 224×224 image with P=16 that's N = 196 patches (a 14×14 grid). Each patch is P·P·C raw numbers (e.g. 16·16·3 = 768). You then:

1. Flatten each patch into a vector and linearly project it to the model dimension d: x_patch = W_e · flatten(patch) (W_e is a learned d × (P²·C) matrix — essentially a strided conv).
2. Prepend a learnable `[CLS]` token whose final-layer output serves as the whole-image summary.
3. Add learnable position embeddings so the model knows where each patch sat (a transformer is permutation-blind without them — see [[positional-encoding]]). ViT uses learned 1D position embeddings rather than RoPE.
4. Feed the resulting sequence of N+1 vectors through a standard transformer encoder — exactly the [[attention-and-transformers|self-attention]] stack you already know, no convolutions.

# ViT patch embedding, schematically
patches = image.unfold(P)                  # -> (N, P*P*C)
tokens  = patches @ W_e.T                   # -> (N, d)   linear projection
tokens  = concat([cls_token, tokens])       # -> (N+1, d)
tokens  = tokens + pos_embed                # add learned positions
features = transformer_encoder(tokens)      # -> (N+1, d)

The key payoff: a ViT outputs a sequence of `d`-dimensional patch features — structurally identical to what an LLM consumes. That's why fusing vision into a transformer is even possible. The price (honest caveat): ViT is data-hungry. It lacks a CNN's built-in locality/translation priors, so it underperforms ResNets on small datasets and only wins at scale (large pretraining sets).

B. CLIP — aligning images and text into one space

CLIP↗ is a dual-encoder: an image encoder f_I (a ViT or ResNet) and a text encoder f_T (a transformer). Each produces a vector; both are projected into a shared embedding space and L2-normalized. The whole thing is trained with a contrastive objective (InfoNCE).

Take a batch of N matched (image_i, text_i) pairs. Compute all N×N cosine similarities between every image and every text. The diagonal (image i with its own caption i) should be high; every off-diagonal (image i with someone else's caption) should be low. Concretely, with image embeddings I and text embeddings T (rows normalized) and a learned temperature τ:

logits = (I @ T.T) * exp(τ)            # N×N similarity matrix, scaled
labels = [0, 1, 2, ..., N-1]           # correct match for each row is the diagonal
loss   = ( cross_entropy(logits, labels, axis=rows)      # image->text
         + cross_entropy(logits, labels, axis=cols) ) / 2 # text->image (symmetric)

This is symmetric InfoNCE: each image is pushed toward its caption and away from the other N−1 captions in the batch, and vice-versa. Trained on ~400M web pairs, the result is a space where embed("a photo of a Shiba Inu") lands right next to embeddings of actual Shiba Inu photos.

Zero-shot classification then falls out for free: to classify an image into classes {cat, dog, car}, embed the strings "a photo of a cat", "a photo of a dog", "a photo of a car", embed the image, and pick the text with the highest cosine similarity. No classifier head, no fine-tuning. CLIP matches ResNet-50's ImageNet accuracy using none of ImageNet's 1.28M labeled training images. This is the same geometry-encodes-meaning principle from [[embeddings]], now spanning two modalities. CLIP's ViT image encoder is the workhorse that most adapter-style VLMs reuse.

C. LLaVA — grafting vision onto a frozen LLM (the connector recipe)

LLaVA↗ is the canonical "adapter" VLM and the most copied design. The pieces:

• Vision encoder: frozen CLIP ViT-L/14. Run the image through it and grab the grid of patch features (not just [CLS]): Z_v = g(X_v). For a 336px image at P=14 that's a 24×24 grid → 576 visual feature vectors.
• Connector (the bridge): a trainable projection that maps each vision feature into the LLM's word-embedding space. In the original LLaVA it's a single linear matrix; in LLaVA-1.5 it's a 2-layer MLP:

  H_v = W · Z_v          (with  Z_v = g(X_v))

W is d_llm × d_vision. After projection, each of the 576 patch features is now a d_llm-dimensional vector — a soft visual token, indistinguishable in shape from a real word embedding.

• LLM: Vicuna (a LLaMA fine-tune). You build the prompt by literally interleaving the 576 visual tokens with the text-token embeddings (<image_tokens> What is in this photo?) and run the LLM as normal. Attention does the rest: text tokens attend to visual tokens and pull in what they need — exactly the [[attention-and-transformers|Query/Key/Value]] mechanic, now reaching across modalities.

Two-stage training is the other half of the recipe:

1. Stage 1 — feature alignment (pretraining). Freeze both the vision encoder and the LLM. Train only `W` on ~595K image-caption pairs. The paper frames this beautifully as "training a compatible visual tokenizer" for the frozen LLM — you're just teaching the connector to speak the LLM's embedding dialect.
2. Stage 2 — visual instruction tuning. Unfreeze the LLM (and W); keep the vision encoder frozen. Fine-tune end-to-end on ~158K instruction-following examples (conversation, detailed description, complex reasoning). The clever trick: this instruction data was generated by text-only GPT-4 fed image captions + bounding boxes — bootstrapping multimodal training data from a model that couldn't even see.

# LLaVA forward pass, schematically
Z_v   = clip_vit(image)            # (576, d_vision)   frozen
H_v   = mlp_projector(Z_v)         # (576, d_llm)      the connector
H_txt = embed(text_tokens)         # (T,   d_llm)
seq   = interleave(H_v, H_txt)     # one sequence the LLM consumes
out   = llm(seq)                   # autoregressive generation

Other connector designs you'll meet (all solving the same "vision-features → LLM-tokens" problem):

• Q-Former (BLIP-2↗): a small transformer with a fixed set of learnable query vectors that cross-attend into the image features and distill them down to a fixed, small number of tokens (e.g. 32). Pro: controls token count tightly; lets you keep both encoder and LLM frozen with few trainable params. Con: harder to train, and empirically the LLaVA-1.5 study found a 2-layer MLP beats the Q-Former for alignment quality.
• Perceiver Resampler + gated cross-attention (Flamingo↗): instead of splicing visual tokens into the sequence, insert new cross-attention layers into the LLM that attend to image features, gated so the pretrained LLM isn't disrupted. This is what makes Flamingo handle interleaved image-text and few-shot prompting.

D. Chameleon — the native / early-fusion alternative

Chameleon↗ throws out the "separate vision encoder + connector" idea entirely. Instead it tokenizes pixels into discrete tokens, the same way text is tokenized:

• A learned image tokenizer (VQ-style) encodes a 512×512 image into 1024 discrete tokens drawn from a codebook of 8192 entries. Now an image is a sequence of integer token-IDs, exactly like text.
• The model's vocabulary is the union of text tokens and image tokens. A single transformer is trained from scratch on interleaved mixed-modal sequences — text, then image tokens, then more text — with one ordinary next-token objective over the combined vocabulary. This is early fusion: modalities are merged at the input layer, not bolted together late.
• Because it generates both text tokens and image tokens autoregressively, the same model can read and draw — interleaved image-and-text generation from one set of weights.

The catch is stability: training a from-scratch early-fusion model is notoriously unstable (text and image tokens have very different statistics, causing divergence). Chameleon's contribution is the engineering to fix it — query-key normalization (QK-norm) and careful normalization placement to keep logits from exploding at scale. Reported result: it matches or exceeds much larger models (Gemini Pro, GPT-4V) on long-form mixed-modal generation by human eval, while staying competitive on text-only.

Early vs late fusion in one line: late fusion (LLaVA/CLIP-adapter) reuses strong pretrained unimodal models and trains a small bridge — cheap, modular, but the LLM only ever sees vision through a frozen lens. Early fusion (Chameleon, and the spirit behind GPT-4o / Gemini's "natively multimodal" claims) trains one model on everything at once — maximal cross-modal flexibility (including generating images/audio) at much higher training cost and difficulty.

Key ideas & tradeoffs

Other tradeoffs worth holding in your head:

• Visual tokens are expensive. Each visual token costs the same attention as a word token, and attention is O(n²). LLaVA's 576 tokens per image can dwarf the text. High-res / multi-image / video makes this brutal. Hence token-count compression: BLIP-2's Q-Former caps it at ~32; Qwen2.5-VL concatenates groups of 4 ViT tokens before the MLP to quarter the count. There's a real accuracy↔cost frontier here.
• Frozen encoder = information bottleneck. If CLIP's encoder threw away the small text in an image, no amount of LLM cleverness recovers it. Adapter VLMs inherit their vision encoder's blind spots (fine-grained OCR, counting, precise spatial layout). Research like "Lost in Embeddings" quantifies this information loss at the connector.
• Resolution matters more than people expect. A 224/336px encoder literally cannot resolve dense document text. Modern VLMs do tiling / "AnyRes" (slice a big image into sub-images, encode each, concatenate) — which again multiplies token count.
• MLP beat the clever connector. A recurring lesson: the dumb 2-layer MLP projector (LLaVA-1.5) outperformed the fancy Q-Former for alignment. Simplicity won; the Q-Former survives mainly where you must cap token count.

Honest caveats & open questions

• VLMs hallucinate about images, confidently. They'll describe objects that aren't there ("object hallucination") because the language prior is strong and the visual grounding is weak. This is worse than text-only hallucination because users assume "it can see, so it must be right."
• Counting, reading dense text, precise spatial/geometric reasoning, and reading clocks/charts are still shaky. These are exactly the tasks where the frozen-encoder bottleneck and patch-grid coarseness bite hardest.
• "Natively multimodal" is partly marketing. GPT-4o and Gemini are described as native/early-fusion, but the exact architectures are closed. We genuinely don't know how much is true early fusion vs. sophisticated adapters. Treat public claims with skepticism; the open reference point for native any-to-any is Chameleon.
• Evaluation is immature. Many VLM benchmarks can be partly solved from the language prior alone (answer without looking). Strong leaderboard numbers don't always mean strong seeing.
• Audio and video are far less mature than images. Audio-in (Whisper-style encoders feeding tokens) and true video understanding (temporal modeling, not just sampled frames) remain open. Most "video" VLMs just sample a few frames and treat them as images.
• No CLIP-style alignment guarantee inside the LLM. CLIP explicitly aligns image/text with a contrastive loss. An adapter VLM only aligns implicitly through next-token loss — there's no proof the visual tokens occupy a clean shared space, which may underlie some failure modes.

How it connects to OpenAlice

• Same engine, two modalities. Everything here rides on [[attention-and-transformers]] and [[embeddings]] — the two pages to read first. A VLM is not a new kind of model; it's a new kind of input to a model you already understand. Visual tokens are just embeddings that happen to come from pixels, mixed in by the same self-attention.
• Tokenization symmetry. [[tokenization]] explains how text becomes tokens; Chameleon's image tokenizer is the visual analog (discrete codebook). The "tokenize everything, then run one transformer" philosophy is the unifying thread.
• Position embeddings, generalized. [[positional-encoding]] is what tells a ViT where each patch sat — the 2D-position problem is the multimodal version of the 1D sequence-position problem.
• Screen-reading & avatar awareness. Alice's agentic and streaming work (computer-use / UI agents that read a screen, an on-stream avatar aware of what's displayed) is a VLM application: an adapter that turns a screenshot into visual tokens spliced into the agent's prompt is the natural first build — cheap, modular, reuses a frozen CLIP-ViT — before considering anything native.
• Cost discipline carries over. The visual-token O(n²) cost ties straight into [[llm-inference-internals]] and [[flash-attention]]: every image you feed inflates the KV cache and attention budget. Token-count compression (Q-Former, group-merge MLP) is the multimodal cousin of the inference-efficiency work the lab already cares about.
• For deeper foundations of the contrastive idea behind CLIP, see [[embeddings]] (cosine similarity, InfoNCE-style objectives) — CLIP is "embeddings, but across two modalities at once."