Embeddings & Vector Spaces

One sentence: An embedding turns a discrete thing (a word, a sentence, an image, a user) into a list of numbers — a vector — positioned in a high-dimensional space so that things with similar meaning land near each other.

This is one of the most load-bearing ideas in modern ML. Almost everything downstream — transformers, RAG, semantic search, recommendation, clustering — is built on top of a vector space where geometry encodes meaning. If you understand embeddings, a huge amount of the field stops looking like magic.

What it is (intuition first)

Computers can't do arithmetic on the word "king." So the first thing every language model does is look the word up in a table and replace it with a vector of numbers, e.g. a 768-dimensional list like [0.12, -0.94, 0.31, ...]. That vector is the word's embedding.

The trick is where the vector sits. We don't choose the numbers by hand — we learn them so that the layout of the space is meaningful:

• Nearness = similarity. "cat" and "dog" land close together; "cat" and "democracy" land far apart.
• Directions = relationships. Famously, in good word embeddings the direction from "man" to "woman" is roughly the same as the direction from "king" to "queen." So you can do arithmetic on meaning: vec("king") − vec("man") + vec("woman") ≈ vec("queen") This "analogy arithmetic" emerged for free from word2vec (Mikolov et al. 2013↗) and stunned the field — nobody told the model what a king was; the geometry just fell out of predicting which words appear near which.

A useful mental picture: imagine a vast map where every concept is a pin. The map has hundreds of dimensions instead of two, but the principle is the same as a real map — distance and direction carry information. An embedding model is the cartographer that decides where every pin goes.

The discrete → continuous jump is the whole point. Words are symbols with no inherent order or distance ("apple" isn't "between" cat and dog). Embeddings replace that symbolic chaos with a smooth, continuous space where "close" and "in the same direction" are well-defined and differentiable — which is exactly what gradient-based learning needs.

Why it matters

1. It's the input layer of every LLM. Before a transformer can do anything, [[tokenization]] splits text into tokens and the embedding layer maps each token ID to a vector. No embeddings, no transformer. (See [[attention-and-transformers]] for what happens next.)
2. It makes meaning computable. "How similar are these two documents?" becomes "what's the cosine of the angle between their vectors?" — a single cheap arithmetic operation instead of fragile keyword matching.
3. It powers retrieval / RAG. Modern retrieval ([[graphrag]], standard RAG) embeds your query and every document into the same space, then finds the nearest documents by vector similarity. This is semantic search — it finds "car" when you asked about "automobile," because meaning, not spelling, decides nearness.
4. It's modality-agnostic. The same idea works for images (CLIP), audio, code, graph nodes, and users. Once everything is a vector in a shared space, you can compare across modalities — search images with text, recommend products from behavior, etc.
5. It compresses. A 50,000-word vocabulary as one-hot vectors is 50,000-dimensional and carries zero similarity structure. A 300-dim embedding is ~166× smaller and encodes relationships. Density + structure is the win.

How it works (the real mechanics)

0. The embedding matrix

The core object is a single big lookup table: the embedding matrix E of shape [V × d], where V is the vocabulary size and d is the embedding dimension (e.g. V = 50,000, d = 768).

• Row i of E is the embedding for token i.
• "Looking up" a token is just selecting its row — equivalently, multiplying a one-hot vector by E. (One-hot × matrix = "grab one row," which is why people say the embedding layer is "a matmul that's secretly an index.")
• E starts as random numbers and is learned by backpropagation, just like any other weight matrix. Nothing about the initial values means anything; meaning is carved into the matrix by training.

That's it. The whole "semantic space" lives in this one matrix of learned floats.

1. Static word embeddings — word2vec (predict-the-neighbors)

word2vec↗ (Mikolov et al. 2013) learns E from a simple self-supervised game over raw text, in two flavors:

• CBOW (Continuous Bag of Words): given the surrounding context words, predict the missing center word.
• Skip-gram: given the center word, predict each surrounding context word.

The driving principle is the distributional hypothesis: "a word is characterized by the company it keeps" (Firth, 1957). Words appearing in similar contexts get pushed toward similar vectors.

Skip-gram, conceptually, maximizes the probability of real context words given the center word:

maximize   Σ over (center, context) pairs:  log P(context | center)

with P(o | c) = softmax over the vocabulary of (u_o · v_c) — i.e. the score of a context word o given center c is the dot product of their two vectors. (word2vec keeps two matrices, one for "center" roles and one for "context" roles.) Because a full softmax over a 50k vocabulary is expensive, word2vec uses negative sampling: instead of normalizing over all words, push the real (center, context) pair's dot product up and a handful of random "negative" words' dot products down. This is an early cousin of the contrastive loss we'll see later.

Result: ~300-dim vectors with the famous analogy structure. Critically, word2vec is static — "bank" gets one vector whether you mean a river bank or a money bank. That's the central limitation contextual models later fix.

2. Static word embeddings — GloVe (factorize the co-occurrence stats)

GloVe↗ (Pennington, Socher & Manning 2014) reaches a similar place from the opposite direction. Instead of streaming local windows, it first builds a global word–word co-occurrence matrix X, where X_ij = how often word j appears near word i across the whole corpus.

Its key insight: it's not raw co-occurrence but ratios of co-occurrence probabilities that carry meaning. (The Stanford write-up's example: for the probe word "solid," P(solid|ice) / P(solid|steam) is large; for "gas" it's small — the ratio cleanly separates ice from steam where individual probabilities don't.) GloVe is "a log-bilinear model with a weighted least-squares objective," and it trains word vectors so that their dot product equals the log of co-occurrence count:

minimize   Σ_ij  f(X_ij) · ( w_i · w̃_j + b_i + b̃_j − log X_ij )²

where f(X_ij) is a weighting that down-weights very rare and very frequent pairs. Because log(ratio) = log a − log b becomes a vector difference, semantic ratios show up as directions — which is why GloVe also nails analogy tasks. word2vec ≈ implicitly factorizing a (shifted PMI) co-occurrence matrix, so the two approaches are deeply related even though one is "predictive" and the other "count-based."

3. Contextual embeddings — BERT and friends

The big shift: a token's vector should depend on its sentence, not just its identity. BERT↗ (Devlin et al. 2018) "pre-trains deep bidirectional representations by jointly conditioning on both left and right context in all layers." The same word now gets different vectors in different sentences — "bank" by a river ≠ "bank" with money.

How it learns: the Masked Language Model (MLM) objective. Randomly hide ~15% of tokens with a [MASK], then make the model predict them from both-sides context. To fill a blank correctly you must build a rich, context-aware representation of every token — so deep contextual embeddings fall out of the task. (BERT tokenizes with WordPiece subwords, so even unseen words decompose into known pieces — see [[tokenization]].)

Mechanically, the "embedding" of a token in a transformer is no longer a fixed table row — it's the hidden state at that position after the [[attention-and-transformers]] layers have let every token mix information with every other token. The static row from the embedding matrix is just the input; context-aware embeddings are the output. (And because attention is order-blind, the input also gets a [[positional-encoding]] added so word order survives.)

4. Sentence & retrieval embeddings — Sentence-BERT and the bi-encoder

Now you want one vector for a whole sentence or document (for search, clustering, RAG). The naive move — average BERT's token vectors, or use its [CLS] token — produces surprisingly bad similarity vectors. And feeding pairs of sentences through full BERT to compare them is brutal: Sentence-BERT↗ (Reimers & Gurevych 2019) notes that finding the most similar pair among 10,000 sentences would need ~50 million BERT inferences (~65 hours).

SBERT's fix: a siamese / bi-encoder — run BERT once per sentence independently, pool the token vectors (mean pooling works well) into a single fixed vector, and fine-tune two shared-weight copies so that cosine similarity between the two output vectors reflects semantic similarity. Now each sentence is encoded once into a vector you can store; comparison is just cosine. The 65-hour search drops to ~5 seconds. This "encode-everything-once, compare-by-cosine" pattern is the foundation of every modern vector database and RAG pipeline.

Modern training = contrastive. State-of-the-art retrieval embedders (Cohere embed, OpenAI text-embedding-3, BGE-M3, Qwen3-Embedding, Gemini Embedding) are trained with a contrastive InfoNCE objective (cross-checked across 2024–2025 retrieval literature↗): pull a query and its relevant ("positive") document together, push it away from many irrelevant ("negative") documents. For a query q, positive d⁺, and negatives d⁻:

loss = − log [  exp(sim(q, d⁺)/τ)
               ─────────────────────────────────── ]
              Σ over {d⁺, d⁻...}  exp(sim(q, d)/τ)

where sim is cosine similarity and τ is a temperature. "Negatives" are often in-batch (the other examples in the minibatch, free) plus hard negatives (plausible-but-wrong docs that force fine distinctions). This is the same shape as word2vec negative sampling — push real pairs up, random pairs down — scaled up to sentences and trained at huge scale. Typical recipe: weakly-supervised contrastive pre-training on noisy web pairs, then contrastive fine-tuning on curated/labeled pairs.

5. Cosine similarity — the workhorse metric

Given two vectors a and b, cosine similarity is the cosine of the angle between them:

cos(a, b) =  (a · b) / (‖a‖ · ‖b‖)   ∈ [−1, 1]

• a · b is the dot product Σ aᵢbᵢ; ‖a‖ is the vector's length.
• It ranges from +1 (same direction / very similar) through 0 (orthogonal / unrelated) to −1 (opposite).
• Why cosine and not raw distance? Cosine ignores vector magnitude and only cares about direction. In text embeddings, magnitude often correlates with token frequency or length — noise you don't want — while direction carries the meaning. So cosine is more robust than Euclidean distance for semantic comparison.
• The normalization shortcut: if you L2-normalize every vector to unit length (‖a‖ = 1), then cos(a,b) = a · b exactly, and Euclidean²(a,b) = 2 − 2·cos(a,b). So on normalized vectors, cosine, dot product, and Euclidean nearest-neighbor all rank results identically. This is why production vector DBs normalize on ingest and then use the fastest available inner-product index.

At scale you don't brute-force cosine against millions of vectors — you use Approximate Nearest Neighbor (ANN) indexes (HNSW, IVF, etc.) that trade a sliver of recall for orders-of-magnitude speed. (pgvector, used across OpenAlice, supports both exact and HNSW indexes.)

Key ideas & tradeoffs

Rules of thumb: for semantic search/RAG use a contrastively-trained retrieval embedder (not raw BERT, not word2vec). Normalize vectors and use cosine. Pick d by your storage/latency budget. Consult the [MTEB](https://arxiv.org/abs/2210.07316) / [MMTEB](https://arxiv.org/abs/2502.13595) leaderboards — but on your own data, because leaderboard rank ≠ your-domain rank.

Honest caveats & open questions

• Analogy arithmetic is cherry-picked. king − man + woman ≈ queen is real but fragile: it works on hand-curated examples and often excludes the input words from the answer set (otherwise the nearest vector is usually just "king" itself). Don't oversell vector arithmetic as general reasoning — it isn't.
• Embeddings inherit and amplify bias. Because they're trained on human text, static embeddings notoriously encode stereotypes (e.g. gendered occupation associations). The geometry that makes analogies work also encodes "doctor − man + woman" distortions. This is a documented, hard, unsolved problem, not a footnote.
• "Similar" is task-dependent and under-specified. Cosine similarity gives a number, but similar how? Topically? Sentimentally? Stylistically? A single embedding space bakes in one notion of similarity decided by its training data; it may not be the one your task needs. High cosine ≠ "relevant for my use case."
• Anisotropy / the "cone" problem. Raw transformer (e.g. BERT) token embeddings tend to occupy a narrow cone in the space, so everything looks fairly similar and cosine is poorly calibrated. This is exactly why SBERT-style fine-tuning (and whitening tricks) are needed — and why you shouldn't compute cosine on raw LM hidden states naively.
• Out-of-distribution drift. An embedder trained on web text can be confidently wrong on legal/medical/code/multilingual data. Quality is silent — bad embeddings still return some nearest neighbor, just the wrong one. Always evaluate on your domain.
• The interpretability gap. We can't say what dimension 412 "means." Embeddings are dense and entangled; individual axes rarely correspond to human concepts. Research into sparse/monosemantic features and dictionary learning is active and unresolved.
• Versioning is a real operational hazard. Re-embedding your whole corpus with a new model produces vectors that live in a different space — you cannot mix old and new vectors in one index. Embedding model upgrades = full re-index. People forget this and silently corrupt their retrieval.
• Static vs contextual is not strictly "contextual wins." Static embeddings are still excellent when you need a fast, fixed dictionary-style lookup (e.g. fast keyword expansion, lightweight features). Use the simplest thing that works.

How it connects to OpenAlice

Embeddings are foundational plumbing across the ecosystem (verified via Atlas semantic search):

• `openalice-rag` — model-agnostic RAG ingestion + retrieval: BGE-M3 / OpenAI / Mistral embedders × in-memory or pgvector stores. This is the canonical "chunk → embed → store → cosine-retrieve" pipeline described above; it's where contrastive retrieval embedders and ANN search earn their keep for knowledge injection. See [[graphrag]] for the graph-augmented variant.
• `openalice-atlas` (this platform) — replaces literal text matching with pgvector-backed semantic search for feature/code discovery: the /api/v1/search?q= endpoint embeds your query and finds the nearest indexed items by vector similarity. This article was retrieved by the very mechanism it describes.
• `openalice-ml-gateway` — exposes an OpenAI-compatible /v1/embeddings surface so any standard SDK can request embeddings through one routed entry point. Pairs naturally with [[model-routing]] for choosing the embedder per request.
• Alice's memory. Vector retrieval over embedded memories is the standard substrate for long-term recall in agent systems — see [[agent-memory-systems]] and [[mempalace]]. The same cosine-nearest-neighbor pattern lets Alice surface relevant past context without keyword brittleness.
• Caveat for OpenAlice specifically: because [[model-routing]] can route embedding requests to different providers, and providers produce incompatible spaces, the "re-index on model change" hazard above is a live operational concern — never mix vectors from two embedders in one pgvector index.

See also

• [[tokenization]] — what produces the token IDs the embedding matrix indexes
• [[attention-and-transformers]] — turns input embeddings into contextual ones
• [[positional-encoding]] — added to embeddings so order survives attention
• [[graphrag]] · [[agent-memory-systems]] · [[mempalace]] — retrieval/memory built on embeddings
• [[model-routing]] — choosing embedders per request
• [[math-for-ml-foundations]] — dot products, norms, and the linear algebra behind cosine
• [[llm-from-scratch]] · [[microgpt-build-an-llm-from-scratch]] — see the embedding matrix wired into a real model

Primary sources

• Mikolov et al. (2013), Efficient Estimation of Word Representations in Vector Space — word2vec. https://arxiv.org/abs/1301.3781
• Pennington, Socher & Manning (2014), GloVe: Global Vectors for Word Representation. https://nlp.stanford.edu/projects/glove/
• Devlin et al. (2018), BERT: Pre-training of Deep Bidirectional Transformers. https://arxiv.org/abs/1810.04805
• Reimers & Gurevych (2019), Sentence-BERT. https://arxiv.org/abs/1908.10084
• Muennighoff et al. (2022), MTEB: Massive Text Embedding Benchmark. https://arxiv.org/abs/2210.07316
• Enevoldsen et al. (2025), MMTEB: Massive Multilingual Text Embedding Benchmark. https://arxiv.org/abs/2502.13595