Retrieval-Augmented Generation (RAG)

For NAO + anyone building grounded LLM applications. RAG is the dominant pattern for giving language models access to private or up-to-date knowledge without retraining. It powers most production chat-with-your-docs systems, including OpenAlice's own persona knowledge pipelines. This article covers the core architecture, retrieval strategies, chunking, reranking, and the main failure modes — grounded in the original Lewis et al. paper, the 2024 survey by Gao et al., and practical lessons from building RAG at scale.

The one-sentence idea

Retrieve relevant documents from an external store, paste them into the prompt, then generate. The model sees evidence it was never trained on, so it can answer questions about private data, recent events, or domain-specific knowledge — without fine-tuning.

Why RAG exists

LLMs have two fundamental limitations RAG addresses:

1. Knowledge cutoff — the model only knows what was in its training data. Anything after that date (or anything private) is invisible.
2. Hallucination — without grounding, models confabulate plausible-sounding facts. RAG gives the model evidence to cite, reducing (not eliminating) hallucination.

Fine-tuning can inject knowledge, but it's expensive, slow, and doesn't scale to frequently-changing corpora. RAG decouples what the model knows how to do (reason, summarize, follow instructions) from what facts it has access to (the retrieval index).

Core architecture

A minimal RAG pipeline has three stages:

┌──────────┐     ┌──────────────┐     ┌────────────┐
│  Query   │────▶│   Retrieve   │────▶│  Generate  │
│          │     │  (top-k docs)│     │  (LLM)     │
└──────────┘     └──────────────┘     └────────────┘
       │                │
       ▼                ▼
  embed query      rank by similarity
  (same model)     against doc embeddings

1. Indexing (offline)

• Chunk the corpus into passages (typical: 256–512 tokens with overlap).
• Embed each chunk with a dense encoder (e.g., text-embedding-3-large, BGE, GTE, Nomic, or Cohere Embed). Store vectors in a vector database (pgvector, Qdrant, Pinecone, Weaviate, Milvus, FAISS).
• Optionally store sparse representations (BM25, SPLADE) alongside dense vectors for hybrid search.

2. Retrieval (online)

• Embed the user query with the same encoder.
• Approximate nearest-neighbor search retrieves top-k chunks (typically k=5–20).
• Rerank with a cross-encoder (e.g., Cohere Rerank, BGE-reranker, ColBERT) to re-score the top-k by relevance. This is the single highest-ROI addition to a naive RAG pipeline.

3. Generation

• Stuff retrieved chunks into the prompt (or use map-reduce / refine for long context).
• The LLM generates an answer grounded in the retrieved evidence.
• Optionally cite sources with chunk IDs or page numbers.

Chunking strategies

Chunking is deceptively important — bad chunks → bad retrieval → bad answers.

Overlap (typically 10–20% of chunk size) prevents information from being split across chunk boundaries.

Retrieval quality: the 80% problem

In practice, retrieval quality determines ~80% of RAG answer quality. Common failure modes:

• Embedding mismatch — query and document use different vocabulary; the right chunk ranks low. Fix: query expansion, HyDE (hypothetical document embeddings), or hybrid search.
• Lost in the middle — LLMs attend less to information in the middle of long contexts (Liu et al., 2023). Fix: put most relevant chunks first/last, or limit context length.
• Chunk boundary splits — the answer spans two chunks, neither of which is self-contained. Fix: overlap, parent-document retrieval, or larger chunks.
• Stale index — the corpus changed but the index wasn't rebuilt. Fix: incremental indexing, change-detection triggers.

Advanced patterns

Hybrid search (dense + sparse)

Combine BM25 (lexical, exact-match) with dense embeddings (semantic). Typically fused with Reciprocal Rank Fusion (RRF) or a learned combiner. Hybrid consistently outperforms either method alone across benchmarks.

HyDE — Hypothetical Document Embeddings

Generate a hypothetical answer first (without retrieval), embed that hypothetical answer, then retrieve. The hypothesis is often closer in embedding space to the real document than the original question is. (Gao et al., 2022, arXiv:2212.10496)

Self-RAG

The model learns when to retrieve and when to ignore retrieved passages via special reflection tokens. Outperforms standard RAG on knowledge-intensive tasks because it doesn't blindly trust every retrieved chunk. (Asai et al., 2024, arXiv:2401.15884)

Agentic RAG

The retriever becomes a tool the agent calls on demand — potentially multiple times per turn, with query reformulation between calls. This is how most production agent systems (including OpenAlice) structure retrieval: the agent decides if and what to retrieve rather than always retrieving.

GraphRAG

Build a knowledge graph from the corpus, then retrieve over graph communities rather than flat chunks. Best for global sensemaking queries ("what are the main themes?") where the answer is spread across the whole corpus. See → GraphRAG.

Evaluation

RAG evaluation requires measuring both retrieval and generation:

Frameworks: RAGAS, TruLens, DeepEval, Phoenix (Arize).

Practical checklist

1. Start simple — fixed-size chunks + single dense embedding + top-5 retrieval. Measure baseline quality before adding complexity.
2. Add reranking — cross-encoder reranking is the single biggest quality lift for minimal effort.
3. Add hybrid search — BM25 + dense with RRF catches lexical matches dense embeddings miss.
4. Tune chunk size — experiment with 256 vs 512 vs 1024 tokens; optimal size depends on your corpus.
5. Evaluate continuously — build a golden test set of question/answer pairs with known source chunks.

Key papers

• Lewis et al., *Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks* (NeurIPS 2020, arXiv:2005.11401)↗ — the original RAG paper.
• Karpukhin et al., *Dense Passage Retrieval* (EMNLP 2020, arXiv:2002.08909)↗ — DPR, the dense retriever that made RAG practical.
• Borgeaud et al., *RETRO* (ICML 2022, arXiv:2112.09332)↗ — retrieval baked into the transformer architecture itself.
• Gao et al., *RAG for LLMs: A Survey* (2024, arXiv:2312.10997)↗ — comprehensive taxonomy of RAG variants.
• Asai et al., *Self-RAG* (ICLR 2024, arXiv:2401.15884)↗ — adaptive retrieval with reflection tokens.

See also

• Embeddings & Vector Spaces — the representation layer RAG builds on.
• GraphRAG — graph-structured retrieval for global queries.
• Retrieval-Augmented Fine-Tuning (RAFT) — teaching the model to use retrieved context during fine-tuning.
• Long Context — when context windows grow large enough, do you still need RAG? (Yes, but differently.)
• Semantic Caching for LLMs — caching RAG results by query similarity.