Atlas
kb://library/retrieval-augmented-finetuningstable2026-06-17

Retrieval-augmented fine-tuning (RAFT) — teaching a model to use retrieved context well

libraryeducationragretrievalfine-tuningraftdistractorscitationgroundingdomain-adaptationsftm13

Retrieval-augmented fine-tuning (RAFT) — teaching a model to use retrieved context well

For NAO + anyone deciding between "just do RAG" and "just fine-tune" for a domain corpus. The one-sentence version: instead of treating retrieval and fine-tuning as rivals, RAFT fine-tunes a model on the retrieval setting it will actually face at inference — given a question plus a pile of retrieved documents (some relevant, some pure distractors), train it to ignore the noise, cite the right passages, and reason to the answer. It is, in the authors' words, "open-book exam prep": RAG is the open book, fine-tuning is studying how to use the book. This article goes intuition-first, then the real training recipe (the distractor mix, the chain-of-thought-with-citations target, the P% oracle-dropout trick), and is honest about the contested RAG-vs-FT-vs-RAFT question and a real acronym collision. Anchored on RAFT (Zhang et al., arXiv:2403.10131, UC Berkeley / Gorilla team), with the RAG baseline (Lewis et al. 2020), the "Fine-Tuning or Retrieval?" study (Ovadia et al. 2024), and RA-DIT (Lin et al., Meta 2024) for context.

This is the training-side companion to the retrieval articles [[graphrag]] and [[context-engineering]] (which assume a frozen model and engineer what goes into the prompt) and the parameter-efficient training articles [[lora-and-peft]] and [[distillation]]. Related: [[in-context-learning]], [[long-context]].

⚠️ Acronym collision — read this first. "RAFT" denotes two unrelated methods in the literature. This article is about Retrieval-Augmented Fine-Tuning (Zhang et al. 2024, arXiv:2403.10131). It is not Reward-rAnked FineTuning (Dong et al. 2023, arXiv:2304.06767), an iterative best-of-N / rejection-sampling alignment method — that one is referenced in our [[synthetic-data-generation]] article. Same four letters, different papers. When you see "RAFT" cited, check which one.

What it is (intuition first)

You have a domain corpus — internal docs, an API reference, a legal codebase — and you want an LLM to answer questions over it accurately. Two reflexive options:

  1. Just do RAG. Keep the model frozen. At query time, retrieve the top-k most relevant chunks, stuff them into the prompt, and let in-context learning do the work ([[in-context-learning]], [[context-engineering]]). The model has never seen your domain in training; it improvises from whatever you hand it.
  2. Just fine-tune. Train the model's weights on your corpus so the knowledge is "baked in," then answer closed-book. The model has studied the domain but has no document in front of it at inference.

The RAFT paper's framing is a school-exam analogy. Plain RAG is like an open-book exam where you never studied — you have the book, but you flip through it cold, get distracted by irrelevant pages, and miss the relevant one. Plain fine-tuning is like a closed-book exam you crammed for — you studied, but you can't look anything up, so you confabulate when memory fails. RAFT is studying *for an open-book exam*: you practice with the book open, learning which pages matter, how to ignore the decoys, and how to cite the passage that actually supports your answer.

Concretely, RAFT is supervised fine-tuning (SFT) where each training example is shaped exactly like the inference-time prompt: a question + a set of retrieved documents, where the document set deliberately contains a mix of golden (answer-bearing) and distractor (plausible-but-irrelevant) documents. The training target is a chain-of-thought answer that quotes the relevant passages verbatim before concluding (Zhang et al., arXiv:2403.10131). You are not teaching the model new facts so much as teaching it the skill of reading a noisy retrieved context and extracting the grounded answer.

Why it matters

  • It names and fixes a real failure of naive RAG: distractor sensitivity. A frozen model in a RAG pipeline is easily derailed by retrieved-but-irrelevant chunks — the retriever is never perfect, so the context window is always part signal, part noise. RAFT trains the model to be robust to exactly that, which the paper reports as consistent gains over both domain-specific SFT and RAG on PubMed, HotpotQA, and the Gorilla API-function datasets (arXiv:2403.10131; see Honest caveats for how to read those numbers).
  • It dissolves a false dichotomy. "RAG vs fine-tuning" is the wrong question; the honest answer is both, and trained together for the deployment setting. RAFT is one clean instance of that thesis; RA-DIT (Meta) is another — it instruction-tunes the LM and the retriever jointly (Lin et al., arXiv:2310.01352).
  • It's cheap relative to its effect. RAFT is plain SFT on synthetically constructed (question, context, cited-CoT-answer) triples — no RL, no reward model. It composes with parameter-efficient methods like [[lora-and-peft]], so domain-adapting a model this way is within reach of a small team / single-GPU budget.
  • It directly serves grounding and citation. Because the training target quotes the supporting passage, RAFT pushes the model toward answers that are traceable to a source — the property you want for any system where a wrong, uncited claim is expensive (legal, medical, internal-knowledge assistants).

How it works (the real mechanics)

RAFT has three moving parts: the data construction, the distractor mixing (including the key oracle-dropout trick), and the citation-style CoT target.

1. Data construction — make training look like inference

For a domain corpus, build SFT examples of the form:

INPUT :  Question Q  +  a set of documents { D_golden? , D_distractor_1 , ... , D_distractor_k }
TARGET:  a chain-of-thought answer A* that quotes the supporting span(s) from the golden doc,
         then states the final answer

The golden document D* is the one that actually contains the answer to Q (typically the chunk Q was generated from). The distractors D_i are other chunks from the same corpus retrieved as "near misses" — topically similar, plausibly relevant, but not answer-bearing. This is the whole point: the model sees the realistic, noisy retrieval setting during training, not a clean single-document setting.

2. The distractor mix and the P% oracle-dropout trick

This is the part people get wrong when they reimplement RAFT. The training set is split into two kinds of examples, controlled by a fraction P:

  • For P% of the questions: include the golden document D* plus distractors. (Learn to find and use the right doc amid noise.)
  • For the remaining (100−P)%: drop the golden document entirely — train on only distractors. (The answer is still the correct one, forcing the model to answer from memorized domain knowledge when retrieval fails to surface the right passage.)

That second bucket is the counterintuitive, load-bearing trick. Training without the golden doc some fraction of the time stops the model from learning a lazy "copy from the obviously-relevant doc" shortcut and pushes it to internalize domain knowledge — so it degrades gracefully when the retriever misses. The paper treats P as a tunable hyperparameter and reports that the best value is dataset-dependent (it is not a universal constant — see caveats; do not quote a single magic P% as gospel). The headline qualitative finding is that a nonzero distractor-only fraction helps; the exact optimum you must tune.

3. The training target — chain-of-thought with citations

RAFT's answer target is not a bare answer. It is a reasoning chain that quotes the relevant context before answering — the paper shows the answer string including verbatim spans copied from the golden document (often delimited so the citation is explicit), followed by the conclusion. Two things fall out of this:

  • Grounding by construction. The model is rewarded for producing an answer whose support is literally present in the cited span, which is a far stronger anti-hallucination signal than "just output the answer."
  • Reasoning helps. This is the same lesson as [[in-context-learning]] and the broader chain-of-thought literature: writing intermediate reasoning before the answer improves correctness. RAFT's ablations indicate the CoT-with-citation target matters, not just the distractor mixing.

4. Where RAFT sits relative to RAG and FT

ApproachTrains weights?Sees docs at inference?"Studied" the domain?
RAG (frozen LM)noyesno — improvises from retrieval
Domain SFT (closed-book)yesnoyes — but can't look up
DSF + RAG (SFT then retrieve)yesyesyes, but not trained on the noisy-retrieval setting
RAFTyesyesyes — and trained to handle distractors + cite

RAFT's claim is that the last row dominates because it is the only one that train-test matches: the model is fine-tuned on the exact messy distribution (question + mixed-quality retrieved docs) it will be deployed on. RA-DIT pushes the same idea further by also tuning the retriever, not just the generator (Lin et al., arXiv:2310.01352).

Key ideas & tradeoffs

  • Train-test matching is the core principle. The single most transferable takeaway is not "RAFT" the brand — it's fine-tune on the inference distribution, including its noise. If your deployed system retrieves k noisy chunks, train on k noisy chunks. This generalizes beyond RAFT.
  • Distractor robustness vs over-reliance on memory. The P% knob trades two failure modes: too high P (always show the golden doc) → the model leans on retrieval and collapses when retrieval misses; too low P → the model over-memorizes and ignores the provided context even when it's correct. There's a sweet spot and it's dataset-specific.
  • Domain-specialization vs generality. RAFT adapts a model to one domain's retrieval setting. A model RAFT-tuned on PubMed-style QA is specialized; it is not a free upgrade to a general assistant. Expect to train (and store, via [[lora-and-peft]] adapters) one specialization per domain.
  • Citation as supervision, not just output formatting. Asking for quoted spans is cheap and doubles as a hallucination check at eval time — you can verify the cited span actually exists in the context. This is a practical grounding lever even outside RAFT.
  • Synthetic-data dependence. RAFT needs (Q, golden-doc, distractors, CoT-answer) quadruples, usually generated by a stronger teacher model over your corpus. So RAFT inherits all the quality/diversity/contamination concerns of [[synthetic-data-generation]] — garbage triples in, brittle model out.

Honest caveats & open questions

  • The benchmark gains are real but domain-specific — read them as such. RAFT reports improvements over DSF and RAG baselines on PubMed, HotpotQA, and Gorilla API datasets (arXiv:2403.10131). I am deliberately not quoting specific percentage-point deltas here because they vary by dataset and base model and I will not risk a transposed number; consult the paper's tables directly for the exact figures on the task you care about.
  • "RAG vs fine-tuning" is genuinely contested. Ovadia et al., Fine-Tuning or Retrieval? (arXiv:2312.05934), found that for injecting new factual knowledge, RAG generally beat unsupervised fine-tuning — fine-tuning struggled to absorb new facts reliably. RAFT doesn't contradict this so much as sidestep it: RAFT fine-tunes the skill of using retrieval, not the facts themselves, and keeps retrieval at inference. The field has not converged on a single rule of thumb; the answer is workload- and goal-dependent.
  • P% is a hyperparameter, not a discovery. Treat any "use X% distractor-only" recipe you see repeated online as a starting point to tune, not a law. The paper itself frames P as tunable and dataset-dependent.
  • It assumes a roughly working retriever. RAFT makes the generator robust to some retrieval noise, but if the retriever almost never surfaces the golden doc, RAFT's memorize-fallback bucket is doing all the work and you've effectively reinvented closed-book SFT. Retriever quality still bounds the ceiling — which is exactly why RA-DIT tunes the retriever too.
  • The acronym collision is a live source of citation errors. As flagged at the top: Reward-rAnked FineTuning (Dong et al. 2023, arXiv:2304.06767) is a different method. Bibliographies and blog posts conflate the two. Verify the arXiv ID, not the letters.
  • Forgetting and maintenance. Like any SFT, RAFT can erode general capabilities (catastrophic forgetting) and goes stale as the corpus changes — pairing it with PEFT adapters ([[lora-and-peft]]) and periodic re-tuning is the pragmatic mitigation, but it is ongoing cost, not a one-shot fix.

How it connects to OpenAlice

  • Atlas itself is a RAG system that could be RAFT-shaped. The Atlas knowledge library + semantic search (GET /api/v1/search) is a retrieval layer over the org's code and docs. Today an agent retrieves chunks and a frozen model reasons over them — pure RAG, with all the distractor sensitivity that implies. RAFT is the recipe if we ever want a model specialized at reading Atlas's noisy retrieved context (entity graphs from [[graphify]], community summaries from [[graphrag]]) and citing the right symbol/file. That's a future bet, flagged as such — not something wired in.
  • Citation discipline maps onto our grounding goals. RAFT's "quote the supporting span" target is exactly the behavior we want from any OpenAlice assistant answering over internal docs: an answer you can trace to a source. Even without full RAFT training, the eval idea — check that the cited span exists in the provided context — is a cheap grounding check worth adopting.
  • Sibling positioning. This is the training-time counterpart to [[context-engineering]] (which optimizes the prompt for a frozen model) and [[graphrag]] (which optimizes what gets retrieved). RAFT optimizes the model's skill at consuming whatever those produce. The three compose: better retrieval (GraphRAG) + better context assembly (context engineering) + a model trained to use noisy context (RAFT).
  • Cost-aware adoption. RAFT is SFT + synthetic data, both of which our lab/bench rig can already produce; with [[lora-and-peft]] it's a per-domain adapter, not a full re-train. If a domain-specialized retrieval assistant ever earns its keep, RAFT is the lowest-ceremony way to get there — but it should be prototyped in the bench rig and measured against a plain-RAG baseline before any prod wiring.

Sources