RAG, for people building on top of an LLM
How to ground a language model in your own, current data — the pipeline, the failure modes, and the point past which adding more retrieval tricks stops helping.
Retrieval instead of memorization
Retrieval-Augmented Generation solves one specific problem from the LLM guide: a model's knowledge is frozen at training time, and its context window is too small and too expensive to just paste your entire knowledge base into every prompt. RAG's answer is to fetch only the handful of passages relevant to the current question, and hand those to the model alongside it — so the model is reasoning over retrieved facts instead of recalling (or guessing at) facts baked into its weights.
That reframes what an LLM-backed system is actually being asked to do. Instead of "what do you know about X," the question becomes "given these specific documents, what do they say about X" — a task the model is much better and more checkable at, because the grounding text is right there in the prompt to verify against.
RAG is not a replacement for fine-tuning, and it isn't magic: it's only as good as what gets retrieved. Most of what makes RAG work — or fail — in practice comes down to the mechanics in the rest of this guide, not the LLM call at the end of the pipeline.
Ingestion happens offline; retrieval happens on every query
Production RAG is really two separate pipelines running on completely different schedules, and conflating them is a common source of confusion.
Ingestion runs whenever your source data changes — batch jobs, scheduled re-indexing, or event-driven updates. Retrieval runs fresh on every single user query. Most RAG debugging starts by figuring out which of the two pipelines a problem actually belongs to.
This is where most RAG quality is won or lost
A chunk is the unit your system can retrieve — never smaller, never more precise than that. Cut it wrong and no amount of clever retrieval afterward can recover the information you split across a boundary.
| Strategy | How it works | Trade-off |
|---|---|---|
| Fixed-size | Splits at a set token/character count | Trivial and fast; routinely cuts sentences and ideas in half |
| Recursive | Tries paragraph → sentence → word boundaries until chunks fit | Good baseline; still blind to document structure |
| Document-aware | Splits on real structure — headers, tables, code blocks | Big quality jump on structured content, no extra compute |
| Semantic | Uses embedding similarity to detect topic shifts | High precision on mixed content; 3–10× slower to index |
| Hierarchical (parent/child) | Small chunks for retrieval, larger parent chunks for generation context | The de facto production standard — resolves precision vs. context |
| Contextual retrieval | An LLM prepends a short context summary to each chunk before embedding | Cuts retrieval failures substantially; costs an LLM call per chunk at index time |
There's no universally correct chunk size — teams typically start around 512 tokens with 10–20% overlap between chunks, then adjust based on where evaluation shows retrieval actually failing, not on intuition. The practical sequence is: start with recursive splitting as a baseline, measure failures systematically, and only reach for semantic, hierarchical, or contextual chunking where the failure mode actually justifies the added cost.
The uncomfortable truth about chunking
No chunking strategy fixes stale, duplicated, or poorly governed source data. Teams that get RAG reliable tend to stop obsessing over chunk size and start treating the source corpus itself — freshness, ownership, de-duplication — as the thing that actually needs maintaining.
Neither search method wins on its own
Dense (vector/embedding) search finds semantic and paraphrased matches — it knows "car" and "automobile" are related. Sparse search (classically BM25, a refined keyword-matching algorithm) finds exact matches — product codes, error messages, proper nouns — that embeddings often blur past. Production systems need both, because each one's blind spot is the other's strength.
Hybrid search runs both in parallel and merges the results, typically via Reciprocal Rank Fusion (RRF) — which combines results by their rank position in each list rather than by raw score, sidestepping the problem that BM25 scores and cosine similarities aren't on comparable scales to begin with. On a public e-commerce benchmark, RRF-fused hybrid search outperformed either method alone by roughly 7%, which is a representative, not universal, margin — the exact gain depends heavily on your query distribution.
If your users ever search for SKUs, ticket numbers, or exact product names, pure dense retrieval will quietly fail them — that's the single most common reason teams bolt sparse search back onto a vector-only pipeline after launch.
Retrieve broad, then rerank narrow
The embeddings used for initial retrieval are bi-encoders — the query and each document are encoded independently, which is what makes searching thousands of documents fast. That speed has a cost: independent encoding can't capture fine-grained relevance the way a model that reads the query and a candidate document together can.
A cross-encoder reranker does exactly that — it scores each (query, candidate) pair jointly, at real precision, but it's far too expensive to run against your entire index. The standard two-stage pattern is: retrieve a broad shortlist (hybrid search, top 100–1000 candidates) cheaply, then rerank just that shortlist with a cross-encoder before handing the survivors to the LLM. You get retrieval's speed and reranking's precision, applied where each is actually affordable.
What to reach for when retrieve-then-generate isn't enough
"Naive" RAG — embed the query, retrieve top-k, generate — has a real accuracy ceiling; published benchmarks put naive RAG around 44% factual accuracy versus roughly 63% for systems layering in the techniques below. None of these are free, so each is worth adding only once you've confirmed, through evaluation, which specific failure it would fix.
| Technique | Problem it solves |
|---|---|
| HyDE | Embeds a hypothetical generated answer instead of the raw query, closing the vocabulary gap between a short question and technical source documents |
| Query rewriting / RAG-Fusion | Generates several phrasings of an ambiguous query and fuses the results, covering angles a single phrasing misses |
| Multi-hop (RAPTOR, GraphRAG) | Builds hierarchical summaries or knowledge graphs so questions spanning multiple documents or relationships can actually be answered |
| Agentic / iterative (Self-RAG, Adaptive RAG) | Lets the system decide whether and how much to retrieve per query, instead of always retrieving a fixed amount |
This is also where RAG starts to blur into Layer 04 (AI Agents) — an agentic retrieval loop that decides when to search again, refine its query, or stop is a small agent in its own right.
Most failures aren't the LLM's fault
A widely-repeated finding in production RAG work: roughly 80% of failures trace back to the ingestion and chunking layer, not the language model generating the final answer. Blaming the model for a bad answer, when the real problem is what got retrieved, is the single most common misdiagnosis in RAG debugging.
| Failure mode | What's actually happening |
|---|---|
| Broken chunking | A semantic unit got split across a boundary, so no single retrieved chunk contains the full answer |
| Dense-only retrieval | Exact-match queries (codes, names) fail silently because nothing in the pipeline does keyword matching |
| No reranking | The system trusts raw similarity scores, surfacing plausible-looking but low-relevance chunks |
| Stale index | Source documents changed; the index wasn't re-run, so retrieval confidently returns outdated information |
| No evaluation | There's no way to tell whether a pipeline change improved or degraded retrieval quality |
| No observability | Zero-result queries, latency spikes, and drift go unnoticed until a user complains |
RAG is the bridge, not the storage or the judge
RAG exists to patch two specific LLM limitations named in the LLM guide: the knowledge cutoff and the context window's cost/accuracy ceiling. It doesn't do that alone — it leans on the layer right below it and hands off to the layer above.
Vector DB (Layer 03)
RAG describes the pattern; a vector database is what actually makes retrieval fast at any real scale. The Vector DB guide covers indexing and search structures in depth.
Evals (Layer 07)
Context precision/recall, faithfulness, and answer relevancy — the metrics that tell you whether your RAG pipeline is actually working — are covered in full in the Evals guide, §04, rather than repeated here.
All seven layers are now published — back to the hub for the full stack.
Start simple, add complexity where evaluation demands it
- Ship a baseline first: recursive chunking, dense retrieval, no reranking. It's fast to build and gives you something real to measure against.
- Let error analysis pick your next investment — the same discipline from the Evals guide applies here directly: read failing queries before reaching for a fancier technique.
- Add hybrid search the moment exact-match queries show up in real usage — product codes, IDs, names. This is rarely optional in practice.
- Add reranking before you add fancier retrieval tricks — it's usually the highest-leverage single change once a baseline is in place.
- Reserve agentic/multi-hop retrieval for genuinely multi-document questions — it's expensive and slow, and most queries don't need it.
- Monitor production, not just your test set: retrieval latency, zero-result rate, reranker score trends, and sampled faithfulness checks catch drift a static eval set won't.
Before you build a RAG pipeline
- Confirm the task actually needs retrieval — fresh, private, or large-corpus knowledge — rather than something a well-crafted prompt already handles.
- Pick a chunking strategy appropriate to your source format (structured docs vs. free-flowing prose) rather than defaulting to fixed-size splits.
- Build a small, real evaluation set of query → expected-source pairs before writing any retrieval code, so every later change has something to be measured against.
- Plan for hybrid search from the start if your users will ever search for exact identifiers, not as an afterthought once it's clearly missing.
- Decide how the index gets refreshed when source documents change — an undated answer to "how fresh is this?" is a production incident waiting to happen.
- Read the Evals guide's RAG section before shipping — you'll want retrieval and generation metrics in place before, not after, your first regression.
RAG doesn't make an LLM smarter. It makes the LLM's answer checkable — provided what you hand it to check against was retrieved correctly in the first place.
What this guide draws on
Read in full and used to shape the structure and claims of this guide.
| Source | Publisher | Used for |
|---|---|---|
| Chunking Strategies for RAG: Methods, Trade-offs & Best Practices | Atlan | Chunking strategy comparison, practical chunk-size guidance, the ingestion-quality framing (§02) |
| Hybrid Search: BM25, Vector & Reranking Reference 2026 | Digital Applied | Dense/sparse retrieval mechanics, Reciprocal Rank Fusion, cross-encoder reranking (§03–04) |
| 12 Advanced RAG Techniques: Beyond Naive Retrieval | Atlan | HyDE, query rewriting/RAG-Fusion, multi-hop and agentic retrieval patterns, naive-RAG accuracy benchmark (§05) |
| Building Production RAG: Architecture, Chunking, Evaluation & Monitoring | PremAI | Ingestion vs. query pipeline architecture, production failure modes, monitoring signals (§01, §06) |
Compiled 2026. Specific benchmark numbers (accuracy percentages, NDCG scores) come from the cited sources' own test setups — treat them as illustrative of direction and rough magnitude, not as guarantees for your corpus.