Field guide · Layer 02

RAG, for people building on top of an LLM

How to ground a language model in your own, current data — the pipeline, the failure modes, and the point past which adding more retrieval tricks stops helping.

00 — What is RAG

Retrieval instead of memorization

Retrieval-Augmented Generation solves one specific problem from the LLM guide: a model's knowledge is frozen at training time, and its context window is too small and too expensive to just paste your entire knowledge base into every prompt. RAG's answer is to fetch only the handful of passages relevant to the current question, and hand those to the model alongside it — so the model is reasoning over retrieved facts instead of recalling (or guessing at) facts baked into its weights.

That reframes what an LLM-backed system is actually being asked to do. Instead of "what do you know about X," the question becomes "given these specific documents, what do they say about X" — a task the model is much better and more checkable at, because the grounding text is right there in the prompt to verify against.

RAG is not a replacement for fine-tuning, and it isn't magic: it's only as good as what gets retrieved. Most of what makes RAG work — or fail — in practice comes down to the mechanics in the rest of this guide, not the LLM call at the end of the pipeline.

01 — Two pipelines

Ingestion happens offline; retrieval happens on every query

Production RAG is really two separate pipelines running on completely different schedules, and conflating them is a common source of confusion.

Ingest
Parse documents
Extract text/structure from source files — PDFs, wikis, tickets, whatever your corpus is.
Ingest
Chunk
Split into retrievable units small enough to be precise, large enough to stay meaningful (§02).
Ingest
Embed & index
Convert chunks to vectors, store them in a searchable index — the Vector DB layer.
Query
Retrieve
Search the index for chunks relevant to the incoming question (§03).
Query
Rerank
Re-score the retrieved candidates for actual relevance before committing to them (§04).
Query
Generate
Assemble the surviving chunks into context and let the LLM answer from them.

Ingestion runs whenever your source data changes — batch jobs, scheduled re-indexing, or event-driven updates. Retrieval runs fresh on every single user query. Most RAG debugging starts by figuring out which of the two pipelines a problem actually belongs to.

02 — Chunking

This is where most RAG quality is won or lost

A chunk is the unit your system can retrieve — never smaller, never more precise than that. Cut it wrong and no amount of clever retrieval afterward can recover the information you split across a boundary.

StrategyHow it worksTrade-off
Fixed-sizeSplits at a set token/character countTrivial and fast; routinely cuts sentences and ideas in half
RecursiveTries paragraph → sentence → word boundaries until chunks fitGood baseline; still blind to document structure
Document-awareSplits on real structure — headers, tables, code blocksBig quality jump on structured content, no extra compute
SemanticUses embedding similarity to detect topic shiftsHigh precision on mixed content; 3–10× slower to index
Hierarchical (parent/child)Small chunks for retrieval, larger parent chunks for generation contextThe de facto production standard — resolves precision vs. context
Contextual retrievalAn LLM prepends a short context summary to each chunk before embeddingCuts retrieval failures substantially; costs an LLM call per chunk at index time

There's no universally correct chunk size — teams typically start around 512 tokens with 10–20% overlap between chunks, then adjust based on where evaluation shows retrieval actually failing, not on intuition. The practical sequence is: start with recursive splitting as a baseline, measure failures systematically, and only reach for semantic, hierarchical, or contextual chunking where the failure mode actually justifies the added cost.

The uncomfortable truth about chunking

No chunking strategy fixes stale, duplicated, or poorly governed source data. Teams that get RAG reliable tend to stop obsessing over chunk size and start treating the source corpus itself — freshness, ownership, de-duplication — as the thing that actually needs maintaining.

03 — Dense, sparse & hybrid

Neither search method wins on its own

Dense (vector/embedding) search finds semantic and paraphrased matches — it knows "car" and "automobile" are related. Sparse search (classically BM25, a refined keyword-matching algorithm) finds exact matches — product codes, error messages, proper nouns — that embeddings often blur past. Production systems need both, because each one's blind spot is the other's strength.

Hybrid search runs both in parallel and merges the results, typically via Reciprocal Rank Fusion (RRF) — which combines results by their rank position in each list rather than by raw score, sidestepping the problem that BM25 scores and cosine similarities aren't on comparable scales to begin with. On a public e-commerce benchmark, RRF-fused hybrid search outperformed either method alone by roughly 7%, which is a representative, not universal, margin — the exact gain depends heavily on your query distribution.

If your users ever search for SKUs, ticket numbers, or exact product names, pure dense retrieval will quietly fail them — that's the single most common reason teams bolt sparse search back onto a vector-only pipeline after launch.

04 — Reranking

Retrieve broad, then rerank narrow

The embeddings used for initial retrieval are bi-encoders — the query and each document are encoded independently, which is what makes searching thousands of documents fast. That speed has a cost: independent encoding can't capture fine-grained relevance the way a model that reads the query and a candidate document together can.

A cross-encoder reranker does exactly that — it scores each (query, candidate) pair jointly, at real precision, but it's far too expensive to run against your entire index. The standard two-stage pattern is: retrieve a broad shortlist (hybrid search, top 100–1000 candidates) cheaply, then rerank just that shortlist with a cross-encoder before handing the survivors to the LLM. You get retrieval's speed and reranking's precision, applied where each is actually affordable.

05 — Beyond naive RAG

What to reach for when retrieve-then-generate isn't enough

"Naive" RAG — embed the query, retrieve top-k, generate — has a real accuracy ceiling; published benchmarks put naive RAG around 44% factual accuracy versus roughly 63% for systems layering in the techniques below. None of these are free, so each is worth adding only once you've confirmed, through evaluation, which specific failure it would fix.

TechniqueProblem it solves
HyDEEmbeds a hypothetical generated answer instead of the raw query, closing the vocabulary gap between a short question and technical source documents
Query rewriting / RAG-FusionGenerates several phrasings of an ambiguous query and fuses the results, covering angles a single phrasing misses
Multi-hop (RAPTOR, GraphRAG)Builds hierarchical summaries or knowledge graphs so questions spanning multiple documents or relationships can actually be answered
Agentic / iterative (Self-RAG, Adaptive RAG)Lets the system decide whether and how much to retrieve per query, instead of always retrieving a fixed amount

This is also where RAG starts to blur into Layer 04 (AI Agents) — an agentic retrieval loop that decides when to search again, refine its query, or stop is a small agent in its own right.

06 — Where it breaks

Most failures aren't the LLM's fault

A widely-repeated finding in production RAG work: roughly 80% of failures trace back to the ingestion and chunking layer, not the language model generating the final answer. Blaming the model for a bad answer, when the real problem is what got retrieved, is the single most common misdiagnosis in RAG debugging.

Failure modeWhat's actually happening
Broken chunkingA semantic unit got split across a boundary, so no single retrieved chunk contains the full answer
Dense-only retrievalExact-match queries (codes, names) fail silently because nothing in the pipeline does keyword matching
No rerankingThe system trusts raw similarity scores, surfacing plausible-looking but low-relevance chunks
Stale indexSource documents changed; the index wasn't re-run, so retrieval confidently returns outdated information
No evaluationThere's no way to tell whether a pipeline change improved or degraded retrieval quality
No observabilityZero-result queries, latency spikes, and drift go unnoticed until a user complains
07 — Where it sits in the stack

RAG is the bridge, not the storage or the judge

RAG exists to patch two specific LLM limitations named in the LLM guide: the knowledge cutoff and the context window's cost/accuracy ceiling. It doesn't do that alone — it leans on the layer right below it and hands off to the layer above.

Vector DB (Layer 03)

RAG describes the pattern; a vector database is what actually makes retrieval fast at any real scale. The Vector DB guide covers indexing and search structures in depth.

Evals (Layer 07)

Context precision/recall, faithfulness, and answer relevancy — the metrics that tell you whether your RAG pipeline is actually working — are covered in full in the Evals guide, §04, rather than repeated here.

All seven layers are now published — back to the hub for the full stack.

08 — Practical guidance

Start simple, add complexity where evaluation demands it

  • Ship a baseline first: recursive chunking, dense retrieval, no reranking. It's fast to build and gives you something real to measure against.
  • Let error analysis pick your next investment — the same discipline from the Evals guide applies here directly: read failing queries before reaching for a fancier technique.
  • Add hybrid search the moment exact-match queries show up in real usage — product codes, IDs, names. This is rarely optional in practice.
  • Add reranking before you add fancier retrieval tricks — it's usually the highest-leverage single change once a baseline is in place.
  • Reserve agentic/multi-hop retrieval for genuinely multi-document questions — it's expensive and slow, and most queries don't need it.
  • Monitor production, not just your test set: retrieval latency, zero-result rate, reranker score trends, and sampled faithfulness checks catch drift a static eval set won't.
09 — Getting started

Before you build a RAG pipeline

  • Confirm the task actually needs retrieval — fresh, private, or large-corpus knowledge — rather than something a well-crafted prompt already handles.
  • Pick a chunking strategy appropriate to your source format (structured docs vs. free-flowing prose) rather than defaulting to fixed-size splits.
  • Build a small, real evaluation set of query → expected-source pairs before writing any retrieval code, so every later change has something to be measured against.
  • Plan for hybrid search from the start if your users will ever search for exact identifiers, not as an afterthought once it's clearly missing.
  • Decide how the index gets refreshed when source documents change — an undated answer to "how fresh is this?" is a production incident waiting to happen.
  • Read the Evals guide's RAG section before shipping — you'll want retrieval and generation metrics in place before, not after, your first regression.
RAG doesn't make an LLM smarter. It makes the LLM's answer checkable — provided what you hand it to check against was retrieved correctly in the first place.
10 — Sources

What this guide draws on

Read in full and used to shape the structure and claims of this guide.

SourcePublisherUsed for
Chunking Strategies for RAG: Methods, Trade-offs & Best Practices Atlan Chunking strategy comparison, practical chunk-size guidance, the ingestion-quality framing (§02)
Hybrid Search: BM25, Vector & Reranking Reference 2026 Digital Applied Dense/sparse retrieval mechanics, Reciprocal Rank Fusion, cross-encoder reranking (§03–04)
12 Advanced RAG Techniques: Beyond Naive Retrieval Atlan HyDE, query rewriting/RAG-Fusion, multi-hop and agentic retrieval patterns, naive-RAG accuracy benchmark (§05)
Building Production RAG: Architecture, Chunking, Evaluation & Monitoring PremAI Ingestion vs. query pipeline architecture, production failure modes, monitoring signals (§01, §06)

Compiled 2026. Specific benchmark numbers (accuracy percentages, NDCG scores) come from the cited sources' own test setups — treat them as illustrative of direction and rough magnitude, not as guarantees for your corpus.