Vector DBs, for people building on top of an LLM
The RAG guide describes retrieval as a pattern. This is what actually makes it fast — the index structures, trade-offs, and failure modes behind "search a billion embeddings in a few milliseconds."
Why you can't just loop over the embeddings
A vector database stores embeddings — the numeric representations from the LLM guide's discussion of embeddings — and finds the ones closest to a query vector, fast, at whatever scale your corpus reaches. It pairs a search-optimized index with the operational features of a normal database: inserts and deletes, metadata storage and filtering, horizontal scaling, backups, and multi-tenant isolation.
The naive version of this problem is genuinely simple: compute the distance from your query vector to every stored vector, sort, take the top few. It's exact — 100% correct, by construction — and for a few thousand vectors it's fast enough that you'd never notice. The trouble is that this brute-force scan scales linearly with both the number of vectors and their dimensionality. In one benchmark, an exact scan over a million 128-dimension vectors took roughly 18 milliseconds per query; an approximate index reached comparable accuracy in under 2 milliseconds. At real production scale — tens or hundreds of millions of vectors, many concurrent queries per second — linear scanning simply runs out of time and memory.
That gap is the entire reason vector databases exist as a category. Trade a small, controllable amount of accuracy for a large, predictable amount of speed — that's what an approximate nearest neighbor (ANN) index buys you, and it's the subject of most of this guide.
"Closest" has to mean something specific
Before any index can find "nearest" vectors, something has to define what nearness means. Three metrics cover almost every production system:
| Metric | What it measures | Use when |
|---|---|---|
| Cosine similarity | The angle between two vectors — ignores magnitude entirely | The default for semantic/sentence embeddings; most embedding models are trained and evaluated against it |
| Dot product | Angle and magnitude — a vector's length can carry meaning | Models that deliberately encode signal strength (popularity, authority) into vector length |
| Euclidean (L2) | Straight-line distance in the vector space | Magnitude genuinely matters and the model wasn't optimized for cosine |
The rule that actually matters in practice: use whichever metric the embedding model was trained and evaluated against — not whichever one feels intuitive. Every one of these will still return a number if you use the wrong metric with a given model; nothing errors out. The ranking just quietly degrades, which is a hard bug to notice and an easy one to avoid. Since cosine similarity and dot product become mathematically equivalent once vectors are L2-normalized, many systems normalize at ingestion time and use the cheaper dot-product computation internally while still getting cosine semantics — an implementation detail worth knowing, not a decision you usually have to make yourself.
Three ways to avoid scanning everything
Approximate nearest neighbor search is the actual engineering underneath every vector database. Three approaches make up most of what's deployed in production, each trading recall, speed, and memory differently.
HNSW — Hierarchical Navigable Small World graphs
A multi-layer graph, inspired by skip lists: the top layer has few nodes and long-range links, each layer below has more nodes and shorter links, and the bottom layer holds every vector. A query starts at an entry point in the top layer, greedily walks toward the target, and drops a layer each time no closer neighbor is found — "zoom out, then zoom in." It's generally the fastest option at a given recall level, at the cost of the highest memory footprint, since the full graph structure sits alongside the vectors themselves.
IVF — Inverted File Index
Clusters the vector space into cells (via k-means), and an inverted list maps each cell to the vectors assigned to it. A query only scans the nearest few cells instead of the whole index. The classic failure mode is the "edge problem" — a true nearest neighbor sitting just across a cell boundary — which is why queries typically probe more than one cell. IVF lands in the middle of the pack: less memory than HNSW, slower per query, and a reasonable default when memory is tighter than latency.
Product Quantization (PQ)
A lossy compression technique: split each vector into sub-vectors, replace each with the ID of its nearest centroid in a learned codebook. A 4,096-byte vector can compress to 64 bytes — a 64× reduction. Used alone, that compression drops recall substantially, which is why PQ is almost always paired with a coarse index (IVF+PQ, or HNSW with quantized vectors) rather than used by itself. It's the go-to choice once a corpus is too large to keep as full-precision vectors in memory at all.
None of these is "the correct one" — they sit on the same recall/speed/memory curve at different points. HNSW for latency-sensitive workloads with memory to spare, IVF as a balanced middle ground, PQ (usually combined with IVF) once scale forces a memory budget the other two can't meet.
Recall, latency, memory, build time — pick which two matter most
Every ANN index exposes a small number of parameters that move these four properties against each other. Recall is never free: every gain in it costs query latency, memory, or both — and build-time investment front-loads some of that cost so it doesn't have to be paid again on every query.
Taking HNSW as the running example, since it's the most widely deployed: M (max links per node) trades memory and build time for recall and speed — on a benchmark million-vector dataset, memory ranged from roughly 0.5GB at M=2 up to 5GB at M=512. efConstruction (candidate pool size at build time) trades build time for graph quality, with little effect on per-query latency once built. efSearch (candidate pool size at query time) is the dial you actually turn in production — it can be changed per query without rebuilding anything, and on that same benchmark it moved recall from roughly 80% at 0.1ms per query up to 100% at around 50ms. IVF's equivalent knobs are nlist (cluster count — more favors speed) and nprobe (cells scanned per query — more favors recall).
A practical tuning method used in production (OpenSearch's own guidance is explicit about this): don't tune each parameter in isolation. Test a short ladder of configurations from sparse to dense, and stop at the first one that clears your recall target — denser graphs reliably trade throughput for recall, so there's rarely a reason to keep pushing once the bar is met.
Pure similarity search is rarely the whole query
Real queries are almost never "find the closest vectors" alone — they're "find the closest vectors that also belong to this tenant, fall in this date range, or match this access-control scope." Combining a metadata filter with vector similarity is one of the harder problems in this layer, and it has two classic, both-flawed solutions plus a more capable third.
| Approach | How it works | Where it breaks |
|---|---|---|
| Pre-filtering | Compute which points match the filter first, then search only that subset | Cost scales with collection size — a permissive filter degrades toward a full brute-force scan |
| Post-filtering | Run the ANN search first, discard non-matching results afterward | A strict filter can eliminate the entire candidate set, forcing unpredictable over-fetching to compensate |
| Filter-aware traversal | The index evaluates the filter during graph traversal, skipping non-matching candidates directly and adding extra edges so filtered subgraphs stay connected | More engineering complexity inside the index itself, but avoids both failure modes above |
This matters well beyond an edge case: multi-tenant SaaS RAG (isolating each customer's data by a tenant filter), recency-scoped retrieval, and access-control-scoped retrieval are all metadata-filtered vector queries in disguise. Get this wrong and you either leak data across a filter boundary that should have been a hard wall, silently return zero results under a strict filter, or eat a severe latency penalty under a permissive one — worth asking a candidate vector database specifically how it filters, not just whether it claims to support filtering at all.
Sharding, and where the market actually sits in 2026
Vector indexes don't scale the way traditional database indexes do. A B-tree removes a deleted key cleanly; an ANN graph or cluster structure degrades instead — deleted nodes are typically tombstoned rather than removed, and index quality decays gradually (and probabilistically) rather than failing outright. Sharding a vector index is also harder than sharding rows by key, because vector data doesn't split cleanly along any obvious boundary — some shards end up overloaded while others sit idle, and rebalancing is a real engineering problem purpose-built systems solve with dedicated architecture (separating recently-written, unindexed data from indexed historical data, auto-balanced across machines) rather than manual shard placement.
The market itself has a clear 2026 shift worth knowing about: vector search is increasingly becoming a feature of databases people already run, rather than a reason to adopt a new, separate system. Postgres extensions (pgvector and pgvectorscale) have matured quickly, and a wave of acquisitions and funding rounds in the Postgres ecosystem reflects that momentum. For workloads under roughly 100 million vectors with moderate query volume, an integrated option is often genuinely sufficient — especially for teams already running that database. Purpose-built systems (Pinecone, Weaviate, Qdrant, Milvus, Chroma) remain the more proven path once you're past billion-scale vector counts, need thousands of queries per second, or need latency guarantees under 50ms — the workloads where vector search is the primary system, not an add-on to one.
The failures that don't throw an error
Almost everything that goes wrong at this layer is silent — the query still returns a plausible-looking, confidently-scored result. Nothing crashes; the numbers just stop meaning what you think they mean.
| Failure mode | What's actually happening |
|---|---|
| Stale index after deletes | Tombstoned nodes still consume the search budget without contributing valid results — one documented case saw recall drop from 0.95 to 0.85 once 20% of rows were deleted but never reindexed |
| Embedding model version mismatch | Documents embedded with one model version, queries with another — the two vector spaces are no longer comparable, but similarity scores still compute and look normal |
| Recall set too low for speed | Aggressive latency-favoring parameters mean true matches never enter the candidate set — no reranker downstream can recover a document that was never retrieved |
| Chunk-boundary or corpus drift | A change to chunking logic or an unrefreshed source corpus produces a second population of vectors that encode similar content differently — retrieval consistency degrades without any code change being the obvious cause |
| Curse of dimensionality | As vector dimensionality grows, distances between points become less discriminative — part of why exact search and simple pruning heuristics stop working well at high dimensions |
| Cost surprises | Memory is the dominant cost driver at scale — full-precision vectors and graph overhead both need to live in RAM for low latency, which is where quantization gets adopted reactively rather than planned for |
One documented production case: retrieval recall degraded from 0.92 to 0.74 over time, with previously top-ranked documents drifting to rank 8, with no code or infrastructure change — the root cause was silent embedding-model drift, invisible to standard monitoring until someone went looking specifically for it.
RAG describes the pattern; this is what makes it real
Everything downstream in the RAG guide — reranking, RRF, HyDE, agentic retrieval — operates on whatever the vector database hands back. Retrieval quality at this layer is a hard ceiling on RAG quality: no amount of clever reranking recovers a document that was never in the candidate set to begin with.
RAG (Layer 02)
The RAG guide covers the query pipeline this index feeds into, and the reranking step that operates on what this layer retrieves.
AI Agent (Layer 04)
Agents that maintain long-term memory across sessions are typically backed by a vector database under the hood — covered next.
Deep dives on AI Agent, MCP, and Guardrails are next — back to the hub to check what's published.
What actually matters when you're choosing one
- Start from scale, not brand recognition: under ~100M vectors and moderate query volume, an option integrated into a database you already run is often enough — evaluate it before adding a new system to operate.
- Check the filtering mechanism specifically, not just whether metadata filtering is supported — pre-filter, post-filter, and in-graph filtering behave very differently under a multi-tenant or access-control-heavy workload.
- Match the distance metric to the embedding model's training setup, and normalize vectors at ingestion if you want cheaper dot-product math with cosine semantics.
- Ask how the index handles updates and deletes, not just inserts — tombstoning behavior and reindex cadence are a common source of silent recall decay.
- Pin your embedding model version deliberately. Never mix embedding versions in one index, and re-embed the full corpus — not a partial subset — on any model upgrade.
- Tune against your own recall target, not the library's defaults — test a short ladder of parameter configurations and stop at the first that clears your bar.
Before you pick a vector database
- Estimate your real scale (vector count, query volume, latency target) before comparing products — the right answer changes completely above and below roughly 100M vectors.
- Write down your metadata filtering requirements explicitly — tenant isolation, access control, recency — before evaluating how each candidate implements filtering.
- Confirm which distance metric your embedding model expects, and don't assume cosine is automatically correct.
- Decide your reindexing cadence before launch — how deletes are handled and how often the index gets rebuilt, not as something to figure out after recall quietly degrades.
- Budget for memory, not just vector count — dimensionality × count × precision, plus index overhead, is the number that actually determines cost at scale.
- Read the RAG guide if you haven't — this layer only matters in the context of the pipeline built on top of it.
A vector database doesn't make retrieval smart. It makes retrieval fast enough that the smart part — chunking, hybrid search, reranking — gets to run at all.
What this guide draws on
Read in full and used to shape the structure and claims of this guide.
| Source | Publisher | Used for |
|---|---|---|
| What is a Vector Database & How Does it Work? | Pinecone | Why brute-force search fails at scale; what a full vector database adds beyond a bare ANN index (§00) |
| Hierarchical Navigable Small Worlds (HNSW) | Pinecone | HNSW graph structure, M/efConstruction/efSearch parameters, benchmark recall/latency/memory numbers (§02–03) |
| Nearest Neighbor Indexes for Similarity Search | Pinecone | IVF clustering and the nlist/nprobe trade-off, head-to-head comparisons against flat search and HNSW (§02) |
| Product Quantization | Pinecone | PQ codebook compression mechanics and combined IVF+PQ trade-offs (§02) |
| Vector Similarity Explained | Pinecone | Cosine similarity, dot product, and Euclidean distance — formulas and when each applies (§01) |
| A Practical Guide to Selecting HNSW Hyperparameters | OpenSearch | Practical parameter-tuning method — testing a configuration ladder against a recall target (§03) |
| Pre-Filtering vs Post-Filtering (and Why Qdrant Does Neither) | Qdrant | Metadata filtering strategies and their failure modes, filter-aware graph traversal (§04) |
| Why Manual Sharding is a Bad Idea for Vector Databases | Milvus | Sharding and scaling challenges specific to vector indexes (§05) |
| What's Changing in Vector Databases in 2026 | Actian | 2026 market landscape — consolidation toward vector search as a built-in database feature (§05) |
| pgvector in Production: Five Failure Patterns | Perun Engineering | Concrete pgvector production failure patterns (§06) |
| The RAG Freshness Problem: How Stale Embeddings Silently Wreck Retrieval Quality | TianPan.co | Embedding staleness, model-version drift, and chunk-boundary drift as silent failure modes (§06) |
Compiled 2026. Specific benchmark numbers come from the cited sources' own test setups — treat them as illustrative of direction and rough magnitude, not guarantees for any particular dataset or hardware.