Definition
Vector database is a data system whose primary query is "the rows whose embedding is closest to this vector, among rows matching this filter". It stores each row as an id, one or more dense vectors, structured metadata (often called a payload) and usually the source text. It keeps an approximate nearest neighbour index over the vectors and provides the usual database guarantees around them: durable writes, updates and deletes, replication and horizontal scaling. Milvus (Wang et al., SIGMOD 2021) describes one purpose-built design; Pan, Wang and Li survey the field in "Survey of Vector Database Management Systems" (VLDB Journal, 2024).
In a RAG system (see RAG architecture) the vector database sits between the indexing pipeline and the retriever. Offline, documents are cut into chunks (Document chunking), embedded (Embedding models) and upserted. Online, the query is embedded with the same model and sent as a vector plus a filter such as tenant, language, date or access group. The returned chunks then go to Reranking or straight into the prompt. The same store backs Semantic caching, recommendation and deduplication.
An ANN library such as FAISS or hnswlib gives you only the index: an in-process structure that you build, save and query. A vector database wraps that index in a server. It adds a schema, persistence and crash recovery, metadata indexes, filtered search, concurrent writes, access control, backups and distribution. General-purpose databases have converged on the same feature set: PostgreSQL with pgvector, Elasticsearch and OpenSearch with dense-vector fields. Which one to use depends far more on your existing stack, data size and filter patterns than on any single benchmark.
What is stored, and how it is compared
- query and stored embedding, from the same model and version
- squared L2 distance; for unit vectors it ranks rows exactly like cosine and inner product
Every collection fixes a dimension and a metric at creation time, and the metric must be the one the embedding model was trained with. Normalising vectors at write and query time makes the three common metrics interchangeable. Memory is dominated by the vectors: bytes for bytes per component, plus the index (about bytes per vector for HNSW layer-0 links; see Approximate nearest neighbor search (HNSW, IVF, PQ)).
Metadata filtering
Filters are where vector search and databases collide. An ANN index orders rows by distance only, and the filter must be applied somewhere. Let selectivity be the fraction of rows that pass. There are three basic plans:
- 1Post-filter. Ask the index for nearest rows, then drop those that fail the filter. About survive, so returning rows needs . A 1% filter with needs about 1,000 candidates.
- 2Pre-filter. Use a metadata index (B-tree, inverted index, bitmap) to get the matching ids, then compute exact distances for all of them. Cost is distances and recall is 1. This is cheap when is small and hopeless when it is large.
- 3In-filter (filtered traversal). Check a filter bitmap while walking the index and only accept matching rows, continuing until are found. On IVF this means probing more lists; on HNSW it means skipping non-matching nodes. Very selective filters can strand a graph search in regions with no matches, so some engines add extra edges per filter value or fall back to brute force.
- selectivity: fraction of rows passing the filter, assumed independent of vector position
- candidates fetched from the ANN index before filtering (post-filter)
Real engines combine these under a query planner that estimates from metadata statistics. They pre-filter with an exact scan when few rows match, filter during traversal otherwise, and skip filtering when everything matches. If the filter correlates with vector position (for example, all French documents cluster together), the independence assumption breaks in either direction, which is another reason to measure.
Writes: CRUD and index maintenance
ANN indexes are built for reads. IVF needs trained centroids, PQ needs trained codebooks, and HNSW inserts are expensive and deletes do not shrink the graph. Databases therefore borrow the log-structured design. Writes are appended to a write-ahead log for durability and buffered in a small mutable segment that is searched by brute force. When the buffer fills, it is sealed and an index is built for it in the background. A query searches every segment and merges the results.
A delete marks the id in a tombstone bitmap that every search treats as a filter. An update is a delete plus an insert, because the old vector sits inside an index structure. Compaction later merges small segments and drops tombstoned rows, which also restores recall. Graphs with many deleted nodes route searches through dead ends. Because a new row reaches the index asynchronously, systems expose read consistency as a setting. Milvus, for example, offers Strong, Bounded staleness, Session and Eventually levels, trading freshness for latency.
Scaling out: shards and replicas
A collection larger than one machine is split into shards, usually by hashing the id. A query is scattered to every shard; each returns its local top- and the coordinator merges candidates. Each shard must return , not , because all of the true top- may live on one shard. Latency is set by the slowest shard, so p99 grows with shard count. Replicas of each shard add read throughput and availability. Partitioning by tenant instead of by hash turns a tenant filter into routing: one shard, no wasted work, and per-tenant deletes become dropping a partition.
pgvector as a concrete reference (PostgreSQL extension; as of the 0.8 releases, 2024)
- Distance operators
<->L2,<#>negative inner product,<=>cosine distance;ORDER BY embedding <=> $1 LIMIT 10triggers the ANN index.- Index types
ivfflat(needs data before building;ivfflat.probesdefaults to 1) andhnsw(added in 0.5.0;m= 16,ef_construction= 64,hnsw.ef_search= 40 by default).- Filtering caveat
- The index scan is approximate and the
WHEREclause is applied to its output. The documentation's own example: a condition matching 10% of rows withef_search= 40 leaves about 4 rows. 0.8.0 added iterative index scans to keep fetching. - Transactions
- Rows, metadata and vectors share PostgreSQL's MVCC, WAL, backups and replication, so an upsert is just
INSERT … ON CONFLICT.
Worked example: sizing 10M chunks
| Item | Formula | Size |
|---|---|---|
| float32 vectors | 10⁷ × 768 × 4 B | 30.72 GB |
| HNSW layer-0 links | 10⁷ × 32 × 4 B | 1.28 GB |
| Total per copy | 30.72 + 1.28 | 32.0 GB |
| 4 hash shards | 32.0 / 4 | 8.0 GB per shard |
| 2 replicas of everything | 32.0 × 2 | 64.0 GB cluster RAM |
| int8 scalar quantization | 10⁷ × 768 × 1 B | 7.68 GB of vectors |
The vectors, not the graph, dominate. Scalar quantization to int8 cuts them by 4×, or PQ by 32× or more, at some recall cost that a re-ranking step on full vectors (kept on disk) recovers. Metadata and the source text often add as much again, so many deployments keep the text in a separate store and return only ids.
1CREATE EXTENSION IF NOT EXISTS vector;23CREATE TABLE chunks (4 id text PRIMARY KEY, -- "doc-42#3": document id + chunk number5 tenant text NOT NULL,6 year int,7 body text,8 embedding vector(768)9);10CREATE INDEX ON chunks USING hnsw (embedding vector_cosine_ops) WITH (m = 16, ef_construction = 64);11CREATE INDEX ON chunks (tenant, year); -- metadata index for the planner1213-- upsert = delete + insert under the hood; same statement for new and changed chunks14INSERT INTO chunks VALUES ($1, $2, $3, $4, $5)15ON CONFLICT (id) DO UPDATE SET body = EXCLUDED.body, embedding = EXCLUDED.embedding;1617SET hnsw.ef_search = 100;18SET hnsw.iterative_scan = relaxed_order; -- keep scanning until LIMIT rows pass the filter19SELECT id, body, embedding <=> $1 AS cosine_distance20FROM chunks21WHERE tenant = 'acme' AND year >= 202222ORDER BY embedding <=> $123LIMIT 10;Use when
- +One query combines semantic similarity with exact metadata constraints and access control.
- +Durable upserts and deletes keep the index in step with the source documents.
- +Sharding and replicas scale to billions of vectors and high QPS.
Avoid when
- −Recall becomes a property of the plan: filters, segment count and tombstones all move it.
- −Fresh writes reach the ANN index asynchronously; strong consistency costs latency.
- −RAM-resident indexes are expensive; quantized or disk-based indexes trade recall or latency for cost.
Check yourself
A filter matches 2% of rows. With post-filtering, about how many ANN candidates must you fetch to expect 10 results?
Check yourself
A collection is hash-sharded 4 ways and the query asks for top-10. What must each shard return for the merged result to equal an unsharded search?
Remember
- ✓A row is id + vector(s) + metadata (+ text); the metric and dimension are fixed per collection and must match the embedding model.
- ✓Filters: post-filter needs candidates; pre-filter costs exact distances; in-filter checks a bitmap during traversal. Planners choose by estimated selectivity.
- ✓Writes go WAL → growing segment (flat scan) → sealed, indexed segment; deletes are tombstones until compaction; updates are delete + insert.
- ✓Hash shards: scatter the query, each shard returns its own top-k, merge S·k; tenant partitioning turns filters into routing.
- ✓Memory is mostly vectors (N·d·4 B); quantize, and re-rank with full vectors, to cut it.