What is it?
Once content is converted into What are Embeddings in AI?, you need somewhere to keep the vectors and a fast way to find the closest ones to a new query. A vector database (or vector store) does both: it stores each vector with an ID and metadata (source, date, permissions, text) and exposes a similarity search operation.
Some products are purpose-built vector databases; many conventional databases and search engines also offer vector types or extensions (for example the pgvector extension for PostgreSQL). Which is right depends on scale and what else you need.
Why does it matter?
Comparing a query against every stored vector is exact but slow at large scale. Vector databases build indexes that make search fast, and they support the practical features a real application needs: filtering by metadata ("only documents this user may see", "only 2024 policies"), updates and deletions, and persistence. They are the retrieval layer of most What is RAG (Retrieval-Augmented Generation)? systems.
How does it work?
Exact search compares the query with every vector — simple and perfectly accurate, fine for thousands or even many tens of thousands of items.
Approximate nearest-neighbour (ANN) search trades a little accuracy for large speed gains by organising vectors in an index structure. Well-known families include graph-based indexes (such as HNSW), clustering-based indexes (IVF) and quantisation (compressing vectors). Settings let you tune the balance between speed, memory and recall (how often the true nearest neighbours are actually returned).
A typical pipeline:
- Split documents into chunks (Chunking: How to Split Documents for RAG).
- Embed each chunk and store vector + text + metadata.
- Embed the query and search, optionally with metadata filters.
- Return top-k chunks to the application, which places them in the prompt.
Example
Internal wiki assistant: each chunk is stored with metadata { space: "HR", updated: "…", allowedGroups: ["all-staff"] }. A query from an engineer filters to spaces they can access, then searches by similarity. Metadata filtering is what stops the assistant from retrieving a document the user should not see — an access-control concern that must be enforced at retrieval time, not trusted to the prompt (AI Privacy and Security: What to Know Before You Paste Data).
When should I use it?
You likely do not need a dedicated vector database if you have a small corpus (an in-memory array, a file, or a database table with a vector column may be enough), if you already run a database with vector support, or if the data is mostly structured and exact-match.
Consider one when you have large collections, frequent updates, strict latency needs, or need combined vector + keyword + metadata search at scale. See Vector Database vs Traditional Database.
Common mistakes
- Adopting a new database for a tiny dataset. Extra infrastructure, extra cost.
- Ignoring metadata and permissions.
- Judging by speed alone. A fast index that misses relevant results harms answer quality; measure recall on your data (How to Evaluate AI Systems and Outputs).
- Not planning re-indexing when you change embedding model or chunking.