Choosing a Vector Database for RAG

November 19, 2025 (updated July 27, 2026) / Arvid Andersson

Retrieval-augmented generation (RAG) is the standard pattern for grounding LLM responses in your own data. At the core of most RAG pipelines is a vector database that stores embeddings and handles similarity search. The options range from managed cloud services to lightweight libraries you embed in your application. This post covers the trade-offs.

Why a dedicated vector database?

You can add vector search to PostgreSQL with pgvector, or use built-in vector features in Elasticsearch. For many applications, that's enough. A dedicated vector database becomes valuable when you need to handle millions of high-dimensional vectors with consistent low-latency queries, or when you need features like filtering, multi-tenancy, and hybrid search (combining vector similarity with keyword matching) at scale.

Managed cloud services

Pinecone is the most established managed vector database. It handles indexing, scaling, and infrastructure. You interact through an API, and Pinecone manages the rest. It supports metadata filtering, namespaces for multi-tenancy, and scales to billions of vectors. Pricing is serverless by default (billed on reads, writes, and storage; the older pod-based plans are legacy), which can get expensive at scale but means zero operational overhead.

Zilliz offers a managed cloud version of Milvus (the open-source vector database). If you want the flexibility of Milvus without managing the infrastructure, Zilliz provides that. It supports the same feature set: multiple index types, hybrid search, and GPU-accelerated indexing.

Turbopuffer focuses on cost efficiency by treating object storage as the source of truth, with NVMe and memory acting as caches above it. The trade-off is visible in their own published numbers: a first query against a namespace reads object storage and is slow (p50 874ms for 1M documents), while subsequent cached queries on that node are fast (p50 14ms). That suits workloads where most queries hit a working set and you would rather not pay to keep everything in RAM. Pricing moved in your favour through 2026 (the entry tier minimum dropped to $16/month in June 2026), though there is still no free tier.

Amazon S3 Vectors went generally available in December 2025 and is worth knowing about even though it is not a database in the usual sense. It adds native vector storage and querying to S3 itself, supporting up to 2 billion vectors per index, and AWS positions it as cutting the cost of storing and querying vectors by up to 90% compared with conventional approaches. The trade-off is latency: AWS describes frequent queries as around 100ms or less, which is slower than a warm dedicated index. It plugs into Bedrock Knowledge Bases and OpenSearch, so it fits cost-sensitive or infrequently queried corpora rather than latency-critical serving paths.

Open-source, self-hosted

Qdrant is written in Rust and designed for production workloads. It supports filtering during search (not just post-filtering), which matters for multi-tenant applications where you need to scope results by user or organization. Qdrant also offers a managed cloud service, so you can start self-hosted and move to managed later.

Weaviate stands out for its built-in vectorization modules. Instead of generating embeddings externally and storing them, you can configure Weaviate to call an embedding model (OpenAI, Cohere, HuggingFace) as part of the ingestion pipeline. It also supports hybrid search combining vector and BM25 keyword scoring. Available as open-source or managed cloud.

Milvus is one of the oldest purpose-built vector databases. It supports multiple index types (IVF, HNSW, DiskANN), GPU-accelerated search, and scales to billions of vectors. The trade-off is operational complexity, as a production deployment still involves several components (etcd for metadata, object storage, and a message queue). If that is the part putting you off, Milvus Lite runs in-process via `pip install pymilvus` against a local file, which is useful for prototyping, though it is limited to FLAT indexing and drops partitions, users, and roles. Note that the docs site now surfaces 3.0 pages, but 3.0 is still a beta at the time of writing and 2.6.x is the stable line.

Lightweight and embedded

Chroma is designed for simplicity. It runs in-process (no separate server needed), making it ideal for prototyping, local development, and small-scale RAG applications. You can get started with a few lines of Python. For production, Chroma offers a client-server mode and a cloud service.

LanceDB takes a serverless approach built on the Lance columnar format. It runs embedded in your application (Python, JavaScript, Rust) with no server to manage, and stores data in object storage, making it cost-effective for large datasets. Note that the project now markets itself as a multimodal lakehouse rather than a vector database, with the emphasis on training-data workflows like curation and feature engineering. It still works well as embedded vector search, but the roadmap is pointed at a broader problem, and LanceDB Cloud is still labelled public beta.

PostgreSQL-based

pgvector adds vector similarity search to your existing Postgres database. If your data is already in PostgreSQL, keeping vectors alongside it avoids adding infrastructure, and it performs well up to a few million vectors. For many RAG applications at moderate scale, that is the simplest architecture. It ships two index types, ivfflat and hnsw, and is released under the PostgreSQL License. Development is steady rather than fast moving: releases through the first half of 2026 were mostly correctness fixes, including HNSW vacuuming and index corruption bugs (0.8.5, July 2026).

pgvecto.rs was a separate Rust extension in the same space, but development stopped in February 2025 and its maintainers moved to a successor, VectorChord, which builds on pgvector with faster index builds and disk-based storage. One licensing detail matters before you adopt it: VectorChord is dual licensed under AGPLv3 or the Elastic License v2, not a permissive licence like pgvector's. The Elastic option restricts offering it as a managed service, so check which of the two you intend to rely on. New projects that want permissive licensing should stay on plain pgvector.

Comparison

How to choose

If you're already on PostgreSQL and your dataset is under a few million vectors, start with pgvector. You avoid adding infrastructure and keep your data in one place. VectorChord is the option if you outgrow pgvector's index build times, as long as its AGPLv3 or Elastic licensing works for you. For prototyping and local development, Chroma gets you started fastest.

If cost per vector matters more than latency, the cheap-storage end of the market got more crowded through 2026. Turbopuffer and Amazon S3 Vectors both lean on object storage to cut the cost of holding vectors you do not query constantly, and both trade first-query latency for it. Benchmark against your own access pattern rather than a published p50, because how much of your working set stays cached is what actually decides the number you get.

For production RAG at scale, the choice depends on your operational preferences. Pinecone if you want fully managed with no infrastructure to think about. Qdrant or Weaviate if you want open-source with the option to self-host or use managed cloud. Milvus if you need GPU acceleration or have very large datasets.

The embedding model you choose matters as much as the database. Most vector databases are agnostic to the embedding model, so you can switch models without changing your database. Focus on getting the retrieval quality right first, then optimize the infrastructure.