VectorDBBench(VDBBench): A Benchmark Tool for VectorDB
What is VDBBench
VDBBench is not just an offering of benchmark results for mainstream vector databases and cloud services, it's your go-to tool for the ultimate performance and cost-effectiveness comparison. Designed with ease-of-use in mind, VDBBench is devised to help users, even non-professionals, reproduce results or test new systems, making the hunt for the optimal choice amongst a plethora of cloud services and open-source vector databases a breeze.
Understanding the importance of user experience, we provide an intuitive visual interface. This not only empowers users to initiate benchmarks at ease, but also to view comparative result reports, thereby reproducing benchmark results effortlessly. To add more relevance and practicality, we provide cost-effectiveness reports particularly for cloud services. This allows for a more realistic and applicable benchmarking process.
VectorDBBench provides repeatable workloads for insertion, vector search, filtered search, and full-text search. Its registered vector datasets range from classic SIFT and GIST corpora to Cohere embeddings, OpenAI embeddings generated from the public C4 dataset, and the VIBE and VDBBench multimodal datasets hosted on Hugging Face. Together, these datasets cover different corpus sizes, dimensions, modalities, distance metrics, and query distributions.
Prepare to delve into the world of VDBBench, and let it guide you in uncovering your perfect vector database match.
VDBBench is sponsored by Zilliz，the leading opensource vectorDB company behind Milvus. Choose smarter with VDBBench - start your free test on zilliz cloud today!
Leaderboard: https://zilliz.com/benchmark
🎈 Announcement 🎈
June 2026 update: Full Text Search has landed in VectorDBBench. We now benchmark BM25-style retrieval across supported backends, starting with MS MARCO and HotpotQA datasets, payload profiles, recall, QPS, and load metrics ready to compare. See the VectorDBBench Full Text Search Release Note for the full rollout details and caveats.
September 2026 update: VectorDBBench can now run registered datasets hosted
on Hugging Face through the existing Performance case. The initial catalog
includes all 24 VIBE HDF5 datasets and VDBBench multimodal Parquet datasets at
1M, 10M, and 100M corpus vectors.
vectordbbench milvushnsw \
--case-type Performance \
--dataset-name glove-200-cosine \
--k 100
See the Hugging Face datasets release note for the complete catalog, data flow, provenance, constraints, and configuration details.
Quick Start
Prerequirement
python >= 3.11
Install
Install vectordb-bench with only PyMilvus
pip install vectordb-bench
Install the specific database client
pip install 'vectordb-bench[pinecone]'
All the database client supported
| Optional database client | install command |
|---|---|
| pymilvus, zilliz_cloud (default) | pip install vectordb-bench |
| qdrant | pip install vectordb-bench[qdrant] |
| pinecone | pip install vectordb-bench[pinecone] |
| weaviate | pip install vectordb-bench[weaviate] |
| elastic, aliyun_elasticsearch | pip install vectordb-bench[elastic] |
| pgvector, pgvectorscale, pgdiskann, alloydb, vectorchord, lakebase_vector | pip install vectordb-bench[pgvector] |
| pgvecto.rs | pip install vectordb-bench[pgvecto_rs] |
| redis | pip install vectordb-bench[redis] |
| memorydb | pip install vectordb-bench[memorydb] |
| chromadb | pip install vectordb-bench[chromadb] |
| cockroachdb | pip install vectordb-bench[cockroachdb] |
| awsopensearch | pip install vectordb-bench[opensearch] |
| aliyun_opensearch | pip install vectordb-bench[aliyun_opensearch] |
| mongodb | pip install vectordb-bench[mongodb] |
| tidb | pip install vectordb-bench[tidb] |
| vespa | pip install vectordb-bench[vespa] |
| oceanbase | pip install vectordb-bench[oceanbase] |
| hologres | pip install vectordb-bench[hologres] |
| tencent_es | pip install vectordb-bench[tencent_es] |
| alisql | pip install vectordb-bench[alisql] |
| polardb | pip install vectordb-bench[polardb] |
| doris | pip install vectordb-bench[doris] |
| zvec | pip install vectordb-bench[zvec] |
| endee | pip install vectordb-bench[endee] |
| lindorm | pip install vectordb-bench[lindorm] |
| volc_mysql | pip install vectordb-bench[volc_mysql] |
| adbpg | pip install vectordb-bench[adbpg] |
Run
init_bench
OR:
Run from the command line.
vectordbbench [OPTIONS] COMMAND [ARGS]...
To list the clients that are runnable via the commandline option, execute: vectordbbench --help
$ vectordbbench --help
Usage: vectordbbench [OPTIONS] COMMAND [ARGS]...
Options:
--help Show this message and exit.
Commands:
pgvectorhnsw
pgvectorivfflat
vectorchordrq
volcmysqlhnsw
test
weaviate
To list the options for each command, execute vectordbbench [command] --help
Use --note or --note-file to preserve deployment, resource, client, network, and constraint context in each result JSON under task_config.db_config.note. The options are mutually exclusive. Prefer --note-file for structured or multiline context, and never include credentials, tokens, or sensitive connection details.
vectordbbench zillizautoindex \
--note-file ./run-context.json \
<other options>
$ vectordbbench pgvectorhnsw --help
Usage: vectordbbench pgvectorhnsw [OPTIONS]
Options:
--config-file PATH Read configuration from yaml file
--drop-old / --skip-drop-old Drop old or skip [default: drop-old]
--load / --skip-load Load or skip [default: load]
--search-serial / --skip-search-serial
Search serial or skip [default: search-
serial]
--search-concurrent / --skip-search-concurrent
Search concurrent or skip [default: search-
concurrent]
--case-type [CapacityDim128|CapacityDim960|Performance768D100M|Performance768D10M|Performance768D1M|Performance768D10M1P|Performance768D1M1P|Performance768D10M99P|Performance768D1M99P|Performance1536D500K|Performance1536D5M|Performance1536D500K1P|Performance1536D5M1P|Performance1536D500K99P|Performance1536D5M99P|Performance1536D50K]
Case type
--db-label TEXT Db label, default: date in ISO format
[default: 2024-05-20T20:26:31.113290]
--note TEXT Run context stored with each result
[default: ""]
--note-file FILE Read run context from a UTF-8 text file
--dry-run Print just the configuration and exit
without running the tasks
--k INTEGER K value for number of nearest neighbors to
search [default: 100]
--concurrency-duration INTEGER Adjusts the duration in seconds of each
concurrency search [default: 30]
--num-concurrency TEXT Comma-separated list of concurrency values
to test during concurrent search [default:
1,10,20]
--concurrency-timeout INTEGER Timeout (in seconds) to wait for a
concurrency slot before failing. Set to a
negative value to wait indefinitely.
[default: 3600]
--user-name TEXT Db username [required]
--password TEXT Db password