Benchmarks Lie — Vector DBs Deserve a Real Test

What this post added

Introduced VDBBench, an open-source benchmarking tool for vector databases. VDBBench simulates production environments by using modern datasets (e.g., Wikipedia, BioASQ, C4, MSMarco V2) generated by state-of-the-art embedding models, focusing on production-relevant metrics such as P95/P99 latency, sustainable throughput, and recall balanced with performance. It employs a two-phase testing methodology (Serial Test and Concurrency Test) and evaluates performance across static collection, filtering (with varying selectivity), and streaming (search-while-inserting) scenarios. The tool aims to provide accurate and production-relevant insights for capacity planning and system design.