Benchmarks Lie — Vector DBs Deserve a Real Test

This post introduces VDBBench, an open-source benchmarking tool designed to simulate production realities for vector databases. It highlights the limitations of traditional benchmarks (outdated data, vanity metrics, oversimplified scenarios) and details VDBBench's approach to address these by using modern datasets, production-focused metrics (P95/P99 latency, sustainable throughput, recall), and realistic testing methodologies that include static collection, filtering, and streaming scenarios. The post emphasizes the importance of these realistic tests for making informed infrastructure decisions.