TechPrufer
Get started
Milvus
Milvus logoMilvus · Feature trail

Benchmarking and Performance Analysis

18 posts · Nov 2020 – Mar 2026

What this trail covers

Milvus continues to evolve its benchmarking capabilities with VDBBench, an open-source tool designed to simulate production realities. VDBBench focuses on realistic datasets, production-focused metrics (P95/P99 latency, sustainable throughput, recall), and testing methodologies that reflect real-world scenarios like continuous ingestion, filtering, and streaming data. This aims to provide engineers with trustworthy data for infrastructure decisions, moving beyond traditional, often misleading, benchmarks. This post also details optimizations in the Milvus write path to reduce memory movement overhead.

All Milvus trails
  1. Troubleshooting a Search Slowdown After Upgrading Milvus: Lessons from the WPS TeamThis post details the WPS engineering team's troubleshooting of a 3-5x search latency increase after upgrading Milvus from 2.2.16 to 2.5.16. It covers the motivation for the upgrade (stability issues with `planparserv2.HandleCompare`), the use of `milvus-backup` for hot backup and restore across versions, the deployment of a new Milvus 2.5.16 cluster using Helm with `mmap.enabled: false`, and the subsequent investigation into the performance degradation. The post highlights the importance of verifying backups, building new clusters for major upgrades, and keeping old clusters running during cutover.
  2. How to Safely Upgrade from Milvus 2.5.x to Milvus 2.6.xThis post details the architectural evolution from Milvus 2.5 to 2.6, specifically the introduction of a dedicated StreamingNode to separate real-time data processing from batch operations handled by QueryNode and DataNode. It outlines key new features in 2.6 such as RaBitQ quantization, BM25 full-text search, JSON path indexing, and tiered storage. A step-by-step upgrade path from 2.5.x to 2.6.x is provided, including the order of component upgrades (Streaming Node, MixCoord, Query Node, Data Node, Proxy) and notes on operational impacts like DDL unavailability.
  3. How to Debug Slow Search Requests in MilvusThis post details how to debug slow search requests in Milvus by leveraging metrics (Service Quality → Slow Query, Search Latency, Query Node → Search Latency by Phase) and logs (marked with [Search slow]). It analyzes root causes including heavy workload (large NQ, high QPS), inefficient filtering (poorly conducted expressions, lack of scalar indexes, JSON filters, strict consistency), improper vector index choice (HNSW, IVF, DiskANN, MINHASH_LSH for binary vectors), and runtime/environment conditions (compaction, upserts, version-specific issues). It provides actionable fixes for each cause and outlines best practices for prevention, such as resource allocation, proactive alerts, efficient filters, batch upserts, and indexing all filter fields.
  4. Hands-On with VDBBench: Benchmarking Vector Databases for POCs That Match ProductionThis post introduces VDBBench 1.0, an open-source tool for benchmarking vector databases. It details the importance of using custom datasets and production-like workloads for POCs, contrasting this with outdated benchmark datasets. The post provides a comprehensive guide to installing and using VDBBench, including data preparation steps with automated conversion scripts for CSV/NPY to Parquet format, ground truth computation using FAISS, and configuration via a web interface. It highlights key metrics like P95/P99 latency, sustained throughput, and recall accuracy.
  5. https://zilliz.com/blog/vdbbench-1-0-benchmarking-with-your-real-world-production-workloadsThis post announces VDBBench 1.0, an open-source benchmarking tool for Milvus. It emphasizes simulating real-world production workloads by focusing on realistic datasets, production-focused metrics (P95/P99 latency, sustainable throughput, recall), and testing methodologies that reflect actual usage scenarios such as continuous ingestion, filtering, and streaming data. The goal is to provide engineers with reliable data for infrastructure decisions, moving away from misleading traditional benchmarks.
  7. We Benchmarked 20+ Embedding APIs with Milvus: 7 Insights That Will Surprise YouThis post benchmarks the latency of over 20 embedding APIs when integrated with Milvus's TextEmbedding Function. It details the impact of network geography, model size, token length, and batch size on API performance, and compares cloud API latency with local inference options. The post also validates the minimal overhead introduced by Milvus's TextEmbedding Function and provides optimization tips for RAG embedding performance.
  8. Benchmarks Lie — Vector DBs Deserve a Real TestThis post introduces VDBBench, an open-source benchmarking tool designed to simulate production realities for vector databases. It highlights the limitations of traditional benchmarks (outdated data, vanity metrics, oversimplified scenarios) and details VDBBench's approach to address these by using modern datasets, production-focused metrics (P95/P99 latency, sustainable throughput, recall), and realistic testing methodologies that include static collection, filtering, and streaming scenarios. The post emphasizes the importance of these realistic tests for making informed infrastructure decisions.
  9. Introducing the Milvus Sizing Tool: Calculating and Optimizing Your Milvus Deployment ResourcesThis post introduces the Milvus Sizing Tool, a calculator that generates recommended resource estimations for Milvus deployments based on user-defined parameters such as the number of vectors, dimensions, index type, scalar field data size, segment size, and preferred deployment mode. It elaborates on the trade-offs of various index types (HNSW, FLAT, IVF_FLAT, IVF_SQ8, ScaNN, DiskANN, GPU indexes, HNSW variants) concerning memory usage, disk space, query speed, and accuracy. Recommendations for segment sizes are provided based on query node memory. The post also briefly touches upon message queue selection (Pulsar vs. Kafka) and highlights enterprise optimizations available in Zilliz Cloud. Advanced configuration tips for developers are included, emphasizing the use of multiple index types, memory allocation considerations, performance testing, and scaling.
  10. What Milvus Users Taught Us in 2024This post analyzes user discussions from 2024 to identify common challenges and provide guidance on Milvus usage. It categorizes these insights into Performance Optimization, Deployment Strategies, and Data Management. For deployment, it details the differences and use cases for Milvus Lite, Standalone, and Distributed, introduces the Milvus Resource Sizing Tool, and discusses index and distance metric selection (in-memory, on-disk, GPU indexes, and distance metrics). It also addresses troubleshooting distributed deployments and Windows deployment. For performance, it explains how to measure performance using metrics and the WebUI, how to observe internal state, and provides insights into complex internal aspects like segment sealing and memory management. Finally, it touches upon the selection of embedding models.
  11. Introducing PyMilvus Integration with Embedding ModelsThis post introduces the integration of embedding and reranking models within the PyMilvus SDK, enabling users to generate dense and sparse vectors and apply reranking for improved search relevance. It demonstrates practical usage with Milvus Lite, covering dense embeddings using `DefaultEmbeddingFunction` (e.g., `all-MiniLM-L6-v2`), sparse embeddings with `BM25EmbeddingFunction` (including fitting and saving model parameters), and reranking with `JinaRerankFunction`. The post details the installation of the `pymilvus[model]` package and provides code examples for each integration.
  12. Optimize Vector Databases, Enhance RAG-Driven Generative AIThis post details optimizations in Milvus's write path to reduce memory movement overhead in the datanode buffer write. It describes how the merge logic allocates a new buffer to hold old and new data, leading to progressively larger buffers and slower data movement. The post also discusses the use of Linux 'perf' and Intel® vTune™ Profiler for low-level performance analysis, and highlights the importance of realistic datasets and benchmarking frameworks like VectorDBBench and vector-db-benchmark for evaluating vector database performance.
  13. Milvus 2.3.4: Faster Searches, Expanded Data Support, Improved Monitoring, and MoreMilvus 2.3.4 introduces access logs for improved monitoring of gRPC interactions, enabling performance analysis and security auditing. It adds support for importing Parquet files, enhancing data processing efficiency and making Parquet the default output for the BulkWriter tool. Search performance is significantly improved with the implementation of a binlog index on growing segments, leading to up to tenfold faster searches. The system now supports up to 10,000 collections/partitions, addressing needs for nuanced data organization in use cases like knowledge base management and multi-tenancy. Other enhancements include reduced memory usage during data retrieval and variable-length data handling, refined error messaging, accelerated loading speed, and improved query shard balance.
  14. Milvus 2.2.8: Better Query Performance, 20% Higher ThroughputMilvus 2.2.8 addresses peak memory consumption during collection loading by optimizing memory usage. It enables expanded querying scenarios by allowing QueryNode to support plugins. The compaction algorithm has been enhanced for faster convergence and improved query performance. The default number of collection shards has been reduced from two to one to improve resource saving and querying performance. The query grouping algorithm has been further enhanced, leading to a 20% increase in throughput.
  15. Milvus at Its Best: Exploring v2.2 to v2.2.6This post details the evolution of Milvus from v2.2 to v2.2.6, highlighting new features, bug fixes, and performance optimizations. Key additions include bulk inserts, query result pagination, RBAC, quotas and limits, collection-level TTL, DiskANN for disk-based ANNS, and a backup tool in v2.2. Subsequent releases (v2.2.1-v2.2.6) focused on bug fixes, performance improvements (e.g., search performance, scheduler optimization), security enhancements (TLS, Pulsar tenant/auth), availability features (rolling upgrade, coordinator HA), resource management (resource grouping), and usability (collection renaming). Specific critical issues resolved in v2.2.6 include DataCoord GC failure and binlog data recycling problems.
  16. Open Source Software (OSS) Quality Assurance - A Milvus Case StudyThis post details the Quality Assurance framework adopted in developing Milvus, covering its system architecture, test modules (unit, function, deployment, reliability, performance, stability, configuration, compatibility), and issue management workflow. It emphasizes the importance of QA throughout the R&D lifecycle and introduces tools and methods for improving QA efficiency, including chaos engineering with Chaos Mesh for reliability testing and pytest for function testing.
  17. Quickly Test and Deploy Vector Search Solutions with the Milvus 2.0 BootcampThis post introduces the Milvus 2.0 bootcamp, which provides updated guides and code examples for testing and deploying vector search solutions. It includes benchmark tests for 1M and 100M datasets, and solutions for various use cases like image similarity search. The image similarity search example details steps for setting up Milvus and Redis servers, creating a collection with an IVF_SQ8 index, loading data, generating vector embeddings using Resnet-18, inserting vectors into Milvus, and performing similarity searches. It also covers deployment options in different environments.
  18. Milvus performance on AVX-512 vs. AVX2This post details performance comparisons of Milvus on AVX-512 versus AVX2 instruction sets. It outlines system configuration, Milvus parameters (cache size, nlist, nprobe), and the SIFT10M dataset used for testing. Performance results are presented for IVF_FLAT, IVF_SQ8, and HNSW index types, showing consistent speed improvements with AVX-512. The analysis concludes that while AVX-512 offers theoretical benefits, real-world performance gains are influenced by other system tasks. It also highlights IVF_SQ8's efficiency over IVF_FLAT due to memory bandwidth constraints.