Benchmarking and Performance Analysis

3/18/2026

This post details the WPS engineering team's troubleshooting of a 3-5x search latency increase after upgrading Milvus from 2.2.16 to 2.5.16. It covers the motivation for the upgrade (stability issues with `planparserv2.HandleCompare`), the use of `milvus-backup` for hot backup and restore across versions, the deployment of a new Milvus 2.5.16 cluster using Helm with `mmap.enabled: false`, and the subsequent investigation into the performance degradation. The post highlights the importance of verifying backups, building new clusters for major upgrades, and keeping old clusters running during cutover.

12/25/2025

This post details the architectural evolution from Milvus 2.5 to 2.6, specifically the introduction of a dedicated StreamingNode to separate real-time data processing from batch operations handled by QueryNode and DataNode. It outlines key new features in 2.6 such as RaBitQ quantization, BM25 full-text search, JSON path indexing, and tiered storage. A step-by-step upgrade path from 2.5.x to 2.6.x is provided, including the order of component upgrades (Streaming Node, MixCoord, Query Node, Data Node, Proxy) and notes on operational impacts like DDL unavailability.

10/2/2025

This post provides a detailed guide on debugging slow search requests in Milvus. It covers identifying slow searches using metrics (Service Quality → Slow Query, Search Latency, Query Node → Search Latency by Phase) and logs, and analyzes root causes including heavy workloads (large NQ, high QPS), inefficient filtering (poorly conducted expressions, lack of scalar indexes, JSON filters, strict consistency), improper vector index choices (HNSW, IVF, DiskANN, MINHASH_LSH for binary vectors), and runtime/environment conditions (background jobs, frequent upserts, version-specific inefficiencies). It offers specific fixes for each cause and concludes with best practices for prevention.

8/15/2025

Introduces VDBBench, an open-source tool for benchmarking vector databases with custom datasets and real-world production workloads. Details the installation, configuration, and data preparation process for VDBBench, including automated format conversion from CSV/NPY to Parquet and ground truth computation using FAISS. Provides a complete Python script for the conversion process and outlines the custom dataset configuration interface within VDBBench.

7/4/2025

Introduces vdbench 1.0, a benchmarking tool designed for vector databases. The post details how to use vdbench to simulate production workloads, measure performance metrics (e.g., QPS, latency), and analyze results. It emphasizes the importance of using realistic data and query patterns for accurate performance evaluation.

5/23/2025

This post benchmarks the latency of over 20 embedding APIs when integrated with Milvus's TextEmbedding Function. It details the impact of network geography, model size, token length, and batch size on API performance, and compares cloud API latency with local inference options. The post also validates the minimal overhead introduced by Milvus's TextEmbedding Function and provides optimization tips for RAG embedding performance.

5/14/2025

Introduced VDBBench, an open-source benchmarking tool for vector databases. VDBBench simulates production environments by using modern datasets (e.g., Wikipedia, BioASQ, C4, MSMarco V2) generated by state-of-the-art embedding models, focusing on production-relevant metrics such as P95/P99 latency, sustainable throughput, and recall balanced with performance. It employs a two-phase testing methodology (Serial Test and Concurrency Test) and evaluates performance across static collection, filtering (with varying selectivity), and streaming (search-while-inserting) scenarios. The tool aims to provide accurate and production-relevant insights for capacity planning and system design.

4/11/2025

Introduces the Milvus Sizing Tool, a calculator for estimating deployment resources. Details the impact of vector count, dimensions, index type, scalar field data size, segment size, and deployment mode on resource requirements. Provides in-depth explanations of various index types (HNSW, FLAT, IVF_FLAT, IVF_SQ8, ScaNN, DiskANN, GPU indexes, HNSW variants) and their performance/resource trade-offs. Discusses segment size recommendations based on query node memory. Offers guidance on Pulsar vs. Kafka selection. Outlines Zilliz Cloud enterprise optimizations (OOM prevention, compaction, tiered storage). Provides advanced configuration tips for developers regarding multiple indexes, memory, testing, and scaling.

2/18/2025

This post analyzes user discussions from 2024 to identify common challenges and provide guidance on Milvus usage. It categorizes these insights into Performance Optimization, Deployment Strategies, and Data Management. For deployment, it details the differences and use cases for Milvus Lite, Standalone, and Distributed, introduces the Milvus Resource Sizing Tool, and discusses index and distance metric selection (in-memory, on-disk, GPU indexes, and distance metrics). It also addresses troubleshooting distributed deployments and Windows deployment. For performance, it explains how to measure performance using metrics and the WebUI, how to observe internal state, and provides insights into complex internal aspects like segment sealing and memory management. Finally, it touches upon the selection of embedding models.

6/5/2024

This post introduces the integration of embedding and reranking models within the PyMilvus SDK, enabling users to generate dense and sparse vectors and apply reranking for improved search relevance. It demonstrates practical usage with Milvus Lite, covering dense embeddings using `DefaultEmbeddingFunction` (e.g., `all-MiniLM-L6-v2`), sparse embeddings with `BM25EmbeddingFunction` (including fitting and saving model parameters), and reranking with `JinaRerankFunction`. The post details the installation of the `pymilvus[model]` package and provides code examples for each integration.

5/13/2024

This post details optimizations for Milvus's Datanode buffer write to reduce memory movement overhead, improving data loading and indexing performance. It also discusses performance metrics, benchmarking frameworks (VectorDBBench, vector-db-benchmark), and open datasets for evaluating vector databases. The post highlights the use of tools like `perf` and Intel® vTune™ Profiler for in-depth performance analysis.

1/21/2024

This post analyzes community discussions from 2023, revealing key technical topics. It highlights 'Memory' as a significant discussion point, detailing community interest in optimization techniques like MMap and DiskANN, and the inherent trade-offs between performance, accuracy, and memory usage. It also elaborates on efficient data insertion strategies, including batch insertion, strategic `flush()` usage, and the use of `upsert` for primary key deduplication, contrasting it with `insert` for performance implications.

1/12/2024

Milvus 2.3.4 introduces access logs for improved monitoring of gRPC interactions, enabling performance analysis and security auditing. It adds support for importing Parquet files, enhancing data processing efficiency and making Parquet the default output for the BulkWriter tool. Search performance is significantly improved with the implementation of a binlog index on growing segments, leading to up to tenfold faster searches. The system now supports up to 10,000 collections/partitions, addressing needs for nuanced data organization in use cases like knowledge base management and multi-tenancy. Other enhancements include reduced memory usage during data retrieval and variable-length data handling, refined error messaging, accelerated loading speed, and improved query shard balance.

5/12/2023

Milvus 2.2.8 addresses peak memory consumption during collection loading by optimizing memory usage. It enables expanded querying scenarios by allowing QueryNode to support plugins. The compaction algorithm has been enhanced for faster convergence and improved query performance. The default number of collection shards has been reduced from two to one to improve resource saving and querying performance. The query grouping algorithm has been further enhanced, leading to a 20% increase in throughput.

4/22/2023

This post details the evolution of Milvus from v2.2 to v2.2.6, highlighting new features, bug fixes, and performance optimizations. Key additions include bulk inserts, query result pagination, RBAC, quotas and limits, collection-level TTL, DiskANN for disk-based ANNS, and a backup tool in v2.2. Subsequent releases (v2.2.1-v2.2.6) focused on bug fixes, performance improvements (e.g., search performance, scheduler optimization), security enhancements (TLS, Pulsar tenant/auth), availability features (rolling upgrade, coordinator HA), resource management (resource grouping), and usability (collection renaming). Specific critical issues resolved in v2.2.6 include DataCoord GC failure and binlog data recycling problems.

4/25/2022

This post details the Quality Assurance framework adopted in developing Milvus, covering its system architecture, test modules (unit, function, deployment, reliability, performance, stability, configuration, compatibility), and issue management workflow. It emphasizes the importance of QA throughout the R&D lifecycle and introduces tools and methods for improving QA efficiency, including chaos engineering with Chaos Mesh for reliability testing and pytest for function testing.

7/15/2021

This post introduces the Milvus 2.0 bootcamp, which provides updated guides and code examples for testing and deploying vector search solutions. It includes benchmark tests for 1M and 100M datasets, and solutions for various use cases like image similarity search. The image similarity search example details steps for setting up Milvus and Redis servers, creating a collection with an IVF_SQ8 index, loading data, generating vector embeddings using Resnet-18, inserting vectors into Milvus, and performing similarity searches. It also covers deployment options in different environments.

11/10/2020

This post details performance comparisons of Milvus on AVX-512 versus AVX2 instruction sets. It outlines system configuration, Milvus parameters (cache size, nlist, nprobe), and the SIFT10M dataset used for testing. Performance results are presented for IVF_FLAT, IVF_SQ8, and HNSW index types, showing consistent speed improvements with AVX-512. The analysis concludes that while AVX-512 offers theoretical benefits, real-world performance gains are influenced by other system tasks. It also highlights IVF_SQ8's efficiency over IVF_FLAT due to memory bandwidth constraints.