TechPrufer
Get started
Milvus
Milvus logoMilvus · Feature trail

Data Migration Tooling

3 posts · Mar 2021 – Jul 2026

What this trail covers

Milvus 2.6 introduces a hot-cold tiered storage architecture for cost-effective scaling to hundreds of billions of vectors, intelligently managing data placement based on access patterns. It also features a new Streaming Node for real-time vector processing with direct integration to streaming platforms and Woodpecker, making new data searchable immediately. RaBitQ 1-bit quantization significantly reduces memory footprint (up to 97%) while improving query throughput (up to 4x) without compromising accuracy. This post details the integration of RaBitQ, a novel 1-bit vector compression technique, into Milvus, enabling it to serve 3x more queries with lower memory costs while maintaining comparable accuracy. It discusses implementation tradeoffs, hardware acceleration using AVX-512 instructions, and query optimizations.

All Milvus trails
  1. How to Recover Milvus Data After etcd Corruption Without a BackupThis post details a last-resort data salvage procedure for etcd corruption, leveraging insert-log binlogs from object storage to reconstruct collections when metadata is lost and no backup exists. The procedure involves starting a clean Milvus instance, recreating the schema, and importing candidate segments from object storage, with validation steps to avoid duplicate data. It highlights the critical role of etcd for metadata and object storage for data files, and the challenges of rebuilding a collection solely from object storage. The process includes steps for copying binlogs, scanning for candidate segments, and using the Java SDK's bulkInsert API in backup mode for import, emphasizing validation to prevent duplicate segments.
  2. Troubleshooting a Search Slowdown After Upgrading Milvus: Lessons from the WPS TeamThis post details a user's experience upgrading Milvus from v2.2.16 to v2.5.16 to resolve a stability issue in the Proxy component. It outlines the use of `milvus-backup` for hot backup and cross-version restore, emphasizing version compatibility rules and the backup/restore process. The post also describes deploying a new Milvus 2.5.16 cluster using Helm, with key configuration changes including disabling mmap for latency-sensitive workloads and maintaining the same QueryNode count and resource limits. The user observed a 3-5x increase in search latency after the upgrade and details their investigation and resolution, which involved tuning configurations and potentially identifying performance regressions between versions.
  3. How to Safely Upgrade from Milvus 2.5.x to Milvus 2.6.xThis post details the architectural evolution from Milvus 2.5 to 2.6, highlighting the introduction of a dedicated StreamingNode to separate streaming and batch workloads. It compares component-level changes, emphasizing the consolidation of coordinator services into MixCoord and the clearer role definitions for StreamingNode, QueryNode, and DataNode. The post also enumerates key feature highlights in Milvus 2.6, including cost-saving improvements (RaBitQ, BM25), search enhancements (text processing, rankers), and architectural upgrades (Tiered Storage, Woodpecker). Crucially, it provides a step-by-step guide for upgrading from Milvus 2.5.x to 2.6.x, outlining the order of component upgrades and noting potential operational impacts like DDL unavailability. It also briefly touches upon the Milvus Operator for Kubernetes-based upgrades.
  4. Stop Paying for Cold Data: 80% Cost Reduction with On-Demand Hot–Cold Data Loading in Milvus Tiered StorageThis post introduces Milvus 2.6's Tiered Storage architecture, a significant evolution from the previous full-load model. It details the technical mechanisms of lazy loading (minimal metadata at startup), partial loading (on-demand fetching of data chunks and index files, with tenant-aware index loading and column-level scalar field loading planned), and LRU-based cache eviction. The post quantifies the cost savings (up to 80%) and performance benefits (faster upgrades, recovery, and iteration) compared to the full-load model, which required provisioning resources for the entire dataset regardless of access frequency.
  5. Introducing Milvus 2.6: Affordable Vector Search at Billion ScaleMilvus 2.6 introduces RaBitQ 1-bit quantization for significant memory reduction (72%) and performance improvement (4x faster queries) with optional SQ8 refinement. It also implements hot-cold tiered storage for up to 50% cost savings by moving infrequently accessed data to object storage. Full-text search is enhanced with a BM25 implementation offering 3-4x higher throughput than Elasticsearch. A JSON Path Index is introduced, reducing JSON filtering latency by 99% (from 140ms to 1.5ms P99). Text analysis pipeline is revamped with new tokenizers (Lindera, ICU) and enhanced Jieba. Phrase Match Intelligence and Time-Aware Decay Functions are added for more nuanced search. A "Data-In, Data-Out" experience is enabled via the Function interface, integrating third-party embedding models directly into the search pipeline. Architecturally, Milvus 2.6 replaces Kafka/Pulsar with Woodpecker, a cloud-native WAL system with a zero-disk design persisting logs to object storage, offering significantly higher throughput and reduced operational overhead. A new Streaming Node is introduced to improve search freshness economically.
  7. Milvus 2.6 Preview: 72% Memory Reduction Without Compromising Recall and 4x Faster Than ElasticsearchMilvus 2.6 introduces RaBitQ 1-bit quantization, achieving up to a 97% memory reduction with an adjustable Refine optimization capability (SQ4/SQ6/SQ8 options) for ~95% recall. Benchmarks show up to a 4x query throughput improvement. Full-text search performance is enhanced with 3-4x higher throughput than Elasticsearch. JSON Path Index reduces filter latency by 99% for nested JSON fields. New search features include a multi-language analyzer with Lindera and ICU tokenizers, Phrase Match with a `slop` parameter, and Decay Functions (Exponential, Gaussian, Linear) for time-aware ranking. A new Function interface integrates third-party embedding models (OpenAI, AWS Bedrock, Google Vertex AI, Hugging Face) for direct data-to-vector search. Architecturally, a hot-cold tiered storage system is introduced for scaling to hundreds of billions of vectors, and a new Streaming Node with Woodpecker enables real-time search.
  8. Bring Vector Compression to the Extreme: How Milvus Serves 3× More Queries with RaBitQThis post details the integration of RaBitQ, a novel 1-bit vector compression technique, into Milvus. It explains the challenges of extreme vector compression, the RaBitQ algorithm's approach to encoding angular information, and its implementation within Milvus's Knowhere search engine and FAISS. Key engineering aspects include handling per-vector auxiliary data with a tradeoff between memory and query speed, and leveraging AVX-512 instructions for hardware acceleration of binary operations. The post also discusses query optimizations like scalar quantization and introduces the `IVF_RABITQ` index type, providing usage examples and benchmarking results that show a 3x QPS increase with comparable accuracy.
  9. Full-Text Search in Milvus - What's Under the HoodThis post provides a deep dive into Milvus's implementation of Sparse-BM25 for full-text search. It explains the BM25 ranking algorithm, its transformation into an Inner Product calculation for sparse vectors, and the challenges of dynamic statistics. Milvus 2.5's solution involves storing raw TF as document vectors, computing BM25 scores on the fly during query time using a new 'BM25' metric for sparse vectors, and handling text processing and statistics internally. A PyMilvus demo illustrates defining a collection schema with a text field and a sparse vector field mapped via a BM25 Function, creating an AUTO_INDEX with BM25 metric, inserting raw text, and performing searches.
  10. Elasticsearch is Dead, Long Live Lexical SearchThis post details Milvus's native lexical search capabilities via Sparse-BM25, introduced in Milvus 2.5 and enhanced in 2.6. It contrasts Milvus's approach with Elasticsearch's limitations, highlighting Milvus's advantages in algorithm flexibility (e.g., TW-BERT), cost efficiency (e.g., 5x performance improvement, >50% memory reduction), and long query optimization. The post outlines key components of Milvus's Sparse-BM25 implementation, including advanced tokenization via Tantivy, distributed term management, sparse vector generation, and inverted index support with WAND. It also reiterates Milvus's strengths for RAG applications, such as metadata support, multi-tenancy, disk vector index, cloud-native scalability, and ecosystem integration.
  11. Getting Started with Hybrid Semantic / Full-Text Search with Milvus 2.5This post details the implementation of full-text search in Milvus 2.5 by introducing the `Function` object and `FunctionType.BM25` for connecting text fields to sparse vector fields. It demonstrates schema creation with `enable_analyzer=True` for text fields and `SPARSE_INVERTED_INDEX` with `BM25` metric type for sparse vector indexing. The post also shows how to perform hybrid search by combining dense vector search with sparse vector (full-text) search using `RRFRanker`.
  12. Introducing Milvus 2.5: Full-Text Search, More Powerful Metadata Filtering, and Usability Improvements!Milvus 2.5 adds native full-text search via Sparse-BM25, integrating a tokenizer built on tantivy. It supports direct ingestion and retrieval of raw documents, internalizing BM25 scoring based on sparse vectors. This enables optimizations like graph indexing and quantization for sparse vectors. New API examples demonstrate inserting text and performing searches. Text Match filters are introduced, operating on tokenized text with OR logic by default. Scalar filtering performance is enhanced with BitMap Index for tag filtering and improvements to text matching filters, offering significant QPS increases.
  13. Semantic Search vs. Full-Text Search: Which Do I Choose in Milvus 2.5?This post details the integration of full-text search (FTS) with Milvus's existing vector search capabilities, enabling hybrid search. It introduces the use of BM25 for text matching and demonstrates its synergy with dense embedding-based semantic search. The post analyzes performance using the Pass@5 metric and provides case studies comparing semantic search, full-text search, and hybrid search on a code dataset. It highlights how hybrid search, by combining keyword matching with semantic understanding, can yield superior results, particularly in scenarios like AI-assisted coding bots where both precise keyword identification and contextual understanding are crucial. The post also touches upon the nuances of BM25's term frequency weighting in code contexts.
  14. How to Migrate Your Data to Milvus Seamlessly: A Comprehensive GuideThis post introduces the Milvus Migration tool, a Go-based utility designed to facilitate data migration to Milvus 2.x from Milvus 1.x, FAISS, and Elasticsearch 7.0+. It details the architecture and workflow for migrating data from these sources, including parsing original data files, transforming them into Milvus 2.x format, and using bulkInsert. For Elasticsearch migrations, it highlights fetching data via the scroll API and migrating various field types beyond dense vectors. The post outlines the tool's feature set, including supported data sources, interaction modes (CLI, API, Go module), and file format support (local, S3, OSS, GCP). It also defines key API interfaces like /start, /dump, /load, and /get_job, and provides detailed examples for migrating from Elasticsearch and Milvus 1.x, including configuration file examples and execution steps.
  15. Introducing Milvus Data Migration ToolThis post details the Milvus Data Migration Tool (MilvusDM), an open-source tool for importing and exporting data files with Milvus. It covers four primary migration types: Faiss to Milvus, HDF5 to Milvus, Milvus to Milvus, and Milvus to HDF5. For each type, it provides an overview, step-by-step instructions with YAML configuration examples, and sample code snippets. It also outlines the MilvusDM file structure and mentions planned features like binary data import from Faiss, blocklist/allowlist for Milvus-to-Milvus migration, merging data from multiple collections, and backup/recovery. The post also notes that MilvusDM has been deprecated in favor of the Vector Transport Service (VTS).