Data Insertion and Persistence

3 posts · Nov 2019 – Jun 2025

What this trail covers

Milvus 2.6 introduces Woodpecker, a cloud-native Write-Ahead Logging (WAL) system, replacing external dependencies like Kafka and Pulsar. Woodpecker features a Zero-Disk architecture, persisting logs in object storage and managing metadata with etcd. It offers high-throughput data ingestion, low-latency tail reads, pluggable storage backends, and flexible deployment modes. Benchmarks show significant performance improvements over Kafka and Pulsar, especially in cloud storage and local file systems. Milvus 2.3 introduces NATS as an alternative message queue for standalone deployments, offering improved performance for larger data packets compared to RocksMQ, and similar performance for smaller datasets.

All Milvus trails

Jun 2025Introducing Milvus 2.6: Affordable Vector Search at Billion ScaleMilvus 2.6 replaces Kafka and Pulsar with Woodpecker, a purpose-built, cloud-native WAL system. Woodpecker utilizes a zero-disk design, storing logs in object storage and managing metadata with etcd, eliminating external dependencies and operational overhead. Benchmarks demonstrate Woodpecker's superior throughput and lower latency compared to Kafka and Pulsar across various storage backends (MinIO, local file system, S3), enabling more economical search freshness.
May 2025We Replaced Kafka/Pulsar with a Woodpecker for Milvus—Here’s What HappenedThis post details the replacement of Kafka and Pulsar with Woodpecker, a custom-built, cloud-native WAL system, in Milvus 2.6. It highlights the operational complexity, architectural constraints, and resource inefficiencies of using general-purpose message queues for Milvus's specific needs. Woodpecker's Zero-Disk architecture, which leverages object storage for logs and etcd for metadata, is presented as a solution for simplified operations, improved performance, and lower costs. Performance benchmarks demonstrate Woodpecker's superior throughput and lower latency compared to Kafka and Pulsar across local file system, MinIO, and S3 backends. The post also outlines Woodpecker's architecture components (Client, LogStore, Storage Backend, ETCD) and its two deployment modes: MemoryBuffer Mode for batch-heavy workloads and QuorumBuffer Mode for low-latency, high-durability deployments.
Mar 2025A Day in the Life of a Milvus DatumThis post details the internal data flow for insert and search operations in Milvus. It explains how data moves from the MilvusClient through proxy nodes and message queues (Pulsar/Kafka) to data nodes, where it's stored in growing segments and chunks. It also describes the process of sealing, merging, and compacting segments, and how index nodes build indexes for sealed segments. Finally, it outlines how query nodes subscribe to message queues, load segments into memory, and perform searches, with results collated in a MapReduce-like fashion. The core takeaway is the separation of concerns across node types and the separation of storage and compute for independent scaling.
Nov 2023Optimizing Data Communication: Milvus Embraces NATS MessagingMilvus 2.3 introduces NATS as an alternative message queue (MQ) for its standalone deployment mode, replacing RocksMQ. The post details how to enable and configure NATS by setting `mq.type=natsmq` and provides configuration options for NATS server port, JetStream storage directory, max payload size, and retention policies. It also outlines the migration process from RocksMQ to NATS, which involves stopping writes, flushing data, updating configuration, and cleaning up old data. Performance comparisons show NATS outperforming RocksMQ for larger data packets (> 64kb) in Pub/Sub scenarios and offering lower vector search and query latency in Milvus integration tests with a 100 million vectors dataset, while performance is similar for smaller datasets (< 100M).
Apr 2022Data Insertion and Data Persistence in a Vector DatabaseThis post details the internal workings of Milvus's data insertion and persistence mechanisms. It explains the roles of the proxy, data coord, data node, and root coord in handling data requests, allocating storage, and ensuring data durability. It elaborates on the data organization hierarchy (collection, partition, shard, segment) and the lifecycle of segments (growing, sealed, flushed). It also covers channel allocation strategies and the binlog file structure.
Advertisement
Mar 2022How Is Data Processed in a Vector Database?This post details the data processing flow in Milvus, explaining the roles of MsgStream, flowgraphs, and the interaction between components during data write and read operations. It elaborates on how data is written via proxies to log brokers, consumed by data nodes, and stored in object storage, and how query requests are handled by query nodes and aggregated by proxies. It also covers DDL operations and the initial steps of index building.
Jan 2022Milvus 2.0 - A Glimpse at New FeaturesThis post details the introduction of entity deletion in Milvus 2.0, explaining its asynchronous and logical nature and its dependency on consistency levels. It demonstrates entity deletion with Python code examples, showing how to insert, search, delete, and re-query entities. The post also elaborates on the four consistency levels (STRONG, EVENTUALLY, BOUNDED, SESSION) and their impact on performance, using latency measurements to compare STRONG and BOUNDED levels. The 'handoff' feature is introduced, which automatically loads indexed segments to replace streaming data, improving search performance. Code examples are provided for index creation, collection loading, and large-scale data insertion to illustrate the handoff mechanism.
Jan 2022Announcing General Availability of Milvus 2.0Milvus 2.0 GA introduces entity deletion by primary key, automatic load balancing for query and data nodes, handoff tasks for improving search performance, and data compaction for merging segments and cleaning deleted data. It also adds support for embedded etcd and local data storage in standalone mode, and provides multi-language SDKs and a Milvus K8s Operator.
Jan 2022The Story of SmartNews - from a Milvus User to an Active ContributorThis post details SmartNews's contribution of the 'hot reload' functionality to Milvus, achieved through the implementation of collection aliases. This feature allows for dynamic data updates in Milvus without requiring service restarts or manual client-side code changes. The contribution addresses the challenge of updating large datasets (e.g., 100 million vectors) by enabling users to insert new data into a separate collection and then swap the collection alias to point to the new data, significantly improving user experience and operational efficiency for large-scale applications.
Apr 2020How Milvus Implements Dynamic Data Update And QueryThis post details the initial design for dynamic data update and query in Milvus, focusing on the memory buffer (insert buffer) architecture using MemTable and MemTableFile. It describes how data is inserted into memory, managed in mutable and immutable states, and periodically serialized to disk. The design goals of high import efficiency, near real-time visibility, and avoiding file fragmentation are addressed through this in-memory buffering and background serialization process. The post outlines the steps from data insertion into memory to its serialization to disk, making it searchable.
Nov 2019Managing Data in Massive-Scale Vector Search EngineThis post details Milvus's data management strategies for massive-scale vector search. It explains the concepts of Tables and Indexes. For vector insertion, it describes the use of a mutable memory buffer for efficient writes, followed by regular flushing to disk into Raw Data Files. These Raw Data Files are then merged into larger files (e.g., 1GB) to reduce fragmentation and improve search access. Index Files are built from these merged files to enhance search efficiency over brute-force search on raw data. Metadata management uses SQLite (standalone) or MySQL (distributed) to track file statuses and information. The Query Scheduler is introduced as a mechanism to optimize hardware utilization for search, including strategies for 'cold' vs. 'warm' queries, pre-loading tables, and LRU-based memory replacement. The Result Reducer's role in aggregating top-k results from multiple file searches is also mentioned.