Milvus Snapshots

1 post · Aug 2023 – Jul 2026

What this trail covers

Milvus Snapshots provide named, read-only versions of collections by recording references to existing data, index, and metadata files without copying the dataset. This enables lightweight recovery points, fast restores by reusing index files, cross-cluster movement via direct metadata URI access, and stable views for batch jobs via External Collection and Spark connector integration. Snapshots are cost-effective to create but incur storage costs based on retention and file divergence from the live collection. They do not include data in growing segments unless flushed prior to creation. Restore operations are asynchronous and non-destructive, creating new collections. Export is a fallback for when direct storage access is not possible.

All Milvus trails

Jul 2026Milvus Snapshots: Point-in-Time Collection Views Without Copying DataThis post details the introduction and capabilities of Milvus Snapshots. It explains the problem of needing stable data versions for various AI team tasks (recovery points, rollback, testing, batch jobs) and how copying data is expensive. Milvus Snapshots solve this by recording file references instead of copying data. The post describes the four snapshot capabilities: lightweight snapshots (cost tracks file counts), fast restore (reuses index files), cross-cluster movement (access problem, not transfer problem), and stable views (External Collection/Spark connector). It also covers snapshot creation mechanics (Avro manifest, JSON metadata), the exclusion of growing segments unless flushed, storage costs associated with retention, and the `RestoreSnapshot` and `RestoreExternalSnapshot` mechanisms.
Nov 2023Revealing Milvus 2.3.2 & 2.3.3: Support for Array Data Types, Complex Delete, TiKV Integration, and MoreThis post details the introduction of Array data types for filtering, complex delete expressions for data management, and TiKV integration as a scalable metadata storage option. It also notes interface-level support for FP16 vectors, improvements to rolling upgrades, and various performance optimizations including reduced data copying for loading, batch varchar reading for inserts, and addressing high CPU usage during insertions. Incompatible changes include the permanent deletion of TimeTravel code and deprecation of MySQL as a metadata store.
Aug 2023Unveiling Milvus 2.3: A Milestone Release Offering Support for GPU, Arm64, CDC, and Many Other Highly Anticipated FeaturesMilvus 2.3 introduces GPU indexing using the RAFT algorithm via Knowhere, providing 3-10x faster QPS. It adds Arm64 support, upsert operations (implemented as delete+insert), range search with radius and range_filter parameters, ScaNN index integration for faster queries, and a Growing Index for real-time streaming data. An iterator interface in Pymilvus allows batch retrieval of over 16,384 entities. MMap technology is supported to increase single-machine capacity by mapping local disks to memory. Change Data Capture (CDC) is introduced for data synchronization and availability. Performance improvements include optimized HNSW filtering, full multi-core CPU utilization, a refactored stateless QueryNode, and a new NATS/Bookkeeper log broker. The load balancer is optimized for faster fault detection and adjustment. Birdwatcher is updated with a RESTful API, PProf support, and configuration management. Attu receives a redesigned interface.
Advertisement