TechPrufer
Get started
Milvus
Milvus logoMilvus · Feature trail

External Collection for Lake-Resident Data

1 post · Jun 2021 – Aug 2026

What this trail covers

Milvus 3.0 introduces External Collections, enabling Milvus to build and serve indexes directly on data residing in external data lakes (Parquet, Iceberg, Lance, Vortex) without copying it into Milvus-managed storage. This enables a "Vector Lakebase" architecture, unifying AI data serving with open lake storage. The feature maps external fields to Milvus schemas, defines indexes, and uses a "Refresh" operation to prepare the data for Milvus search and query APIs. Loon (Storage v3) is introduced as a new storage engine designed for dynamic data, offering efficient handling of data that is constantly changing, such as streaming data or frequently updated datasets. Loon aims to provide a unified storage solution for both batch and real-time data, optimizing for ingestion, querying, and indexing performance in dynamic environments.

All Milvus trails
  1. Milvus External Collection: Index and Retrieve Lake-Resident Data Without Moving ItThis post introduces Milvus External Collection, a new capability that allows Milvus to index and serve data directly from external data lake formats (Parquet, Iceberg, Lance, Vortex) without requiring data to be copied into Milvus-managed storage. It details the architecture, contrasting it with traditional approaches of copying data or performing lake scans. The post explains the components of an External Collection (external_source, external_spec, external_field mappings), its lifecycle (define schema, define indexes, refresh, load, query), and its benefits in reducing data duplication and synchronization costs. It also covers supported external formats and secure access to external storage.
  2. Announcing Milvus 3.0: Lake-Native Vector Search and a More Powerful Retrieval EngineMilvus 3.0 introduces External Collections, allowing Milvus to index and serve data directly from external data lakes (Parquet, Lance, Iceberg, Vortex) without data duplication. This is achieved by mapping external fields to Milvus schemas and using a "Refresh" operation. Loon (Storage v3) is introduced as a manifest-based columnar storage engine for S3-compatible object storage, optimizing point reads by reducing read amplification, particularly with the Vortex format. Snapshots provide inexpensive, point-in-time, read-only views of collections. A Spark DataSource V2 connector is added for batch workflow integration. Online schema evolution (adding/dropping fields) is now supported without data rewrites. The retrieval engine is enhanced with ORDER BY, aggregation, facets, and StructArray for more complex result processing. New data model features include nullable vectors, TEXT LOB, TTL, MinHash, Woodpecker (a cloud-native WAL system), and ForceMerge compaction.
  3. https://zilliz.com/blog/why-we-built-loon-a-storage-engine-for-ai-data-that-never-stops-changingThis post introduces Loon, a new storage engine for Milvus designed to handle continuously changing AI data. Loon is built to address the challenges of dynamic datasets, offering efficient ingestion, querying, and indexing for data that is frequently updated or streamed. It aims to provide a unified storage solution for both batch and real-time data, complementing the existing External Collection feature by providing a robust backend for dynamic data lakes. The design emphasizes performance and adaptability in scenarios where data is not static.
  4. Stop Paying for Cold Data: 80% Cost Reduction with On-Demand Hot–Cold Data Loading in Milvus Tiered StorageThis post details the introduction of Tiered Storage in Milvus 2.6. It explains the limitations of the previous full-load model, which required all data to be resident locally, leading to high costs and operational inefficiencies. The new Tiered Storage architecture leverages lazy loading of metadata, partial loading of field data and indexes (including tenant-aware and column-level partial loading), and LRU-based cache eviction to dynamically manage data between local compute and remote object storage. This enables on-demand loading, significantly reducing storage and memory costs and improving performance for large datasets with skewed access patterns.
  5. DeepSeek V3-0324: The "Minor Update" That's Crushing Top AI ModelsThis post details the integration of DeepSeek v3-0324, an open-source LLM, with Milvus for building Retrieval-Augmented Generation (RAG) systems. It highlights DeepSeek v3-0324's improvements in model size, context window (128K tokens), and reasoning capabilities, attributing these to architectural innovations like Multi-head Latent Attention (MLA) and Mixture of Experts (MoE) layers, and the introduction of multi-token prediction (MTP). The post provides a practical tutorial on setting up a RAG pipeline using Milvus Lite, DeepSeek v3-0324 via OpenRouter, and Milvus's built-in embedding function. It covers environment setup, data preparation from Milvus documentation, Milvus collection creation, embedding generation, data insertion, and a demonstration of retrieving relevant information and generating a response using the LLM. The tutorial emphasizes the ease of building a RAG system with these components.
  7. Building a Milvus Cluster Based on JuiceFSThis post details the integration of Milvus 1.0.0 with JuiceFS, a shared file system designed for cloud-native environments. It explains how JuiceFS, built on Redis for metadata and object storage for data, can serve as the shared storage layer for a Milvus cluster. The post outlines the steps to set up MySQL for metadata, create a JuiceFS file system, and configure Milvus nodes (read/write roles) to use the JuiceFS mount point. Performance benchmarks comparing JuiceFS with EFS are presented, highlighting JuiceFS's superior IOPS and throughput, and demonstrating near-instantaneous data loading from disk to memory.