Milvus External Collection: Index and Retrieve Lake-Resident Data Without Moving It

This post introduces Milvus External Collection, a new capability that allows Milvus to index and serve data directly from external data lake formats (Parquet, Iceberg, Lance, Vortex) without requiring data to be copied into Milvus-managed storage. It details the architecture, contrasting it with traditional approaches of copying data or performing lake scans. The post explains the components of an External Collection (external_source, external_spec, external_field mappings), its lifecycle (define schema, define indexes, refresh, load, query), and its benefits in reducing data duplication and synchronization costs. It also covers supported external formats and secure access to external storage.