Milvus Supports Imports of Apache Parquet Files for Enhanced Data Processing Efficiency
Mar 8, 2024Cai Zhang, Fendy Feng

Summary
Introduced support for Apache Parquet file imports in Milvus version 2.3.4. This feature enhances data processing efficiency by leveraging Parquet's columnar storage, compression, and encoding for optimized storage and query performance. The post details how to import Parquet files directly from object storage (e.g., MinIO) and how to use the BulkWriter tool, which now defaults to Parquet output, to generate Parquet files for import. Future plans include optimizing large file imports by splitting scalar and vector columns.
More from the AI Data Storage Engine trail
- Milvus Snapshots: Point-in-Time Collection Views Without Copying Data
- Announcing Milvus 3.0: Lake-Native Vector Search and a More Powerful Retrieval Engine
- How to Recover Milvus Data After etcd Corruption Without a Backup
- Open Tag: An Open-Source Claude Tag for Claude Code and Codex
- How Force Merge Compaction Nearly Doubled Milvus Search QPS