Milvus

Milvus Supports Imports of Apache Parquet Files for Enhanced Data Processing Efficiency

Mar 8, 2024Cai Zhang, Fendy Feng
Milvus Supports Imports of Apache Parquet Files for Enhanced Data Processing Efficiency
SummaryAI, from the source post

Introduced support for Apache Parquet file imports in Milvus version 2.3.4. This feature enhances data processing efficiency by leveraging Parquet's columnar storage, compression, and encoding for optimized storage and query performance. The post details how to import Parquet files directly from object storage (e.g., MinIO) and how to use the BulkWriter tool, which now defaults to Parquet output, to generate Parquet files for import. Future plans include optimizing large file imports by splitting scalar and vector columns.

More from the AI Data Storage Engine trail