TechPrufer
Get started
Milvus
Milvus logoMilvus · Feature trail

Parquet File Import

1 post · Mar 2024

What this trail covers

Milvus 2.3.4 introduces support for importing data directly from Apache Parquet files. This feature streamlines data ingestion by leveraging Parquet's columnar storage, efficient compression, and broad language compatibility. Users can upload Parquet files to object storage and import them into Milvus using bulk insert operations. The BulkWriter tool also defaults to generating Parquet files. Future plans include segregating scalar and vector columns for larger imports to alleviate I/O pressure.

All Milvus trails
  1. Milvus Supports Imports of Apache Parquet Files for Enhanced Data Processing EfficiencyThis post introduces the capability to import Apache Parquet files into Milvus. It details the benefits of Parquet, including optimized storage efficiency, streamlined data management, and improved query performance due to columnar storage and compression. The post provides code examples for uploading Parquet files to object storage (MinIO) and performing bulk inserts into Milvus. It also highlights the use of BulkWriter with Parquet as the default output format. Future work will focus on handling larger files by splitting scalar and vector data.