TechPrufer
Sign inGet started

Blogs›Milvus›Data Migration Tooling

Data Migration Tooling

Data Migration Tooling

2
posts
2021–2023

Milvus Migration tool is introduced to facilitate seamless data transition from Milvus 1.x, FAISS, and Elasticsearch 7.0+ to Milvus 2.x. It supports various data sources, interaction modes (CLI, RESTful API, Go module), and file formats (local, S3, OSS, GCP). The tool handles data retrieval, transformation, and insertion via Milvus SDK's bulkInsert. Specific migration strategies are detailed for Elasticsearch (using scroll API) and Milvus 1.x (parsing meta.json and tables folder). The tool also supports HDF5 files and Milvus to HDF5 export. The MilvusDM tool has been deprecated in favor of the more advanced Vector Transport Service (VTS).

2023

How to Migrate Your Data to Milvus Seamlessly: A Comprehensive Guide

12/1/2023

Introduces the Milvus Migration tool, a Go-based utility for migrating data from Milvus 1.x, FAISS, and Elasticsearch 7.0+ to Milvus 2.x. Details the architecture and data flow for migrating from Elasticsearch (using scroll API) and Milvus 1.x (parsing meta.json and tables folder). Outlines the tool's feature set including supported data sources, interaction modes (CLI, RESTful API, Go module), and file format support. Provides specific examples and configuration for Elasticsearch to Milvus 2.x migration, including API definitions for /start, /dump, /load, and /get_job. Also details the process for Milvus 1.x to Milvus 2.x migration, including data export commands and configuration.

2021

Introducing Milvus Data Migration Tool

3/15/2021

This post introduces the Milvus Data Migration (MilvusDM) tool, detailing its capabilities for importing data from Faiss and HDF5 files into Milvus, migrating data between Milvus instances, and exporting Milvus data to HDF5 files. It provides configuration parameters and sample code for each migration type, along with an overview of the tool's file structure and planned future features. The post also notes that MilvusDM has been deprecated in favor of the Vector Transport Service (VTS).