Building a Vector Database for Scalable Similarity Search

What this post added

This post details the thinking process and design principles behind building Milvus, an open-source, cloud-native vector database for scalable similarity search. It explains the Milvus architecture in detail, covering the need for a basic software stack for unstructured data, the conversion of unstructured data to vectors, and the role of similarity metrics. It differentiates Milvus as a vector database from a vector search engine by highlighting features like sharding, partitioning, replication, disaster recovery, load balancing, and query parsing. The cloud-native approach is discussed, including the separation of compute and storage ('shared something' architecture) and the use of Kubernetes for microservices. The design principles of Milvus 2.0 are elaborated: 'Log as data' where logs record all changes and worker nodes consume them; 'Duality of table and log' where logs can be converted to tables and aggregated into segments; and 'Log persistency' using a publish-subscribe system (log broker) for decoupling and statelessness. The deployment modes (standalone and cluster) are described, along with a high-level overview of the Milvus architecture's four layers: access, coordination, distributed execution, and storage.