Evolution of Milvus Cloud-scalable Vector Database

This post details the design thinking behind Milvus 2.0's cloud-scalable cluster architecture. It contrasts with traditional shared-everything clusters, highlighting the challenges of global locking and buffer cross-invalidation in cloud environments. Milvus 2.0 adopts an eventual consistency model and asynchronous processes, utilizing cloud messaging queues (Pulsar/Kafka) for log storage and data nodes to bridge local storage with cloud shared storage (S3). The architecture aims for high scalability and availability by moving away from centralized components and embracing a more distributed, event-driven approach. Data is managed in collections with physical segments, and partitioning is discussed as a strategy for performance.