How Dust Scaled to 5,000+ Data Sources with Qdrant - Qdrant

This post details Dust's successful scaling to over 5,000 data sources by migrating to Qdrant. Key technical contributions include: consolidating thousands of individual collections into a few shared, multi-tenant ones using Qdrant's sharding and payload filtering; implementing scalar quantization to reduce vector storage by 4x and enable in-memory data, drastically improving search latency; building a custom `DustQdrantClient` to abstract Qdrant operations; and deploying Qdrant in isolated VPCs across US and EU regions for high availability and compliance. The post also highlights the architectural flexibility gained by grouping data sources by embedder, enabling smoother migrations and model experimentation.