TechPrufer
Get started
Milvus
Milvus logoMilvus · Feature trail

MinHash LSH for Deduplication

1 post · May 2025

What this trail covers

Milvus 2.6 introduces native MinHash LSH indexing for efficient approximate deduplication of large datasets. This feature allows users to generate MinHash signatures, build MINHASH_LSH indexes with Jaccard similarity, and perform similarity searches directly within Milvus. It supports in-memory and mmap-based implementations, seamless SDK integration, and leverages Milvus's distributed architecture for scalability. This enables end-to-end deduplication workflows within Milvus, reducing operation overhead and costs, particularly for LLM training data. The integration was driven by a real-world need from a leading LLM company to deduplicate billions of documents, resulting in doubled processing speed and 3-5x cost savings compared to previous MapReduce-based approaches.

All Milvus trails
  1. MinHash LSH in Milvus: The Secret Weapon for Fighting Duplicates in LLM Training DataThis post details the integration of MinHash LSH into Milvus 2.6, focusing on its application for deduplicating large datasets, specifically for LLM training data. It explains the underlying principles of MinHash (shingling, hashing) and LSH (banding technique) for approximate near-duplicate detection. The post highlights the native implementation within Milvus, including the `MINHASH_LSH` index type, `MHJACCARD` metric, and support for in-memory and mmap-based implementations. It also provides a hands-on example using Python and MilvusClient to create a schema, define the index, and insert MinHash signatures. The practical benefits are demonstrated through a case study of deduplicating 10 billion documents, achieving significant speed and cost improvements over traditional methods.