Full-Text Search in Milvus - What's Under the Hood
Apr 15, 2025Buqian Zheng

Summary
Introduces and details the implementation of Sparse-BM25 full-text search in Milvus 2.5. Explains the BM25 algorithm and its transformation into an Inner Product calculation for sparse vectors. Details Milvus's novel approach using dynamic statistics (avgdl, term distribution) and search-time encoding to handle the complexities of BM25 computation internally, removing client-side burden. Provides a PyMilvus example for defining a collection with a text field and a BM25 function, creating an AUTO_INDEX with BM25 metric, inserting raw text, and performing searches.
More from the AI Data Storage Engine trail
- Milvus Snapshots: Point-in-Time Collection Views Without Copying Data
- Announcing Milvus 3.0: Lake-Native Vector Search and a More Powerful Retrieval Engine
- How to Recover Milvus Data After etcd Corruption Without a Backup
- Open Tag: An Open-Source Claude Tag for Claude Code and Codex
- How Force Merge Compaction Nearly Doubled Milvus Search QPS