Introducing Milvus 2.5: Full-Text Search, More Powerful Metadata Filtering, and Usability Improvements!

Milvus 2.5 introduces native full-text search using Sparse-BM25, integrating it with semantic search. This involves a tokenizer built on tantivy, direct ingestion/query of raw documents, and internal BM25 scoring based on sparse vectors. It enables hybrid search by representing term frequencies as sparse vectors, allowing for graph indexing, product quantization (PQ), and scalar quantization (SQ) optimizations. The post details configuration for tokenizers and BM25 functions, and search parameters like `drop_ratio_search`. Additionally, Text Match filters are introduced for string matching, and scalar filtering performance is significantly enhanced with new BitMap Index, Array Inverted Index, and tokenized Varchar Inverted Index, with performance gains highlighted through benchmarks.