Multilingual Full-Text Search

7/30/2025

Milvus 2.6 introduces a completely overhauled text analysis pipeline with comprehensive multi-language support for full-text search. This includes the Multi-Language Analyzer, which allows defining different text processing rules for different languages within the same collection, and the Language Identifier Tokenizer, which automatically detects the language of incoming text and applies appropriate rules. The ICU tokenizer is also integrated as a universal foundation. The post details how these features address challenges in multilingual full-text search, such as complex language segmentation and conflicting language-specific rules, and provides code examples for using these new capabilities with pymilvus.

12/17/2024

This post introduces native lexical search support through Sparse-BM25 in Milvus 2.5, building upon existing hybrid search capabilities. It details the implementation using Tantivy for tokenization and preprocessing, distributed vocabulary and term frequency management, sparse vector generation via TF and TF-IDF, and inverted index support with the WAND algorithm. The post highlights advantages over Elasticsearch, including algorithm flexibility, cost efficiency through compression and quantization, and superior performance in long query optimization by combining sparse embeddings with graph indices.

12/17/2024

This post details the implementation of hybrid search combining dense vector embeddings and BM25-based sparse vectors. It showcases the setup of schema with `enable_analyzer=True` and `Function` objects for automatic BM25 vector generation from text fields. It also demonstrates data insertion and performing both full-text and hybrid searches using `AnnSearchRequest` and `RRFRanker`.

12/17/2024

Milvus 2.5 introduces direct full-text search (FTS) capabilities, also known as lexical or keyword search, using the BM25 metric based on sparse vectors. This allows for inserting and querying text directly, with Milvus handling vector conversions internally. The implementation integrates with the tantivy ecosystem for tokenization and supports raw text ingestion and querying. It also introduces Sparse-BM25 for relevance scoring and optimizations like graph indexing and vector quantization. Additionally, a 'Text Match' feature is added for filtering based on specific text strings, built on tokenization and inverted indexes. Scalar filtering performance is significantly enhanced with the introduction of BitMap Index and Array Inverted Index, improving performance for tag filtering and other metadata filtering scenarios.

12/17/2024

This post introduces and demonstrates the integration of full-text search (FTS) with Milvus's existing semantic and sparse vector search capabilities in version 2.5. It details how hybrid search, combining BM25-based FTS with dense embedding semantic search and ranked by RRF, improves relevance. The post provides a case study using code snippets, analyzing query results and presenting performance metrics (Pass@5) to show the benefits of hybrid search over semantic search alone, and discusses the impact of stopword filtering.

3/20/2024

Introduced multi-vector search allowing simultaneous search and reranking of different vector types within a collection, storing/querying multiple vectors per entity, and building reranking algorithms. Added Grouping Search for fields of types BOOL, INT, or VARCHAR, with an iterator-based implementation for efficiency. Introduced beta support for sparse vector embeddings, treating them on par with dense vectors and supporting Inner Product distance metric. Added support for the GPU-based CAGRA index for accelerated graph indexing. Enhanced metadata filtering with Regular Expression support and introduced a scalar inverted index.