Multilingual Full-Text Search

6 posts · Mar 2024 – Apr 2026

What this trail covers

Milvus 2.5 enhances full-text search capabilities by introducing the `Function` object and `FunctionType.BM25` for generating sparse vectors from text fields. This allows for hybrid search by combining traditional semantic search with keyword-based full-text search, improving recall for keyword-specific queries. The system supports lexical parsing and tokenization for string fields, enabling efficient full-text indexing and retrieval. Milvus 2.5 also introduces Text Match filters and significant improvements to the underlying lexical search engine, offering a competitive alternative to traditional search engines like Elasticsearch for AI-native applications.

All Milvus trails

Apr 2026How to Fix Hermes Agent's Learning Loop with Milvus 2.6 Hybrid SearchThis post demonstrates the practical application of Milvus 2.6's hybrid search capabilities (BM25 full-text search and vector retrieval with RRF fusion) to enhance the learning loop of the Hermes AI agent. It details how integrating Milvus as a retrieval backend allows Hermes to overcome the limitations of its previous keyword-only search, enabling it to connect rephrased information across sessions and thus write new Skills. The post provides a concrete example of setting up Milvus standalone with a schema supporting both dense and sparse vectors, and includes a Python script for performing hybrid searches. It also touches upon Milvus's tiered storage for cost-effectiveness on small servers.
Dec 2025Phrase Match with Slop in Milvus 2.6: How to Improve Phrase-Level Full-Text Search AccuracyThis post introduces native phrase matching with configurable 'slop' in Milvus 2.6. It explains how phrase matching enforces word order and proximity, and how 'slop' provides flexibility for real-world text variations. The implementation leverages an inverted index with positional information from the Tantivy search engine. The post details the schema configuration required (`enable_analyzer=True`, `enable_match=True` for VARCHAR fields) and provides examples of how different 'slop' values affect search results by filtering candidate documents before vector similarity ranking.
Oct 2025Tokenize Smarter, Retrieve Better: A Deep Dive into Milvus Analyzer for Full-Text SearchThis post details the Milvus Analyzer, a component for full-text search that preprocesses and tokenizes raw text. It describes the Tokenizer (Whitespace, Jieba, Lindera, ICU) and Filter (Lowercase, Asciifolding, Alphanumonly, Stop, Stemmer, Regex) architecture. It introduces three Analyzer types: Built-in (Standard, English, Chinese), Multi-language (auto-detecting tokenizers based on language codes), and Custom Analyzers for tailored configurations. The post provides Python SDK examples for using built-in and multi-language analyzers, demonstrating how to configure tokenizers and filters within FieldSchema for VARCHAR fields and how to integrate the BM25 function for full-text search.
Jul 2025How Milvus 2.6 Upgrades Multilingual Full-Text Search at ScaleThis post details the technical implementation of Milvus 2.6's enhanced multilingual full-text search capabilities. It introduces three key components: the Multi-Language Analyzer (requiring explicit language tagging), the Language Identifier Tokenizer (offering automatic language detection using 'whatlang' or 'lingua' engines), and the ICU tokenizer as a universal foundation. The post provides code examples demonstrating how to configure and use these features with the pymilvus SDK, including schema definition, analyzer parameter setup, and data insertion with language fields. It highlights the trade-offs of automatic detection (latency, handling short/mixed text) and the limitations of the ICU tokenizer when language-specific stemming or lemmatization is required within a single analyzer.
Apr 2025Full-Text Search in Milvus - What's Under the HoodThis post details the internal implementation of Milvus's Sparse-BM25 full-text search feature, introduced in version 2.5. It explains the BM25 ranking algorithm, its transformation into an Inner Product calculation for sparse vectors, and the challenges of dynamic statistics. The post then elaborates on Milvus's improved approach: storing raw text and computing BM25 scores on the fly during query time, with Milvus handling tokenization, stop-word removal, stemming, and global term distribution statistics. It also provides a PyMilvus demo for inserting raw text and performing searches.
Advertisement
Dec 2024Elasticsearch is Dead, Long Live Lexical SearchThis post introduces Sparse-BM25 as a native lexical search capability in Milvus 2.5, building upon previous hybrid search features. It details the implementation using Tantivy for tokenization, distributed vocabulary management, and TF-IDF for sparse vector generation. The post highlights advantages over Elasticsearch, including algorithm flexibility (e.g., TW-BERT), cost efficiency through combined inverted index and dense embedding compression, and optimized long query performance using graph indices. It positions Milvus as a unified solution for both lexical and semantic search in RAG applications.
Dec 2024Getting Started with Hybrid Semantic / Full-Text Search with Milvus 2.5This post details the implementation of full-text search in Milvus 2.5 by introducing the `Function` object and `FunctionType.BM25`. It demonstrates how to define a schema with a VARCHAR field enabled for analysis and a SPARSE_FLOAT_VECTOR field to store BM25 embeddings. The post shows how to add a `Function` to the schema to automatically generate sparse vectors from text, configure a `SPARSE_INVERTED_INDEX` with `BM25` metric type, and perform full-text searches. It also illustrates how to combine this with dense vector search using `hybrid_search` and `RRFRanker` for improved recall.
Dec 2024Introducing Milvus 2.5: Full-Text Search, More Powerful Metadata Filtering, and Usability Improvements!Milvus 2.5 introduces native full-text search using Sparse-BM25, integrating it with semantic search. This involves a tokenizer built on tantivy, direct ingestion/query of raw documents, and internal BM25 scoring based on sparse vectors. It enables hybrid search by representing term frequencies as sparse vectors, allowing for graph indexing, product quantization (PQ), and scalar quantization (SQ) optimizations. The post details configuration for tokenizers and BM25 functions, and search parameters like `drop_ratio_search`. Additionally, Text Match filters are introduced for string matching, and scalar filtering performance is significantly enhanced with new BitMap Index, Array Inverted Index, and tokenized Varchar Inverted Index, with performance gains highlighted through benchmarks.
Dec 2024Semantic Search vs. Full-Text Search: Which Do I Choose in Milvus 2.5?This post details the integration of full-text search (FTS) with semantic search in Milvus 2.5, enabling hybrid search capabilities. It explains the use of BM25 for lexical matching and Reciprocal Rank Fusion (RRF) for merging and ranking results from both semantic (dense embeddings) and full-text search paths. The post provides a practical example using Anthropic's code snippet dataset to demonstrate how hybrid search outperforms individual semantic or full-text search, particularly in scenarios requiring keyword matching. It analyzes specific cases and presents Pass@5 metrics to quantify the improvements, highlighting the value of combining keyword relevance with semantic understanding for AI-assisted coding bots and similar RAG applications.
Mar 2024Unveiling Milvus 2.4: Multi-vector Search, Sparse Vector, CAGRA Index, and More!Introduced multi-vector search allowing simultaneous search and reranking of different vector types within a collection, storing/querying multiple vectors per entity, and building reranking algorithms. Added Grouping Search for fields of types BOOL, INT, or VARCHAR, with an iterator-based implementation for efficiency. Introduced beta support for sparse vector embeddings, treating them on par with dense vectors and supporting Inner Product distance metric. Added support for the GPU-based CAGRA index for accelerated graph indexing. Enhanced metadata filtering with Regular Expression support and introduced a scalar inverted index.