Regex Filtering

1 post · Jun 2023 – Aug 2026

What this trail covers

Milvus 3.0 introduced native regex filtering (`=~`, `!~`) for string fields, moving complex pattern matching from application-side post-processing into the database query path. This post introduces JSON Shredding, which further accelerates JSON queries by transforming row-based JSON documents into highly optimized columnar storage. Milvus analyzes JSON keys to classify them into typed, dynamic, or shared categories, storing them in dedicated or dynamic columns (Arrow/Parquet) or a compact binary-JSON column with an inverted index for shared keys. This enables schema-free agility with columnar performance for semi-structured data, significantly improving query performance for both typed and sparse keys.

All Milvus trails

Aug 2026Beyond =~: How Milvus 3.0 Turns Regex into a Native Database FilterThis post details the implementation of native regex filtering (`=~`, `!~`) in Milvus 3.0. It covers the design choices for user-facing semantics, the use of raw strings for manageable escaping, the adoption of RE2 for predictable cost and avoidance of ReDoS, and a layered execution path including rewriting simple regex, treating regex as a heavy predicate, using Volnitsky string search for literal prefiltering, and NGRAM indexing for candidate generation. It also addresses the handling of nullability with three-valued logic and the importance of preserving correctness when candidate reduction is not provably safe.
Dec 2025Phrase Match with Slop in Milvus 2.6: How to Improve Phrase-Level Full-Text Search AccuracyThis post introduces Phrase Match with slop to Milvus 2.6, enhancing full-text search capabilities. It details how Phrase Match identifies sequences of terms in order, and how the `slop` parameter adds flexibility by defining the allowed "wiggle room" between query terms. The implementation leverages the Tantivy search engine library, using an inverted index with positional information. Document tokenization with positions and inverted index creation are explained. The post outlines the schema configuration required (`enable_analyzer=True`, `enable_match=True` on VARCHAR fields) and provides examples of how different `slop` values affect search results by filtering candidate documents before vector similarity ranking. This feature builds upon the existing regex filtering capability by adding structured phrase matching.
Dec 2025Introducing the Milvus Ngram Index: Faster Keyword Matching and LIKE Queries for Agent WorkloadsThis post introduces the Ngram Index for Milvus, a new indexing strategy designed to significantly accelerate SQL LIKE queries. It details the mechanics of n-gram generation and inverted indexing, and explains how these n-grams are used to filter candidate records for LIKE pattern matching. The post also discusses the trade-offs, such as increased index size, and evaluates performance against brute-force and conventional inverted index approaches. The Ngram Index complements existing regex filtering by providing a specialized index for LIKE patterns.
Dec 2025JSON Shredding in Milvus: 88.9x Faster JSON Filtering with FlexibilityThis post introduces JSON Shredding, a new feature that transforms row-based JSON documents into optimized columnar storage. It classifies JSON keys into typed, dynamic, and shared categories. Typed and dynamic keys are stored in dedicated or dynamic columns using Arrow/Parquet formats. Shared keys are consolidated into a compact binary-JSON column with an inverted index for efficient querying of infrequent fields. Benchmarks show significant performance boosts, up to 88.9x for shared keys and 15-30x for typed keys, compared to storing JSON as a single field.
May 2025Vector Search in the Real World: How to Filter Efficiently Without Killing RecallThis post details advanced filtering techniques in vector search, including graph index optimization with strategies like Alpha and ACORN, metadata-aware indexing using specialized subgraphs, iterative filtering for batched processing, and external filtering at the SDK level. It highlights how Milvus and Zilliz Cloud optimize these techniques to balance recall and performance, especially under selective filtering conditions. The post introduces the concept of "broken connectivity" in graph indexes due to filtering and presents solutions like the Alpha strategy and dynamic fallback to brute-force scans. It also describes metadata-aware indexing with "column graphs" and iterative filtering for batched processing. Finally, it explains external filtering using the search iterator API for client-side filtering.
Advertisement
Jun 2023Milvus 2.2.9: A Highly Anticipated Release with Optimal User ExperienceThis post details the introduction of JSON data type support, dynamic schema functionality via SDK, and the Partition Key feature for enhanced data separation and search efficiency in Milvus 2.2.9. It also announces support for Alibaba Cloud OSS, database support in RBAC, and connection management.