TechPrufer
Sign inGet started

Blogs›Weaviate›Text Analysis for Hybrid Search

Text Analysis for Hybrid Search

Text Analysis for Hybrid Search

1
posts
2026

This post introduces and details enhancements to Weaviate's text analysis capabilities for hybrid search. It covers tokenization methods (word, lowercase, whitespace, field, and language-specific options), accent folding for multilingual support, and per-property stopword management. A new `/v1/tokenize` endpoint is also introduced for verifying tokenization results. These features aim to improve the recall and accuracy of the BM25 component in hybrid search by providing more control and observability over text processing.

2026

Text Analysis for Hybrid Search: Tokenization, Stopwords & Accent Folding | Weaviate

5/14/2026

Introduced and detailed four general-purpose tokenization methods (word, lowercase, whitespace, field) and highlighted language-specific tokenizers (gse, gse_ch, kagome_ja, kagome_kr, trigram). Explained and demonstrated accent folding using `ascii_fold` and `ascii_fold_ignore` parameters in the `text_analyzer` configuration. Introduced per-property stopword management via `stopword_presets` and `text_analyzer.stopword_preset`. Added a `/v1/tokenize` endpoint for verifying tokenization results.