Text Analysis for Hybrid Search

This post introduces and details enhancements to Weaviate's text analysis capabilities for hybrid search. It covers tokenization methods (word, lowercase, whitespace, field, and language-specific options), accent folding for multilingual support, and per-property stopword management. A new `/v1/tokenize` endpoint is also introduced for verifying tokenization results. These features aim to improve the recall and accuracy of the BM25 component in hybrid search by providing more control and observability.