Text Analysis for Hybrid Search: Tokenization, Stopwords & Accent Folding | Weaviate

What this post added

Introduced and detailed four general-purpose tokenization methods (word, lowercase, whitespace, field) and highlighted language-specific tokenizers (gse, gse_ch, kagome_ja, kagome_kr, trigram). Explained and demonstrated accent folding using `ascii_fold` and `ascii_fold_ignore` parameters in the `text_analyzer` configuration. Introduced per-property stopword management via `stopword_presets` and `text_analyzer.stopword_preset`. Added a `/v1/tokenize` endpoint for verifying tokenization results.