Full Text Search: Architecture and Design

What this post added

This post details the integration of the Tantivy library for advanced text functionality, including Lucene query syntax, 18-language tokenization, stemming, stop-word removal, phrase matching, and new text match filter operators. It also explains the BM25 scoring algorithm and design choices for document ordering and handling dynamic data statistics within Pinecone's slab-based architecture.