Full Text Search: Architecture and Design

This post details the integration of the Tantivy library for full-text search capabilities within Pinecone. It explains the underlying architecture, including the use of Lucene query syntax, BM25 scoring, and language-specific tokenization, stemming, and stop-word removal. The post also describes how Tantivy segments map to Pinecone slabs and how document ordering is influenced by vector proximity to optimize both text and vector search. It further elaborates on the dynamic management of BM25 statistics for growing datasets by merging slab-level statistics.