Full Text Search

9 posts · May 2021 – Sep 2026

What this trail covers

Pinecone now supports full text search by integrating the Tantivy library, enabling advanced text functionality alongside vectors and metadata. This includes full Lucene-syntax queries, multi-field schemas, BM25 scoring, tokenization in 18 languages, stemming and stop-word removal, phrase matching search, and new filters based on text match for vector search. The system leverages Tantivy's Rust implementation and maps cleanly onto Pinecone's slab architecture. Document ordering is informed by vector similarity and BM25 scores, allowing for hybrid search. This complements existing capabilities by providing exact-match precision for identifiers and phrases, addressing limitations of purely semantic search.

All Pinecone trails

Sep 2026Full-Text Search is Now Generally Available In Pinecone DatabaseThis post announces the general availability of full-text search in Pinecone, integrating BM25 keyword ranking and text-match filters directly into the vector database. It highlights the ability to handle queries for identifiers (SKUs, error codes) and exact phrases that semantic search struggles with. The implementation leverages the Tantivy library and maps onto Pinecone's existing slab architecture, allowing for a single index and schema for text, dense, and sparse vectors. Key features include Lucene query syntax, fuzzy matching, multi-language tokenization/stemming, and text-match filters for hybrid search. This addresses the business cost of silent failures and high ownership costs associated with separate lexical search systems.
Jul 2026Text match filters for agentsThis post introduces text match filters as an extension to Pinecone's Full Text Search capability. It details how these filters address the ambiguity problem in semantic search, particularly for agentic applications, by allowing lexical queries to restrict the candidate pool before vector search. The post provides a concrete example using CNN news articles and demonstrates how text match filters, when combined with semantic search, yield more contextually relevant results. It highlights the cost savings for agentic pipelines by preventing wasted token usage and downstream tool calls due to incorrect initial retrievals. The technical contribution is the introduction of a mechanism to combine lexical and semantic filtering at query time, reducing the need for extensive pre-labeling of data.
Jul 2026Sparse V3: how Pinecone's sparse index learned to skipThis post details the V3 sparse index architecture, which reorganizes posting data from a document-major to a term-major layout. This change significantly reduces I/O by only loading blocks relevant to the queried terms, leading to 151x less data loaded for SPLADE queries and 1,428x less for BM25 queries from disk. Key technical contributions include the term-major layout, an in-memory Elias-Fano encoded term directory for O(1) term lookup, metadata blocks for pre-load skipping decisions, and per-term score quantization and document ID offset compression for smaller posting blocks. The post also highlights the empirical, property-based testing methodology used to tune heuristics for optimal performance.
May 2026Full Text Search in Pinecone, Now in Public PreviewThis post announces the public preview of Full Text Search (FTS) in Pinecone, detailing its integration with semantic search. It introduces BM25 scoring, Lucene query syntax, and multi-language tokenization. The core technical contribution is the ability to store text fields, dense vectors, sparse vectors, and metadata within a single index. It explains how text fields can be configured per index for independent scoring and filtering, and how text match filters can narrow candidate sets before vector ranking. The post highlights that scoring operates on one type at a time (BM25, dense, or sparse) within a single query, and that combining scores across modalities requires separate queries and client-side merging. It also notes that schema is fixed at index creation, preventing conversion of existing indexes to FTS until mutable schema is available.
May 2026Full Text Search: Architecture and DesignThis post details the integration of the Tantivy library for full-text search capabilities within Pinecone. It explains the underlying architecture, including the use of Lucene query syntax, BM25 scoring, and language-specific tokenization, stemming, and stop-word removal. The post also describes how Tantivy segments map to Pinecone slabs and how document ordering is influenced by vector proximity to optimize both text and vector search. It further elaborates on the dynamic management of BM25 statistics for growing datasets by merging slab-level statistics.
Advertisement
Dec 2024Introducing cascading retrieval: Unifying dense and sparse with rerankingIntroduced sparse-only vector index types for direct indexing and retrieval of sparse vectors, supporting traditional methods like BM25 and learned sparse models. Released pinecone-sparse-english-v0, a sparse embedding model built on the DeepImpact architecture, which estimates lexical importance of tokens based on context and uses whole-word tokenization for improved precision and model-free queries for speed. Integrated state-of-the-art rerankers (cohere-rerank-3.5 and pinecone-rerank-v0) to refine initial candidate results from dense and sparse retrieval by scoring query-document pairs based on relevance, enabling a unified search pipeline.
Apr 2024Sixfold's Transformation of Insurance Underwriting with PineconeThis post details how Sixfold leverages Pinecone's hybrid search capabilities to transform insurance underwriting. It highlights the use of both semantic (vector) search and keyword-based search to retrieve relevant information from complex datasets like medical records and insurance applications. The post emphasizes the practical application of chunking, metadata extraction, and RAG within the Pinecone ecosystem to provide contextualized insights for underwriters, improving accuracy and efficiency.
Jul 2023Less is More: Why Use Retrieval Instead of Larger Context WindowsThis post provides experimental evidence and analysis supporting the efficacy of retrieval-augmented generation (RAG) over large context windows for LLMs. It details experiments using the Natural Questions (NQ) and QuALITY datasets to show that LLMs struggle with large, unfiltered contexts, leading to decreased accuracy and increased costs. The post highlights how retrieval systems, like Pinecone, can extract relevant information more efficiently, improving accuracy per token and reducing resource consumption. It contrasts the performance of LLMs with large contexts versus those augmented with retrieved segments, demonstrating significant cost savings and maintained accuracy.
Feb 2023Introducing support for sparse-dense embeddings for better search resultThis post introduces public preview support for sparse-dense embeddings, enabling hybrid search. It details the technical differences between dense and sparse vectors, highlights the flexibility and simplicity of the new approach, and provides guidance on generating and upserting sparse-dense vectors. It also includes performance data comparing hybrid search with different sparse models and provides capacity projections for s1 and p1 pods. The post emphasizes the ability to combine any sparse representation with dense embeddings for improved search relevance.
Oct 2022Introducing the hybrid index to enable keyword-aware semantic searchIntroduced the hybrid vector index, a new capability that stores and searches across both dense and sparse vector representations. This includes a new hybrid API endpoint that accepts dense vectors (from language models) and term frequencies (sparse vectors) for uploading and querying. The system automatically normalizes sparse vectors to provide BM25-equivalent scoring and allows users to control the weight of keyword vs. semantic relevance through an 'alpha' parameter during queries. The architecture supports billions of vectors with low latency and zero-downtime scaling, similar to s1 pods.
May 2021The Rise of Vector DataThis post introduces the concept of vector embeddings as rich representations of data and explains how they are generated by embedding models. It highlights the value of storing and searching through this vector data for various applications like search engines and recommendation systems. The post also discusses the challenges of implementing scalable vector search, including the complexity of algorithms and the infrastructure required, and positions Pinecone as a managed solution to address these challenges.