Full Text Search

5/7/2026

Introduces Full Text Search (FTS) as a new capability in Pinecone, allowing BM25 scoring across multiple text fields, Lucene query syntax, and multi-language tokenization. Details the schema definition for text fields, dense vectors, sparse vectors, and metadata within a single index. Explains how FTS can be used as a filter before vector ranking and demonstrates query examples using Python SDK. Notes that scoring is modality-specific per query and that mutable schema is required for existing indexes to adopt FTS.

5/7/2026

This post details the integration of the Tantivy library for advanced text functionality, including Lucene query syntax, 18-language tokenization, stemming, stop-word removal, phrase matching, and new text match filter operators. It also explains the BM25 scoring algorithm and design choices for document ordering and handling dynamic data statistics within Pinecone's slab-based architecture.

12/2/2024

Introduced sparse-only vector index type (in early access), the pinecone-sparse-english-v0 sparse embedding model with whole-word tokenization and model-free queries, and integrated reranking capabilities (cohere-rerank-3.5, pinecone-rerank-v0). These features enable cascading retrieval pipelines combining dense, sparse, and reranking.

4/5/2024

This post details how Sixfold leverages Pinecone's hybrid search capabilities to transform insurance underwriting. It highlights the use of both semantic (vector) search and keyword-based search to retrieve relevant information from complex datasets like medical records and insurance applications. The post emphasizes the practical application of chunking, metadata extraction, and RAG within the Pinecone ecosystem to provide contextualized insights for underwriters, improving accuracy and efficiency.

7/20/2023

This post provides experimental evidence and analysis supporting the efficacy of retrieval-augmented generation (RAG) over large context windows for LLMs. It details experiments using the Natural Questions (NQ) and QuALITY datasets to show that LLMs struggle with large, unfiltered contexts, leading to decreased accuracy and increased costs. The post highlights how retrieval systems, like Pinecone, can extract relevant information more efficiently, improving accuracy per token and reducing resource consumption. It contrasts the performance of LLMs with large contexts versus those augmented with retrieved segments, demonstrating significant cost savings and maintained accuracy.

2/23/2023

Introduces support for sparse-dense embeddings, enabling hybrid search by combining keyword-aware semantic search with traditional keyword search. This allows for flexible ingestion of sparse or dense data into any index, simple integration via REST API or SDK, and supports data from any dense or sparse model, including advanced use cases like multimodal search and boosting. The post details how to upsert and query sparse-dense vectors, including specific pod type and metric requirements, and provides capacity projections for s1 and p1 pods.

10/31/2022

Introduced the hybrid vector index, a novel approach to combine dense (semantic) and sparse (keyword) vector representations within a single index. This enables keyword-aware semantic search by allowing users to upload and query with both dense vectors from language models and sparse vectors derived from term frequencies. The API now supports a hybrid endpoint for upserting and querying, accepting both vector types and an 'alpha' parameter to control the weighting between keyword and semantic relevance. This simplifies hybrid search implementation by eliminating the need for separate keyword and vector search systems and a reranking system.