Introducing cascading retrieval: Unifying dense and sparse with reranking

Introduced sparse-only vector index type for direct indexing and retrieval of sparse vectors, supporting BM25 and learned sparse models. Developed pinecone-sparse-english-v0, a sparse embedding model leveraging DeepImpact architecture for contextual token importance estimation, outperforming BM25 on NDCG@10 and BEIR benchmarks. Enhanced precision with whole-word tokenization and dynamic vocabulary generation, and increased speed with model-free queries. Integrated cohere-rerank-3.5 and pinecone-rerank-v0 for refining search results, showing significant performance improvements over dense/sparse retrieval alone or score fusion.