Pinecone algorithms set new records for BigANN

What this post added

This post details Pinecone's participation and success in the BigANN 2023 competition, showcasing new algorithms and optimizations for vector search. Key contributions include: - Filter track: An IVF architecture combined with a metadata inverted index, enabling query selectivity-based cluster scanning and efficient vector retrieval using pre-computed inverted list intersections and AVX for distance computation. Hyperparameter optimization was formalized as a constrained convex optimization problem. - Sparse track: A novel algorithm based on clustering sparse vectors, an inverted index with a novel structure, and a hybrid approach combining IVF and graph-based methods. This includes a k-MIP graph for expanding retrieved vectors and a compressed forward index for re-ranking. - OOD track: A solution similar to the Sparse track but with an IVF index tailored for inner-product search, a k-MIP graph constructed using co-occurrence of vectors as nearest neighbors, and quantization for SIMD-based acceleration. - Streaming track: An algorithm that achieved high recall under memory and time constraints, demonstrating adaptability to interleaved insertions, deletions, and search requests. These algorithmic advancements are being integrated into Pinecone's core product to enhance performance and cost-effectiveness.