Introducing the hybrid index to enable keyword-aware semantic search
Oct 31, 2022Gibbs Cullen
Summary
Introduced the hybrid vector index, a new capability that stores and searches across both dense and sparse vector representations. This includes a new hybrid API endpoint that accepts dense vectors (from language models) and term frequencies (sparse vectors) for uploading and querying. The system automatically normalizes sparse vectors to provide BM25-equivalent scoring and allows users to control the weight of keyword vs. semantic relevance through an 'alpha' parameter during queries. The architecture supports billions of vectors with low latency and zero-downtime scaling, similar to s1 pods.