TechPrufer
Sign inGet started

Blogs›Exa›BM25 Index Optimization

BM25 Index Optimization

BM25 Index Optimization

1
posts
2025

Exa has significantly optimized its BM25 index, reducing memory overhead by over 50% across billions of documents. This was achieved through a combination of algorithmic improvements like smart candidate selection and dynamic pruning, and low-level memory structure optimizations. Key techniques include frequency-based organization, variable-length + delta encoding for document IDs, Zstd compression, consolidated buffers, singleton optimization, and further consolidation of buffers into a single continuous vector. These optimizations not only reduce memory consumption and operational costs but have also led to a 10% improvement in average query latency due to more predictable memory access patterns.

2025

Optimizing BM25 for the Next Generation of Semantic Search Technology

5/5/2025

This post details the technical optimizations made to Exa's BM25 index to reduce memory overhead by over 50%. It introduces a smart candidate selection and dynamic pruning algorithm for query processing. For memory optimization, it describes grouping documents by term frequency, implementing variable-length + delta encoding for document IDs, applying Zstd compression, consolidating postings lists into single buffers, optimizing for singleton tokens, and further consolidating all postings lists into one continuous vector with offset/length pointers. The post also notes a 10% improvement in query latency as a result of these changes.