Unlocking the Power of Vector Search in Enterprise

This post details the benchmarking of text embedding offerings for enterprise search, comparing commercial API providers (OpenAI, Cohere) with open-source models (E5-large, Instructor-XL, MPNet). It demonstrates that open-source embeddings outperform commercial ones for enterprise search tasks based on NDCG@10 and R@100 metrics. Furthermore, it introduces Glean's method for fine-tuning embeddings with company-specific language, showing significant performance improvements over generic models and continuous adaptation over time. The post also reiterates Glean's hybrid search approach combining vector search, keyword search, and personalization.