What is Vector Search?

This post details the technical underpinnings of vector search, including its definition, how it differs from traditional keyword search, and its role in modern AI systems like RAG. It explains vector embeddings, the process of generating them using models like BERT, and how they capture semantic meaning. The post also discusses customizing vector search for enterprises through fine-tuning embeddings to understand domain-specific terminology. A six-step pipeline for how vector search works is outlined: data ingestion, embedding generation, indexing (HNSW), query processing, similarity search (ANN), and scoring/ranking with permission checks. It highlights benchmarking results showing open-source embeddings can outperform commercial ones and emphasizes the importance of a hybrid search system combining vector search with keyword search and personalization.