NEAREST BY Join: Scaling Vector Search in Databricks Runtime

This post details the engineering effort behind integrating vector search into Databricks Runtime as a SQL join. It covers the architectural design, the query rewrite process to translate vector search operations into SQL, deep optimizations within the Photon execution engine for high-performance vector computations, and the development of a vector index stored in an open format. The core contribution is making vector search a native, scalable SQL operation.