Under the hood: Photo Search

This post details the application of machine learning, specifically deep neural networks, to enhance photo search capabilities on Facebook. It describes the image understanding engine that extracts semantic meaning from billions of photos, producing high-dimensional feature vectors. These vectors are compressed using iterative quantization and locality-sensitive hashing into compact embeddings. The post explains how these embeddings, along with object tags, are used to populate Unicorn, an in-memory and flash storage indexing system, for efficient search. It also elaborates on the relevance modeling, which combines concept relevance and embedding relevance using multimodal learning and ranking loss to provide low-latency, accurate search results. Furthermore, it outlines query understanding techniques like intent recognition, syntactic analysis, entity linking, and query embedding, and describes the multi-stage ranking process (first-stage and second-stage re-ranking) to deliver relevant photos.