Multimodal Semantic Search with Images and Text

This post details the construction of a multimodal semantic search application using Milvus, HuggingFace's Visualized BGE model for joint text and image embedding, and Phi-3 Vision for reranking. It showcases how to index images and text into the same vector space, perform similarity searches with combined text and image queries, and improve result relevance through LLVM-based reranking. The architecture follows a retrieve-and-rerank pattern, similar to RAG, and highlights the ability of embedding models to understand open-ended instructions in relation to query images.