The Journey to Optimizing Billion-scale Image Search (2/2)

This post details the technical implementation of a search-by-image system using CNNs (VGG16) for feature extraction and Milvus as the vector search engine. It covers feature extraction using Keras/TensorFlow, image preprocessing (normalization, bytes conversion, black border removal), Milvus requirements (CPU instruction set), capacity planning for billion-scale datasets (calculating storage for 512-dimensional float32 vectors), system configuration, database design considerations (collections, partitions, metadata management with SQLite/MySQL, limitations on number of collections/partitions), structured data and vector mapping via IDs, index selection, and processing search results (ID + distance, filtering -1 IDs, pagination).