How Shopify improved consumer search intent with real-time ML - Shopify

This post details the implementation of real-time ML embeddings for Shopify Storefront Search, leveraging Google Cloud Dataflow for streaming pipelines. It describes the architecture for processing image and text embeddings, including preprocessing, inference, and writing to a data warehouse and downstream systems. Key technical challenges and trade-offs are discussed, such as managing data in memory by adjusting Dataflow worker threads (reducing memory footprint by 2.6x and eliminating cost increase), managing models in memory (opting for default Dataflow configuration due to small model size and throughput considerations), and implementing batching for GPU utilization by overriding `batch_element_kwargs` in the ModelHandler. The post highlights the decision to use near real-time embeddings over batch solutions to meet merchant and customer expectations for instant updates, despite increased pipeline complexity.