Search Algorithm Evaluation Framework

4 posts · Jan 2021 – Nov 2025

What this trail covers

Shopify has developed and enhanced its search algorithms and the methodologies used to evaluate them. Initial efforts focused on improving search relevance within the Help Center. This post details a three-step framework for evaluating new search algorithms, encompassing data collection (Kafka events, annotation), offline metric evaluation (MAP, NDCG), and online metric evaluation (user interaction, scroll depth, search abandonment). The framework aims to provide a robust system for deploying ML

All Shopify trails

Nov 2025Building world-class product search at Shopify: Where C++ excellence meets ML innovation (2025) - ShopifyIntroduced RankFlow, a domain-specific language, and TurboDSL, an execution engine, to enable data scientists to deploy ML models trained on billions of queries in minutes with C++ performance. Detailed the two-phase rollout of the engine (SimScorerDSL and TurboDSL) and the performance visibility provided on every PR, including component-level timings and statistical significance testing.
Jan 2022Search at Shopify—Range in Data and Engineering is the Future - ShopifyThis post introduces the 'range' principle as a core team philosophy for building search and discovery systems at Shopify. It argues against hyper-specialization and advocates for engineers and data scientists to possess skills in both domains. The post details the dysfunction caused by silos between data science and engineering, where data scientists may focus on bleeding-edge ML without production considerations, and engineers may lack the context to optimize relevance. The proposed solution is a unified team where individuals develop skills in both areas, enabling them to make informed trade-offs between relevance, performance, stability, and maintainability. The author highlights that search, and increasingly other core product features, require this blend of skills to address future challenges involving personalized and conversational interactions.
Apr 2021Building Smarter Search Products: 3 Steps for Evaluating Search Algorithms - ShopifyThis post details a three-step framework for evaluating new search algorithms: 1. Data Collection (Kafka events for real-time monitoring and A/B testing, manual annotation for ground truth relevance ratings). 2. Offline Metric Evaluation (using Mean Average Precision (MAP) and Normalized Discounted Cumulative Gain (NDCG) to assess algorithm performance without user impact, and analyzing query composition for deeper insights). 3. Online Metric Evaluation (using search logs to understand user behavior and evaluate A/B tests, focusing on metrics like interaction rates, scroll depth, and search abandonment). The post introduces 'Vanilla Pagerank' and 'Query-specific Pagerank' as examples.
Jan 2021Apache Beam for Search: Getting Started by Hacking Time - ShopifyThis post introduces Apache Beam as a unified batch and stream processing system for handling clickstream data to improve search relevance. It explains the core challenge of time management in streaming systems, differentiating between event time and processing time, and demonstrates how to configure timestamp policies in Apache Beam, including custom policies for specific event data like SearchQueryEvent, to ensure accurate data ordering and processing. The post uses a movie analogy to explain Beam's time-bending capabilities and mentions the use of windows for further data buffering and collection.
Advertisement