TechPrufer
Sign inGet started

Blogs›Shopify›Search Algorithm Evaluation Framework

Search Algorithm Evaluation Framework

Search Algorithm Evaluation Framework

4
posts
2021–2025

Shopify has developed and enhanced its search algorithms and the methodologies used to evaluate them. Initial efforts focused on improving search relevance within the Help Center. This post details a three-step framework for evaluating new search algorithms, encompassing data collection (Kafka events, annotation), offline metric evaluation (MAP, NDCG), and online metric evaluation (user interaction, scroll depth, search abandonment). The framework aims to provide a robust system for deploying ML models like transformers and gradient boosting algorithms at native C++ speed using RankFlow and TurboDSL, enabling rapid iteration and millisecond latency at scale.

2025

Building world-class product search at Shopify: Where C++ excellence meets ML innovation (2025) - Shopify

11/12/2025

Introduced RankFlow, a domain-specific language, and TurboDSL, an execution engine, to enable data scientists to deploy ML models trained on billions of queries in minutes with C++ performance. Detailed the two-phase rollout of the engine (SimScorerDSL and TurboDSL) and the performance visibility provided on every PR, including component-level timings and statistical significance testing.

2022

Search at Shopify—Range in Data and Engineering is the Future - Shopify

1/14/2022

This post emphasizes the importance of 'range' (cross-disciplinary expertise) in search development, advocating for a unified team structure where engineers and data scientists collaborate closely. It highlights the dysfunction that arises from silos and proposes that individuals should possess both data science and engineering skills to make optimal trade-offs in relevance, performance, and stability. The post argues that this integrated approach is the future of building intelligent and personalized user experiences.

2021

Building Smarter Search Products: 3 Steps for Evaluating Search Algorithms - Shopify

4/1/2021

This post introduces a three-step framework for evaluating search algorithms. Step 1 details data collection methods using Kafka for real-time user interactions and manual annotation for ground truth relevance ratings. Step 2 explains offline evaluation using Mean Average Precision (MAP) and Normalized Discounted Cumulative Gain (NDCG) to assess algorithm performance on historical data. Step 3 outlines online evaluation through A/B testing and user behavior metrics. The post also introduces a new algorithm, 'Query-specific Pagerank', and contrasts it with the existing 'Vanilla Pagerank'.

Apache Beam for Search: Getting Started by Hacking Time - Shopify

1/8/2021

This post introduces Apache Beam as a unified batch and stream processing system for handling clickstream data to improve search relevance. It explains the core challenge of time management in streaming systems, differentiating between event time and processing time, and demonstrates how to configure timestamp policies in Apache Beam, including custom policies for specific event data like SearchQueryEvent, to ensure accurate data ordering and processing. The post uses a movie analogy to explain Beam's time-bending capabilities and mentions the use of windows for further data buffering and collection.