Building Smarter Search Products: 3 Steps for Evaluating Search Algorithms - Shopify

This post details a three-step framework for evaluating new search algorithms: 1. Data Collection (Kafka events for real-time monitoring and A/B testing, manual annotation for ground truth relevance ratings). 2. Offline Metric Evaluation (using Mean Average Precision (MAP) and Normalized Discounted Cumulative Gain (NDCG) to assess algorithm performance without user impact, and analyzing query composition for deeper insights). 3. Online Metric Evaluation (using search logs to understand user behavior and evaluate A/B tests, focusing on metrics like interaction rates, scroll depth, and search abandonment). The post introduces 'Vanilla Pagerank' and 'Query-specific Pagerank' as examples.