Building Smarter Search Products: 3 Steps for Evaluating Search Algorithms - Shopify

What this post added

This post introduces a three-step framework for evaluating search algorithms. Step 1 details data collection methods using Kafka for real-time user interactions and manual annotation for ground truth relevance ratings. Step 2 explains offline evaluation using Mean Average Precision (MAP) and Normalized Discounted Cumulative Gain (NDCG) to assess algorithm performance on historical data. Step 3 outlines online evaluation through A/B testing and user behavior metrics. The post also introduces a new algorithm, 'Query-specific Pagerank', and contrasts it with the existing 'Vanilla Pagerank'.