A practical guide to search relevance metrics and evaluation

This post provides a comprehensive overview of search relevance metrics, detailing various offline metrics (precision, recall, F1, P@k, R@k, MAP, MRR, DCG, nDCG, ERR, AUC) and online metrics (CTR). It explains their formulas, interpretations, and use cases in search systems. The post also covers methods for measuring relevance, including offline evaluation with labeled datasets and online evaluation using user behavior signals, as well as techniques for measuring relevance without manual labeling (weak supervision, behavioral signals). It highlights how these metrics are crucial for search engineers and data scientists to evaluate ranking algorithms, optimize search quality, and connect ranking performance to business outcomes.