Measure Less to Learn More: Using Fewer, Higher-quality Metrics to Capture What Matters

This post details Discord's journey to address the issue of having too many experiment metrics. It highlights the tradeoff between leaving p-values as-is (leading to potential false positives) and adjusting p-values (leading to worse recall). The core contribution is the conclusion that the best solution is to use fewer, high-quality metrics that capture distinct concepts.