10 Lessons Learned From Online Experiments - Shopify

This post details lessons learned from online experiments, focusing on the importance of choosing appropriate randomization units (e.g., users over sessions to avoid carryover effects), the utility of simulations for validating statistical assumptions, and the application of data to inform both large and small decisions. It also highlights the need to understand system behavior, log generously, and run A/A tests to validate experimental setups, particularly in the context of redirects and sample ratio mismatches.