How to Build an Experiment Pipeline from Scratch - Shopify

This post details the creation of an email experimentation pipeline from scratch to address the limitations of Shopify's existing web-based A/B testing framework (Verdict) for external channels like email. The pipeline enables randomized assignment of users to experiment variations, exclusion of subjects from other experiments, and ongoing randomization. It outlines the problem, planning phases (understanding the problem, system diagram, ideal output, technical planning), building the pipeline in PySpark, and shipping and documenting the tool. Key requirements gathered include excluding subjects from other experiments, identifying missing experiment category tags, excluding linked shops, enabling ongoing randomization, and backfilling past experiments.