Scaling Shopify's BFCM Live Map: An Apache Flink Redesign - Shopify

This post details the redesign of the BFCM live map's data infrastructure using Apache Flink to address scalability issues and introduce new metrics. The previous Go-based streaming service (Cricket) was a bottleneck. The new Flink pipeline filters irrelevant checkout events, handles high volumes (up to 1 million events/sec in load tests), and processes new metrics like product trends and unique shoppers. The system achieved 100% uptime during BFCM with no manual intervention. A minor issue with Redis was quickly resolved. Future plans involve consolidating all processing logic within Flink to further simplify the system.