Scaling Shopify's BFCM Live Map: An Apache Flink Redesign - Shopify

What this post added

This post details the redesign of Shopify's BFCM live map infrastructure using Apache Flink to address scalability issues and introduce new metrics. The previous system, Cricket (a Go-based stateful streaming service), was unable to handle the projected volume of checkout events. The new solution leverages Flink for filtering irrelevant checkout events and processing new metrics like product trends and unique shoppers. Flink was placed on the critical path, allowing Cricket to handle a reduced event volume for existing metrics. New metrics were computed by Flink and relayed through Cricket. The product trends metric involved calculating sales quantity percentage change over time using a specific formula and tumbling windows. The Flink jobs achieved 100% uptime during BFCM. A minor issue with Redis CPU load due to message fetching and the memory footprint of the new metric was resolved by dropping unnecessary Redis state.