Building a Real-time Buyer Signal Data Pipeline for Shopify Inbox - Shopify

What this post added

This post details the design and implementation of a real-time buyer signal data pipeline for Shopify Inbox. It introduces the use of Apache Kafka (Monorail and CDC), Apache Beam, and Google Cloud Dataflow to surface active cart activities and order completion information to merchants during customer conversations. Key architectural components include event filtering jobs and a customer events aggregation job that leverages stateful processing, timers, and global windows to provide up-to-date buyer context. The post also discusses A/B testing and experiment measurement for evaluating the pipeline's success.