TechPrufer
Get started
Shopify
Shopify logoShopify · Feature trail

Message Classification Model for Shopify Inbox

1 post · Dec 2021 – Jul 2022

What this trail covers

Shopify has developed and enhanced its AI-powered message classification model for Shopify Inbox. Initial efforts focused on building a prototype using a data-centric approach and a pre-trained Universal Sentence Encoder model to understand buyer intent. This led to the identification of key areas like order status requests. Subsequent development involved creating a more robust taxonomy with over 40 topics, refining annotation processes, and fine-tuning a DistilBERT model for improved accuracy. This post details the development of a real-time buyer signal data pipeline using Apache Kafka and Apache Beam to surface relevant customer context—including active cart activities and order completion information—to merchants while they're chatting with their customers. The pipeline leverages Monorail and Change Data Capture (CDC) for data ingestion, Apache Beam for stream processing with windowing, timers, and stateful processing, and Google Cloud Dataflow for deployment. The system filters events and aggregates customer activities to provide real-time, high-intent buyer signals to merchants in Shopify Inbox, aiming to optimize sales opportunities and prioritize conversations.

All Shopify trails
  1. Data-Centric Machine Learning: Building Shopify Inbox’s Message Classification Model - ShopifyThis post details the evolution of Shopify Inbox's message classification model by applying a data-centric approach. It describes the transition from an initial prototype using Universal Sentence Encoder and clustering techniques to a more robust system utilizing DistilBERT. Key contributions include the development of a new taxonomy with over 40 topics grouped into five categories (Products, Pre-Purchase, Post-Purchase, Store, Miscellaneous), a systematic annotation process with inter-annotator agreement measurement, and the fine-tuning of the DistilBERT model for improved accuracy and coverage. The post highlights the importance of high-quality training data and the benefits of a data-centric ML process.
  2. Building a Real-time Buyer Signal Data Pipeline for Shopify Inbox - ShopifyThis post details the design and architecture of a real-time buyer signal data pipeline for Shopify Inbox. It introduces the use of Apache Kafka (Monorail and CDC) for data ingestion, Apache Beam for unified batch and stream processing (including windowing, timers, and stateful processing with Global Window and ValueState), and Google Cloud Dataflow as the runner. The pipeline includes Events Filtering Jobs to retain mission-critical events and a Customer Events Aggregation Job that uses CoGroupByKey to group events by buyer identifier and maintains state for conversation, checkout, and cart activities. The system emits events only when a buyer has started a conversation and differentiates between pre-conversation and post-conversation events to provide timely and relevant buyer signals to merchants.