Blogs›Shopify›Message Classification Model for Shopify Inbox

Message Classification Model for Shopify Inbox

posts

2022

This feature thread tracks the development and enhancement of Shopify's AI-powered message classification model for Shopify Inbox. Initial efforts focused on building a prototype using a data-centric approach and a pre-trained Universal Sentence Encoder model to understand buyer intent. This led to the identification of key areas like order status requests. Subsequent development involved creating a more robust taxonomy with over 40 topics, refining annotation processes, and fine-tuning a DistilBERT model to achieve higher accuracy and coverage. The focus remains on improving the model's ability to accurately categorize merchant-buyer communications to enhance the merchant experience and inform product development.

2022

Data-Centric Machine Learning: Building Shopify Inbox’s Message Classification Model - Shopify

7/21/2022

This post details the application of a data-centric machine learning approach to build a message classification model for Shopify Inbox. It outlines the transition from a model-centric approach to focusing on data quality. Key contributions include: defining a new, more granular taxonomy with over 40 topics grouped into five categories (Products, Pre-Purchase, Post-Purchase, Store, Miscellaneous); developing an in-house annotation tool; implementing a systematic annotation process with inter-annotator agreement measurement; and fine-tuning a DistilBERT model on the curated dataset to achieve improved accuracy and coverage compared to the initial prototype. The post also discusses the challenges of handling 'other' categories and setting probability thresholds for production deployment.