Data-Centric Machine Learning: Building Shopify Inbox’s Message Classification Model - Shopify

What this post added

This post details the application of a data-centric machine learning approach to build a message classification model for Shopify Inbox. It outlines the transition from a model-centric approach to focusing on data quality. Key contributions include: defining a new, more granular taxonomy with over 40 topics grouped into five categories (Products, Pre-Purchase, Post-Purchase, Store, Miscellaneous); developing an in-house annotation tool; implementing a systematic annotation process with inter-annotator agreement measurement; and fine-tuning a DistilBERT model on the curated dataset to achieve improved accuracy and coverage compared to the initial prototype. The post also discusses the challenges of handling 'other' categories and setting probability thresholds for production deployment.