Using Rich Image and Text Data to Categorize Products at Scale - Shopify

This post details the modernization of Shopify's product categorization model, which now incorporates both rich image and text data. The new model uses transfer learning with pre-trained Multi-Lingual BERT for text and MobileNet-V2 for images, converting them into embeddings. The architecture is framed as a multi-task, multi-class classification problem, with each level of the Google Product Taxonomy (GPT) treated as a separate classification task. The output of each layer is fed back into the next to improve predictions. The model is trained using distributed TensorFlow on Google Cloud Platform with data parallelization to handle the massive dataset and model complexity. During inference, additional logic is applied to ensure hierarchical consistency, selecting the highest confidence category at Level 1 and then choosing the highest confidence descendant at subsequent levels. This approach increased leaf precision by 8% and doubled coverage.