Evolution of Product Classification at Shopify: From Categories to Comprehensive Product Understanding (2025) - Shopify

This post details the evolution of Shopify's product classification system from basic ML models to a multi-modal approach and finally to a Vision Language Model (VLMs) based system integrated with the Shopify Product Taxonomy. Key technical contributions include: 1. Use of VLMs (LlaVA 1.5 7B, LLaMA 3.2 11B, Qwen2VL 7B) for true multi-modal understanding, zero-shot learning, and natural language reasoning. 2. Inference optimizations: FP8 quantization for reduced GPU memory and efficient batching, in-flight batching using Nvidia Dynamo for dynamic request grouping and adaptive processing, and KV cache optimization for LLM inference speed. 3. A near real-time processing pipeline orchestrated by Dataflow, making two calls to the VLM service (category and attribute prediction), with transaction-like handling and retries. 4. A multi-stage annotation system for training data quality, involving multiple LLMs and human validation. 5. Impact metrics showing an 85% acceptance rate of predicted categories, doubled hierarchical precision and recall, and daily processing of over 30 million predictions. Future plans include migrating to a DAG taxonomy and enhancing metadata extraction.