Leveraging multimodal LLMs for Shopify’s global catalogue: Recap of expo talk at ICLR 2025 - Shopify

What this post added

This post details the engineering behind Shopify's Global Catalogue initiative, which uses multimodal Large Language Models (LLMs) to organize and augment product data. It covers data curation, model fine-tuning (moving from LlaVA 1.5 7B to LLaMA 3.2 11B to Qwen2VL 7B), selective field extraction for training to improve generalization and reduce latency/GPU usage, automated annotation pipelines using LLM agents and human expertise, and the infrastructure for making 40 million multimodal LLM-powered inferences daily. The four integrated layers of the Global Catalogue are described: product data foundation, product understanding (classification, attribute extraction, image understanding, title standardization, description analysis, review summarization), product matching (candidate generation, discriminator models, graph-based clustering), and reconciliation (canonical product record construction).