Leveraging multimodal LLMs for Shopify’s global catalogue: Recap of expo talk at ICLR 2025 - Shopify

This post details the engineering behind Shopify's Global Catalogue, which uses multimodal LLMs to organize and augment product data. It covers data curation, model fine-tuning, and the infrastructure for daily inferences. Key technical contributions include: a multi-task, multi-entity approach for product understanding using fine-tuned vision LLMs (LlaVA 1.5 7B, LLaMA 3.2 11B, Qwen2VL 7B); selective field extraction during fine-tuning to improve generalization and reduce latency/GPU usage; an automated annotation pipeline using LLM agents and human expertise for data generation; a product matching system using bipartite graphs and discriminator models for clustering related listings; and a reconciliation layer to construct canonical product records. The post also discusses the challenges of fragmented product data and the impact of these pipelines on search, recommendations, and conversational commerce.