Clustering billions of products for agentic commerce with Catalog API (2026) - Shopify

This post details the development of a product clustering system for Shopify Catalog, which aims to create a unified intelligence layer for product data. It addresses the challenge of standardizing product listings across millions of stores for AI shopping agents. The system uses a two-stage LLM pipeline for intra-store clustering, starting with a singleton detector to identify products that don't require clustering. For shops that do require clustering, it employs Approximate Nearest Neighbor (ANN) retrieval with HNSW and a modified average-linkage approach (UPGMA) to create semantically related product neighborhoods (chunks) before LLM processing. The core principle for LLM prompting is the 'core value proposition' framework, which determines if an attribute is a variant or defines product identity. Precision is prioritized over recall, with quality measured using curated ground-truth datasets and LLM-based judges.