ShopGym: Realistic, reproducible sandboxes for shopping agents (2026) - Shopify

This post introduces ShopGym, an evolution of simulated A/B testing for shopping agents. ShopGym builds upon previous work by creating self-contained, resettable sandbox shops from live storefronts (ShopArena) and generating benchmark tasks tied to their specific products, features, and policies (ShopGuru). It addresses challenges of training and evaluating agents due to changing store conditions, bot detection, and the limitations of hand-built test stores. ShopGym's ShopArena converts live storefronts into anonymized shop specifications and generates sandbox shops through a staged, validated process. ShopGuru synthesizes benchmark tasks across seven skill categories, grounded in the shop's catalog, navigation, policies, and affordances. The framework was validated through graph-based structural analysis and agent-based behavioral evaluation, showing that synthetic shops preserve key structural properties of live storefronts and agent performance correlates positively.