Building production-ready agentic systems: Lessons from Shopify Sidekick (2025) - Shopify

What this post added

This post details the development of Sidekick, an AI-powered assistant for merchants. Key contributions include: 1. Architecture evolution with Just-in-Time (JIT) instructions to manage tool complexity and improve maintainability and performance. 2. Development of robust LLM evaluation systems using Ground Truth Sets (GTX) and LLM-as-a-Judge calibrated against human judgment, along with LLM-powered merchant simulators for comprehensive testing. 3. Implementation of Group Relative Policy Optimization (GRPO) training with N-Stage Gated Rewards, addressing reward hacking through iterative improvements to syntax validators and LLM judges.