Building production-ready agentic systems: Lessons from Shopify Sidekick (2025) - Shopify

This post details the evolution of Sidekick's architecture with the introduction of Just-in-Time (JIT) instructions to manage tool complexity, moving away from a monolithic system prompt. It also elaborates on building robust LLM evaluation systems by using Ground Truth Sets (GTX) and LLM-as-a-Judge calibrated against human judgment, and employing LLM-powered user simulation for comprehensive testing. Furthermore, it discusses GRPO training with N-Stage Gated Rewards, highlighting the challenges of reward hacking and iterative improvements to address it.