AI Assistant (Sidekick) Capabilities and Training

4 posts · Mar 2020 – Aug 2026

What this trail covers

Shopify has developed and enhanced its AI assistant, Sidekick, by implementing a continual learning loop, optimizing prompt compression techniques, and introducing Just-in-Time (JIT) instructions to manage tool complexity. The system now uses Ground Truth Sets (GTX) and LLM-as-a-Judge calibrated against human judgment for robust evaluation, alongside LLM-powered user simulation for pre-production testing. Training has evolved to Group Relative Policy Optimization (GRPO) with N-Stage Gated Reward, addressing reward hacking through iterative improvements in syntax validators and LLM judges. Key takeaways emphasize modular architectures, robust evaluation frameworks, and vigilance against unexpected system failures.

All Shopify trails

Aug 2026Gisting: Compressing LLM Agent context to ↑ throughput and ↓ cost (2026) - ShopifyIntroduced 'gisting', a prompt compression technique that reduces system prompt token count by learning special tokens via knowledge distillation. This technique achieved a 4:1 reduction in system prompt tokens for the Sidekick GraphQL agent, leading to a 19% decrease in median time to first token (TTFT), a 38% decrease in median end-to-end request latency, and a 16% increase in throughput. The method involves training gist embeddings by minimizing KL divergence between teacher and student logits, and can be integrated into continual learning loops.
Aug 2026Sidekick's continual learning loop (2026) - ShopifyThis post details the implementation of a continual learning loop ('flywheel') for AI models, focusing on compressing production experience into model weights. Key technical contributions include: defining quality via rubrics and judges, calibrating judges with DSPy and reflection-based optimizers (GEPA, ACE), using autoresearch for prompt/tool definition optimization, and employing reinforcement learning (GRPO) with hard negative mining for parameter updates. It also introduces 'gist compression' for prompt optimization. The GraphQL agent is presented as a production example demonstrating significant cost and latency improvements.
Jul 2026Building an agentic harness that outlasts the model (2026) - ShopifyThis post details the development of an "agentic harness" named Dispatch for automated security vulnerability discovery and remediation. It outlines the multi-agent orchestration workflow, including stages for test bootstrap, architecture documentation, file cataloging, partitioning, hunting, verification, post-processing, reporting, and remediation. Key technical contributions include strategies for code partitioning to manage costs and improve accuracy, the development of a test oracle for web vulnerabilities, and lessons learned in producing consistent and reliable results from AI models. The post emphasizes the importance of the harness outlasting the models themselves.
Jun 2026Teaching Sidekick to say no: automated data curation with LLM judge consensus (2026) - ShopifyThis post details the development of an automated data curation pipeline for training LLMs to refuse impossible queries. It introduces the use of an LLM judge ensemble (Frontier LLM A, B, C, and D) calibrated with a small, high-quality annotated dataset to evaluate production queries. The system requires strict consensus among judges for label acceptance, prioritizing precision over recall. The pipeline categorizes queries into 'solvable with more context', 'missing capability', 'wrong skill', and 'ambiguous'. The outcome shows a significant improvement in segmentation skill evaluation scores from 0.619 to 0.798, with refusal accuracy at 86.3% and a false positive rate of 4.6%.
Mar 2026Building a Magic Mirror: AI retail experiences with Remix (2026) - ShopifyThis post details the development of a 'magic mirror' concept for retail experiences, leveraging AI for personalized customer interactions. It describes the architecture using Remix, including server-side image processing with LLMs (OpenAI SDK) and client-side state management. Key technical aspects include prompt engineering with product data and inventory, Server-Sent Events (SSE) for remote control, and client-side screen transitions. The implementation focuses on a specific use case for Rare Beauty, demonstrating how AI can be integrated into physical retail to provide personalized recommendations and interactive experiences.
Advertisement
Aug 2025Building production-ready agentic systems: Lessons from Shopify Sidekick (2025) - ShopifyThis post details the evolution of Sidekick's architecture with the introduction of Just-in-Time (JIT) instructions to manage tool complexity, moving away from a monolithic system prompt. It also elaborates on building robust LLM evaluation systems by using Ground Truth Sets (GTX) and LLM-as-a-Judge calibrated against human judgment, and employing LLM-powered user simulation for comprehensive testing. Furthermore, it discusses GRPO training with N-Stage Gated Rewards, highlighting the challenges of reward hacking and iterative improvements to address it.
Jul 2025Augmented commerce: Machine learning at Shopify (2025) - ShopifyThis post details the application of machine learning across various Shopify domains. It highlights the use of finetuned Qwen multimodal models for product classification and metadata enrichment, involving hundreds of millions of inferences daily. It also mentions fast risk models for fraud detection, inspired by Feature Space work. Sidekick, the merchant assistant, is powered by finetuned LLaMa models and large general models. Nomic embeddings are used for vector representations of billions of products, empowering search and recommendations. A tabular transformer-based model is used for forecasting merchant GMV for cash advance risk assessment. The HSTU architecture is being experimented with for customer and merchant behavior modeling. The search team is exploring query rewriting with small language models. The post also details investments in ML infrastructure, including partnerships with GCP and neo-cloud providers for GPUs, CentML for compute acceleration, and Toloka for data services. Early access to frontier models is utilized for annotations, evaluations, and direct deployments like Sidekick, accelerating prototyping and development.
Aug 2023Sidekick’s Improved Streaming Experience - ShopifyThis post details the technical implementation of an improved streaming experience for Sidekick. It introduces a buffering Markdown parser that uses a finite state machine (FSM) implemented in a Node.js Transform stream to handle ambiguous Markdown sequences, preventing rendering jank. It also describes an asynchronous content resolution and multiplexing strategy using Server-Sent Events, where tool content (referred to as 'cards') is resolved asynchronously and multiplexed into the main LLM response stream. This allows for immediate rendering of the initial LLM response while placeholders are populated with resolved card content, improving perceived performance and user experience.
Mar 2020The Evolution of Kit: Automating Marketing Using Machine Learning - ShopifyThis post details the evolution of Kit from a rule-based system to an AI-driven recommendation engine. It describes the machine learning architecture for predicting marketing budget and likelihood of sales, utilizing regression and classification models trained on historical campaign data and store attributes. The system was migrated to Google Cloud Platform for training and prediction, with a real-time prediction architecture introduced to provide more dynamic recommendations, such as acquiring new visitors for lookalike audiences.