TechPrufer
Sign inGet started

Blogs›Shopify›AI Assistant (Sidekick) Capabilities and Training

AI Assistant (Sidekick) Capabilities and Training

AI Assistant (Sidekick) Capabilities and Training

3
posts
2026

Shopify has developed and enhanced its AI assistant, Sidekick, by implementing a continual learning loop. This loop compresses production experience into the continuous space of the model's weights, moving beyond static frontier models. The process involves defining quality through a rubric and annotators, calibrating a judge model using techniques like DSPy, and improving the baseline system through autoresearch by proposing and evaluating changes to prompts, tool definitions, and harness code. For parameter space optimization, a self-healing pipeline mines production traffic for hard negatives, which are then used to train a smaller model via supervised fine-tuning and reinforcement learning (GRPO) with the calibrated judge as the reward signal. Gist compression is used to reduce the system prompt's length and improve serving speed and cost. The GraphQL agent is a key example, demonstrating significant improvements in quality, cost reduction (96%), and latency.

2026

Sidekick's continual learning loop (2026) - Shopify

8/5/2026

This post details the implementation of a continual learning loop for Shopify's AI assistant, Sidekick. It introduces the 'flywheel' concept for compressing production experience into model weights. Key contributions include: defining quality via a rubric and annotator agreement, calibrating a judge model using DSPy and reflection-based optimizers (GEPA, ACE), improving the baseline system through autoresearch (propose-evaluate-keep/discard loop), and optimizing parameters by mining production traffic for hard negatives. The self-healing pipeline uses frontier reasoning models to critique failures, generate repair instructions, and create training trajectories for supervised fine-tuning (SFT) and GRPO. Gist compression is introduced to reduce prompt length and serving costs. The GraphQL agent is presented as a case study, showing quality improvements, cost reductions, and latency improvements.

Building an agentic harness that outlasts the model (2026) - Shopify

7/29/2026

This post details the development of an "agentic harness" named Dispatch for automated security vulnerability discovery and remediation. It outlines the multi-agent orchestration workflow, including stages for test bootstrap, architecture documentation, file cataloging, partitioning, hunting, verification, post-processing, reporting, and remediation. Key technical contributions include strategies for code partitioning to manage costs and improve accuracy, the development of a test oracle for web vulnerabilities, and lessons learned in producing consistent and reliable results from AI models. The post emphasizes the importance of the harness outlasting the models themselves.

Teaching Sidekick to say no: automated data curation with LLM judge consensus (2026) - Shopify

6/15/2026

Introduced an automated data curation pipeline using LLM judge consensus to train Sidekick's models to refuse impossible or ambiguous requests. This involved calibrating LLM judges with a small seed dataset, establishing a strict consensus mechanism for labeling, and defining a taxonomy of refusal categories. The pipeline creates a data flywheel where production traffic from improved models fuels subsequent training runs, leading to significant gains in skill evaluation scores.