TechPrufer
Get started
Shopify
Shopify logoShopify · Feature trail

Structured AI Workflows

6 posts · Apr 2022 – May 2026

What this trail covers

Shopify has developed and open-sourced Roast, a convention-oriented workflow orchestration framework designed for creating structured AI workflows. Roast interleaves non-deterministic AI behavior with deterministic code execution, using YAML configuration and markdown prompts to provide guardrails for AI agents. It supports various step types (directory-based, command execution, inline prompts, custom Ruby), built-in tools (ReadFile, WriteFile, UpdateFiles, Grep, SearchFile, Cmd, Bash), and a powerful CodingAgent integrated with Claude Code. Roast's session replay feature significantly speeds up workflow development. Internally, Roast has been used for test quality at scale, automated type safety with 'Boba', proactive site reliability monitoring, competitive intelligence aggregation, and a 'Chesterton's Fence' code research tool.

All Shopify trails
  1. Under the River (2026) - ShopifyThis post details the development of Aquifer, a platform for running AI agents, and River, an AI agent profile on top of Aquifer. Aquifer provides a durable substrate for AI agent sessions, harnesses, and sandboxes, with a Postgres-backed append-only event log for session durability and disposable session cells. River is presented as a multiplayer, Slack-based agent that interacts with the monorepo, code, data warehouse, and production signals. The post highlights the 'agent-friendly ≈ human-friendly' principle and the 'local agents have a ceiling' problem, leading to the development of open, collaborative agent interactions. The platform supports interactive, automation, and batch modes using a shared session model and sandbox plane.
  2. Flow generation through natural language: An agentic modeling approach (2026) - ShopifyThis post details the fine-tuning of a tool-calling agent to generate Shopify Flow workflows from natural language. It introduces a novel approach using Python as an intermediate representation (DSL) for LLMs, coupled with a round-trip transpiler to Shopify's native JSON DSL. Key contributions include strategies for building training datasets from existing production workflows, meticulous mirroring of production environments in training data (tool naming, ordering, response format, system prompts), and optimizing tool interfaces to minimize context by returning lightweight summaries first. The post also discusses the training infrastructure (H200 GPUs, FSDP, Tangle) and highlights the discrepancy between offline benchmark performance and real-world user activation rates, emphasizing the importance of production feedback for out-of-distribution requests.
  3. Autoresearch isn’t just for training models (2026) - ShopifyThis post introduces `pi-autoresearch`, an extension for the Pi agent harness that implements an iterative optimization loop inspired by Andrej Karpathy's Autoresearch. The extension focuses on improving a target metric (e.g., build time) by forming hypotheses, testing them, and retaining improvements that lead to faster execution or successful builds. It includes features like custom UI for iteration tracking, a focus on a specific metric, and a continuous looping mechanism. The post highlights the value of autonomous agents for tackling tedious but valuable optimization tasks that humans might deprioritize. The project was subsequently open-sourced and saw significant adoption and contributions.
  4. Building a Magic Mirror: AI retail experiences with Remix (2026) - ShopifyThis post details the implementation of a 'magic mirror' retail experience using Remix, integrating AI vision models for personalized customer interactions. The system leverages a full-stack Remix route for server-client logic, with a focus on prompt engineering for LLMs, including product catalog integration and output schema instructions. Server-Sent Events (SSE) are used for remote control and status updates. The client-side state machine manages user interactions, with specific timing for photo capture and response display. Prompt engineering lessons learned include hard constraints on length, curated compliment lists, exclusion lists for safety, and influencing recommendations with inventory data. Configuration is managed via a thin route file exporting a MirrorConfig.
  5. Building production-ready agentic systems: Lessons from Shopify Sidekick (2025) - ShopifyThis post details the development of Sidekick, an AI-powered assistant for merchants. Key contributions include: 1. Architecture evolution with Just-in-Time (JIT) instructions to manage tool complexity and improve maintainability and performance. 2. Development of robust LLM evaluation systems using Ground Truth Sets (GTX) and LLM-as-a-Judge calibrated against human judgment, along with LLM-powered merchant simulators for comprehensive testing. 3. Implementation of Group Relative Policy Optimization (GRPO) training with N-Stage Gated Rewards, addressing reward hacking through iterative improvements to syntax validators and LLM judges.
  7. Introducing Roast: Structured AI workflows made easy (2025) - ShopifyThis post introduces Roast, a new open-source framework for structured AI workflows. It details Roast's convention-over-configuration approach, various step types (directory-based, command execution, inline prompts, custom Ruby), and built-in tools. A key contribution is the CodingAgent, which integrates Claude Code for iterative AI tasks like complex code modifications and bug fixing. The post also highlights Roast's session replay feature for faster development and provides real-world examples of its impact at Shopify, including test grading, Sorbet type annotation ('Boba'), SRE monitoring, competitive intelligence, and code research.
  8. Sidekick’s Improved Streaming Experience - ShopifyThis post details improvements to the streaming experience for Sidekick, an AI assistant. It addresses Markdown rendering jank by implementing a buffering Markdown parser using a Node.js Transform stream and a finite state machine to handle ambiguous Markdown syntax. It also prevents response delays by multiplexing asynchronous content resolution (tool calls) into a single stream using Server-Sent Events. This approach allows for immediate streaming of LLM responses while asynchronously populating resolved tool content, improving user experience and handling multi-intent requests. Special 'card:' protocol links are used to signal asynchronous content resolution, which are then parsed by the Markdown parser.
  9. The Magic of Merlin: Shopify's New Machine Learning Platform - ShopifyThis post introduces Merlin, Shopify's new machine learning platform, built on an open-source stack including Ray. It details Merlin's architecture, focusing on Merlin Workspaces (Ray clusters on Kubernetes) for training and batch inference, and Merlin Projects for defining dependencies. The post explains how users can prototype in Jupyter Notebooks connected to these workspaces via the Ray Client API and productionize workflows using tools like Airflow. It highlights Ray's role in enabling distributed ML workflows in Python and its integrations like Ray Train, Ray Tune, and the Ray Kubernetes Operator.