Structured AI Workflows

5/28/2026

This post introduces Aquifer, a platform for running AI agents, and River, an AI agent that coauthors pull requests. Aquifer provides a durable session, harness, sandbox, and gateway for AI agents, ensuring conversations survive process death. River operates in public Slack channels, leveraging the monorepo 'World' for knowledge and skills. The platform enables various agent profiles (e.g., PR review, research) and supports interactive, automation, and job modes.

4/22/2026

This post details the fine-tuning of a tool-calling agent to generate Shopify Flows from natural language for Sidekick, Shopify's AI commerce assistant. The approach involves creating a training dataset by reverse-engineering existing production workflows, generating natural language queries, and constructing tool trajectories. To improve model performance, workflows are represented in Python as an intermediate DSL, which is then transpiled to Shopify's native JSON DSL. The training data closely mirrors the production environment, including tool naming, ordering, response formats, and system prompts. The tool-calling stack is optimized by returning lightweight summaries first, followed by full details only when necessary. Training is accelerated using H200 GPUs and Shopify's Tangle ML experimentation platform. Initial production deployment revealed that offline benchmarks masked a significant gap in handling out-of-distribution requests, highlighting the importance of real-world user feedback and continuous learning.

8/26/2025

This post details the development of Sidekick, an AI-powered assistant for merchants. Key contributions include: 1. Architecture evolution with Just-in-Time (JIT) instructions to manage tool complexity and improve maintainability and performance. 2. Development of robust LLM evaluation systems using Ground Truth Sets (GTX) and LLM-as-a-Judge calibrated against human judgment, along with LLM-powered merchant simulators for comprehensive testing. 3. Implementation of Group Relative Policy Optimization (GRPO) training with N-Stage Gated Rewards, addressing reward hacking through iterative improvements to syntax validators and LLM judges.

6/18/2025

This post introduces Roast, a new open-source framework for structured AI workflows. Roast addresses the challenges of non-determinism and reliability in AI agent execution by providing a convention-oriented orchestration layer. It defines various step types (directory-based, command execution, inline prompts, custom Ruby), offers built-in tools for file manipulation and command execution, and integrates a powerful CodingAgent powered by Claude Code. Roast supports advanced control flow mechanisms like iteration and conditional execution, and features session replay for development. The framework is built on Raix and has demonstrated impact in areas like test quality, automated type safety, and proactive monitoring.

8/1/2023

This post details improvements to the streaming experience for Sidekick, an AI assistant. It addresses Markdown rendering jank by implementing a buffering Markdown parser using a Node.js Transform stream and a finite state machine to handle ambiguous Markdown syntax. It also prevents response delays by multiplexing asynchronous content resolution (tool calls) into a single stream using Server-Sent Events. This approach allows for immediate streaming of LLM responses while asynchronously populating resolved tool content, improving user experience and handling multi-intent requests. Special 'card:' protocol links are used to signal asynchronous content resolution, which are then parsed by the Markdown parser.

4/6/2022

This post introduces Merlin, Shopify's new machine learning platform, built on an open-source stack including Ray. It details Merlin's architecture, focusing on Merlin Workspaces (Ray clusters on Kubernetes) for training and batch inference, and Merlin Projects for defining dependencies. The post explains how users can prototype in Jupyter Notebooks connected to these workspaces via the Ray Client API and productionize workflows using tools like Airflow. It highlights Ray's role in enabling distributed ML workflows in Python and its integrations like Ray Train, Ray Tune, and the Ray Kubernetes Operator.