ML Experimentation Platform

4/15/2026

This post details the development of an extension for the Autoresearch framework, enabling it to focus on improving specific metrics like build time. The extension allows for iterative hypothesis testing and metric improvement, demonstrating the potential of AI agents for tasks beyond traditional model training, such as optimizing CI/CD pipelines and other performance metrics. It introduces a loop that measures a baseline metric, forms a hypothesis, tests it, and iterates until a desired improvement is achieved or the process is stopped. The post also highlights the collaborative development with Tobi Lutke, leading to features like multi-metric support, consistent iteration execution, auto-commits, and the eventual open-sourcing of the `pi-autoresearch` project.

2/27/2026

This post details the engineering efforts behind SimGym, a simulated A/B testing platform for Shopify merchants. It describes the transition from API-based LLM calls to self-hosted open-source models (`gpt-oss-120b`) and the subsequent infrastructure and optimization challenges. Key contributions include: developing custom FlashInfer kernels for long-context speculative decoding on Blackwell GPUs, integrating async scheduling, guided decoding, and speculative decoding for agentic inference, and leveraging NVIDIA's EAGLE-3 speculative decoding head. The post also discusses the use of MIG partitioning for doubling serving instances, leading to significant latency reduction and throughput increase. Prompt optimizations for prefix caching and investigations into reducing reasoning effort are also detailed. The work involved deep collaboration with NVIDIA and CentML to tailor LLM serving for agentic workloads, moving beyond standard chatbot assumptions to handle correlated, long-context requests with JSON schema enforcement.

12/5/2025

This post introduces Tangle, an open-source ML experimentation platform developed at Shopify. It details the platform's motivation, addressing six common ML development failure modes. The core architecture is explained, focusing on platform agnosticism, robust content-based caching, visual pipeline editing, and a language-neutral design. Key concepts like Components, Tasks, Executions, and Graphs are defined. The post highlights Tangle's communication through file paths, its execution flow, and an optional type system. It also describes the visual editor for pipeline development and monitoring, and provides getting started instructions via HuggingFace.

10/18/2024

This post details the implementation of real-time ML inference streaming pipelines for generating embeddings for Shopify Storefront Search. It covers the architecture using Google Cloud Dataflow, the preprocessing steps for images, and the challenges encountered and solutions implemented, including managing data in memory by adjusting Dataflow worker threads and the trade-offs in managing the ML model in memory. It also discusses the importance of batching for GPU utilization and the implementation of batching via Apache Beam's ModelHandler.

3/10/2023

This post details the development and enhancement of Shopify's Merlin Online Inference capabilities, a component of the broader ML experimentation platform. It focuses on enabling the deployment and serving of machine learning models for real-time predictions at scale. Key contributions include the architecture for dedicated services on Kubernetes, the serving layer design (API for predictions, model loading), and the support for multiple serving layer types (MLServer, FastAPI) to cater to different user needs and customization levels. It also highlights the integration with model registries (Comet ML) and feature stores (Pano) for seamless end-to-end ML workflows.

9/15/2022

This post defines the role of a full-stack data scientist at Shopify, emphasizing end-to-end ownership of data science projects. It outlines the stages of a data science project (discovery, acquisition, data modeling) and the key skills required (business acumen, communication, programming, data analysis, data engineering, machine learning). It also introduces the concept of T-shaped development within the data science team.