TechPrufer
Get started
Shopify
Shopify logoShopify · Feature trail

ML Experimentation Platform

6 posts · Mar 2021 – Apr 2026

What this trail covers

Shopify has developed and enhanced its ML platform, Merlin, by adding online inference capabilities for real-time predictions, a model registry with experiment tracking (Comet ML), Merlin Pipelines for reproducible ML workflows, and the Pano Feature Store. This enables deploying and serving ML models at scale with low latency, supporting various ML libraries and offering state-of-the-art features like rolling deployments and autoscaling. The concept of full-stack data science is being embraced, and this post details the integration of real-time embeddings for improved search intent understanding using Dataflow streaming pipelines, addressing challenges in memory management, model loading, and batching for GPU-accelerated inference.

All Shopify trails
  1. Autoresearch isn’t just for training models (2026) - ShopifyThis post details the development of an extension for the Autoresearch framework, enabling it to focus on improving specific metrics like build time. The extension allows for iterative hypothesis testing and metric improvement, demonstrating the potential of AI agents for tasks beyond traditional model training, such as optimizing CI/CD pipelines and other performance metrics. It introduces a loop that measures a baseline metric, forms a hypothesis, tests it, and iterates until a desired improvement is achieved or the process is stopped. The post also highlights the collaborative development with Tobi Lutke, leading to features like multi-metric support, consistent iteration execution, auto-commits, and the eventual open-sourcing of the `pi-autoresearch` project.
  2. 2,000 robots walk into a shop: Simulated A/B testing (2026) - ShopifyThis post details the engineering efforts behind SimGym, a simulated A/B testing platform for Shopify merchants. It describes the transition from API-based LLM calls to self-hosted open-source models (`gpt-oss-120b`) and the subsequent infrastructure and optimization challenges. Key contributions include: developing custom FlashInfer kernels for long-context speculative decoding on Blackwell GPUs, integrating async scheduling, guided decoding, and speculative decoding for agentic inference, and leveraging NVIDIA's EAGLE-3 speculative decoding head. The post also discusses the use of MIG partitioning for doubling serving instances, leading to significant latency reduction and throughput increase. Prompt optimizations for prefix caching and investigations into reducing reasoning effort are also detailed. The work involved deep collaboration with NVIDIA and CentML to tailor LLM serving for agentic workloads, moving beyond standard chatbot assumptions to handle correlated, long-context requests with JSON schema enforcement.
  3. Tangle: An open-source ML experimentation platform built for scale (2025) - ShopifyShopify has open-sourced Tangle, an ML experimentation platform built to accelerate ML development by addressing six common failure modes: manual query tracking, unstructured notebooks, repeated data preparation, inability to recreate results, slow deployment, and lack of sharing. Tangle offers a platform-agnostic, language-neutral architecture with visual pipeline editing (DAGs), robust content-based caching (global artifact reuse), and a declarative component model defined in YAML. It executes arbitrary containerized CLI programs, allowing users to build pipelines with components written in any language. The platform's caching mechanism reuses identical computations based on output content hashes, significantly reducing execution time. Tangle also features an optional type system for metadata and tooling, and a visual editor for pipeline development and real-time monitoring.
  4. How Shopify improved consumer search intent with real-time ML - ShopifyThis post details the implementation of real-time ML embeddings for Shopify Storefront Search, leveraging Google Cloud Dataflow for streaming pipelines. It describes the architecture for processing image and text embeddings, including preprocessing, inference, and writing to a data warehouse and downstream systems. Key technical challenges and trade-offs are discussed, such as managing data in memory by adjusting Dataflow worker threads (reducing memory footprint by 2.6x and eliminating cost increase), managing models in memory (opting for default Dataflow configuration due to small model size and throughput considerations), and implementing batching for GPU utilization by overriding `batch_element_kwargs` in the ModelHandler. The post highlights the decision to use near real-time embeddings over batch solutions to meet merchant and customer expectations for instant updates, despite increased pipeline complexity.
  5. Unlocking Real-time Predictions with Shopify's Machine Learning Platform - ShopifyThis post details the development of Merlin Online Inference, a system for deploying and serving machine learning models for real-time predictions. It covers the architecture, including the serving layer (using MLServer or FastAPI) and deployment on Kubernetes, and highlights the requirements for low latency, robustness, and integration with the broader Merlin platform. The post also introduces the concept of online inference and its use cases at Shopify.
  7. What is a Full Stack Data Scientist? - ShopifyThis post defines and elaborates on the concept of 'full stack data science' at Shopify. It emphasizes data scientists owning projects end-to-end, covering discovery, analysis, acquisition, and data modeling. It highlights the necessary skills including business acumen, communication, programming (Python, SQL), data analysis, data engineering (building and deploying model pipelines), and machine learning. The post also discusses the benefits of this approach, such as choosing problem-solving methods and gaining broad tech stack exposure, and encourages T-shaped development. It provides tips for aspiring full stack data scientists, focusing on asking questions, gathering context, and honing technical skills.
  8. The Magic of Merlin: Shopify's New Machine Learning Platform - ShopifyThis post details the architecture and user journey of Merlin, Shopify's new machine learning platform. It highlights the use of an open-source stack, focusing on Ray for distributed computing and orchestration. Key components include Merlin Projects (Dockerized environments with specified dependencies) and Merlin Workspaces (short-lived Ray clusters deployed on Kubernetes). The platform enables scalability, fast iterations, and flexibility for data scientists and ML engineers, supporting training and batch inference. The user journey from project creation to prototyping in Jupyter Notebooks and productionization is outlined.
  9. Shopify's Playbook for Scaling Machine Learning - ShopifyThis post outlines Shopify's pragmatic playbook for scaling machine learning, detailing a step-by-step guide for identifying and solving problems with ML. It covers starting from zero by identifying worthy problems, ensuring data accessibility, understanding downstream dependencies, analyzing existing solutions, and optimizing for product outcomes. It also details the transition from 'zero to one' by creating well-defined pipelines for training, verification, and testing, and making deployment decisions based on volume and user commitment. Finally, it addresses scaling from 'one to one hundred' by building trust in models through input/output reconciliation, production backtesting, and monitoring.
  10. Shopify’s Unique Data Science Hierarchy Of Needs - ShopifyThis post introduces Shopify's unique Data Science Hierarchy of Needs, which emphasizes a tool-agnostic philosophy focused on delivering impact rather than prioritizing advanced tools. It breaks down data science work into five levels: Collect & Model, Describe, Predict/Infer, Prescribe, and Influence. The post uses the COVID-19 impact analysis as a case study to illustrate how this hierarchy guides data scientists in creating value for Shopify and its merchants, highlighting that impactful insights can come from simple solutions and do not necessarily require AI or ML.
  11. 5 Steps for Building Machine Learning Models for Business - ShopifyThis post outlines a five-step business-centric approach to developing and iterating on machine learning models: 1. Assess if it's the right time for ML (new product vs. improving existing, performance of non-ML methods). 2. Keep it simple (start with simple models, basic features, off-the-shelf solutions). 3. Measure before optimizing (select business-aligned metrics, tie metrics to business impact, explain tradeoffs). 4. Plan to iterate over models (retraining, architecture updates, phased rollouts). It provides practical tips and examples from Shopify's experience, such as using heuristics for forecasting and recommendations, starting with interpretable models like linear regression, and using confusion matrices to catch issues.
  12. 4 Tips for Shipping Data Products Fast - ShopifyThis post provides four tips for shipping data products quickly: 1. Utilize design sprints for focused problem-solving and timeboxing. 2. Don't skip prototyping, leveraging tools like spreadsheets for rapid iteration and learning. 3. Avoid machine learning in first iterations, opting for simpler proxy metrics or models to validate the problem and user need before investing in complex ML. 4. Talk to users frequently and ask specific questions to ensure the right product is being built.