Shopify Feature Trails

Shopify has developed and enhanced its AI assistant, Sidekick, by implementing a continual learning loop. This loop compresses production experience into the continuous space of the model's weights, moving beyond static frontier models. The process involves defining quality through a rubric and annotators, calibrating a judge model using techniques like DSPy, and improving the baseline system through autoresearch by proposing and evaluating changes to prompts, tool definitions, and harness code. For parameter space optimization, a self-healing pipeline mines production traffic for hard negatives, which are then used to train a smaller model via supervised fine-tuning and reinforcement learning (GRPO) with the calibrated judge as the reward signal. Gist compression is used to reduce the system prompt's length and improve serving speed and cost. The GraphQL agent is a key example, demonstrating significant improvements in quality, cost reduction (96%), and latency.

Shopify is migrating its UI extension surface to remote-dom with Polaris web components, moving away from legacy React components and APIs. This involves upgrading extensions to newer API versions, adopting Preact for rendering, and rewriting codebases to TypeScript. The goal is to reduce bundle sizes, improve load performance, and ensure compatibility with future platform requirements. This initiative also includes developing custom, lightweight alternatives for large libraries like Liquid.js and dayjs to meet strict bundle size constraints.

This feature thread tracks the development of enhanced audio and visual capabilities for music merchants on Shopify. Initial efforts focused on enabling basic audio playback for digital music sales, addressing the gap where artists often default to other platforms. This has evolved to include rich, interactive experiences like custom visualizers, artist toolkits for managing releases, and integrated tour date information with geolocation, aiming to make Shopify a comprehensive platform for selling music. Recent developments include the creation of a first-class `Audio` media type, a standalone Shopify app for release management, a global audio singleton for persistent playback across storefront navigation, and advanced GLSL shaders for dynamic audio visualization. The platform now supports native audio file uploads, per-track buy buttons, and integrated tour date management with geolocation features.

Shopify has developed and enhanced its product categorization system by introducing an AI multi-agent system to continuously evolve the taxonomy. This system goes beyond static classification by actively improving taxonomy labels themselves, keeping the system agile and future-proof. It integrates actual merchant product data for real-world grounding, employs specialized agents for structural consistency and product-driven insights, synthesizes these insights intelligently, and detects complex e. This post details the use of multimodal LLMs to organize and augment product data across Shopify, including data curation, model fine-tuning, training, and the infrastructure for large-scale multimodal LLM inferences, impacting search, recommendations, and conversational commerce.

Shopify has developed and enhanced its hosting solution for Hydrogen custom storefronts, known as Oxygen. This involves leveraging Hydrogen, a React framework for custom storefronts, and integrating it with cutting-edge technologies like React Server Components and component-level data fetching. This post details best practices for using React Server Components (RSC) within Hydrogen applications, emphasizing a default-to-shared component strategy, judicious use of client components for interacti Quick is an internal platform for hosting static sites and providing backend services. It allows users to upload a folder of HTML and assets and get a secure URL. It leverages Google Cloud Storage for hosting, NGINX with gcsfuse for serving, and Identity-Aware Proxy (IAP) for authentication. Quick also offers backend services via APIs for database access (CloudSQL), AI calls (Shopify AI proxy), file uploads, data warehouse integration (BigQuery), and WebSockets. The platform emphasizes simplicity and a fixed set of capabilities to encourage creativity and maintainability. It has seen significant adoption for prototypes, dashboards, dev tools, and games, fostering an emergent ecosystem of shared JS libraries and embedded sites. The platform is maintained on a single VM and uses rate limiting and Go for improved performance and memory management.

Shopify has developed and open-sourced Roast, a convention-oriented workflow orchestration framework designed for creating structured AI workflows. Roast interleaves non-deterministic AI behavior with deterministic code execution, using YAML configuration and markdown prompts to provide guardrails for AI agents. It supports various step types (directory-based, command execution, inline prompts, custom Ruby), built-in tools (ReadFile, WriteFile, UpdateFiles, Grep, SearchFile, Cmd, Bash), and a po This post details the fine-tuning of a tool-calling agent to generate Shopify Flows from natural language for Sidekick, Shopify's AI commerce assistant. The approach involves creating a training dataset by reverse-engineering existing production workflows, generating natural language queries, and constructing tool trajectories. To improve model performance, workflows are represented in Python as an intermediate DSL, which is then transpiled to Shopify's native JSON DSL. The training data closely mirrors the production environment, including tool naming, ordering, response formats, and system prompts. The tool-calling stack is optimized by returning lightweight summaries first, followed by full details only when necessary. Training is accelerated using H200 GPUs and Shopify's Tangle ML experimentation platform. Initial production deployment revealed that offline benchmarks masked a significant gap in handling out-of-distribution requests, highlighting the importance of real-world user feedback and continuous learning.

Shopify has evolved its data store strategies by introducing composite primary keys in its main operational database to improve data access efficiency. This involves modifying database schemas to cluster data by shop ID, reducing buffer pool page reads and improving query performance. The implementation required adjustments to the Rails application layer, database migration infrastructure (LHM), live data migration system (Ghostferry), and data warehousing extraction systems to accommodate the new sharding strategy. This post details the shard balancing process using Ghostferry to migrate shops between MySQL shards with zero downtime, addressing issues of unbalanced database utilization and mitigating the risk of database failure.

Shopify has developed and open-sourced Tangle, an ML experimentation platform designed to accelerate ML development and data pipelines. Tangle provides a visual interface for building pipelines, robust content-based caching for artifact reuse, and a language-neutral architecture that integrates with existing codebases across any programming language and executes on any cloud provider. It addresses common ML development failures such as difficulty in reproducing experiments, repeated data preparation, and inefficient model training. The SimGym project further enhances this by developing specialized infrastructure and optimization techniques for agentic inference, including MXFP4 quantization, FP8 attention, custom FlashInfer kernels for long-context speculative decoding on Blackwell GPUs, and MIG partitioning for increased throughput and reduced latency. This involves deep collaboration with NVIDIA and CentML to tailor LLM serving for agentic workloads, moving beyond standard chatbot assumptions to handle correlated, long-context requests with JSON schema enforcement. The platform is now capable of simulating thousands of concurrent user sessions for A/B testing storefront changes, even for low-traffic merchants, significantly reducing testing time and cost.

Shopify is standardizing on Remix for Admin apps, enhancing the developer experience and performance. This includes modernizing the App Bridge to leverage web standards and custom elements, enabling seamless integration of web development practices and frameworks like Remix within the Shopify Admin. The migration of admin.shopify.com to Remix is a key step in this evolution.

Shopify has developed and enhanced its GraphQL execution engine by introducing GraphQL Cardinal, a breadth-first execution engine. This new engine resolves each field once across all objects, rather than once per object, significantly improving performance for high-cardinality list queries. This addresses the hidden costs of depth-first traversal, such as linear scale amplification and field-level overhead, leading to substantial reductions in execution time and memory usage. The migration of a massive production stack to this new model has been a key focus.

Shopify has developed and enhanced recommender systems to provide personalized insights, actions, tools, and resources to merchants. Initial efforts focused on leveraging collaborative filtering and user-item interactions to predict item relevance. This post details the development of a foundational generative recommender system that learns directly from raw event sequences, operates within real production latency constraints, and delivers measurable impact at scale. Key advancements include architectural decisions for handling sequential data, training strategies incorporating time encoding and improved negative sampling, and optimized CUDA kernels for efficient training and serving. The system demonstrated positive online A/B testing results, increasing shop orders, click-through rates, and conversion rates. Future work explores moving from product ID space to token space through semantic IDs to reduce reliance on large embedding tables and integrate richer signals like text queries and assistant interactions.

Shopify has integrated SkyPilot to manage multi-cloud GPU workloads, enabling engineers to define jobs in YAML and run them on available Kubernetes clusters across different cloud providers (Nebius, GCP). This includes a custom plugin for routing based on hardware requirements (H200s to Nebius, L4s/CPUs to GCP), fair-share scheduling via Kueue, and automatic injection of necessary configurations for specific hardware like InfiniBand. The system also supports development environments with dedicated labels and resource management, and automates storage provisioning and cleanup.

Shopify has developed the Universal Commerce Protocol (UCP) as an open standard for AI agents to transact with merchants. UCP is designed with a layered architecture, separating core transaction primitives (Shopping Service) from major functional areas (Capabilities like Checkout, Orders, Catalog) and domain-specific augmentations (Extensions). It emphasizes discovery and negotiation of capabilities between agents and merchants, allowing for flexible and extensible interactions. MCP UI extends UCP by enabling AI agents to return interactive UI components alongside data, facilitating richer user experiences in commerce by handling complex UI elements like product selectors, bundle builders, and subscription options. This is achieved through various delivery methods including inline HTML, remote resources, and Remote DOM, with an intent-based message system to maintain agent control over user interactions.

Shopify has unified its edge network infrastructure with the introduction of Möbius and Shopify Front End (SFE). Möbius handles TLS and TCP termination close to the client, providing improved performance and DDoS protection. SFE acts as a unified routing layer behind Möbius, dispatching traffic to various applications and services. This initiative standardizes traffic paths, simplifies observability, and automates edge configuration for application owners, enabling faster feature deployment and the development of interactive, real-time visualizations like the BFCM Live Globe pinball machine. The pinball machine itself is a three.js app built with react-three-fiber, utilizing server-sent events for real-time order streaming, custom shaders for dot matrix displays, and 2D physics (rapier2D) for performance. It also incorporates VR support via react-three/xr and displays real-time infrastructure stats.

This feature thread tracks the development and enhancement of webserver connection handling and request dropping mechanisms to improve application resilience under high load. Initial efforts focused on addressing the problem of web applications rendering requests for disconnected clients, particularly during traffic surges like flash sales. This post details a patch for the Unicorn webserver that checks client connection status before application rendering, effectively dropping requests from dis. This post details the creation of a gamified treadmill experience that rewards runners with in-store credit, integrated with a custom POS UI extension for seamless redemption. The system involves hardware (manual treadmill, sensors, receipt printer, displays, server) and software components (Python script for FTMS data, Remix app for game logic and state management, POS UI extension for discount application). Key technical challenges included real-time pace detection, normalization of irregular sensor data, and secure discount redemption via a custom POS extension.

Shopify has evolved its resiliency planning and operational readiness, particularly for high-traffic events like Black Friday Cyber Monday (BFCM). Initial efforts focused on load and stress testing to identify system limits and regressions. This has evolved to include capacity planning based on traffic forecasts and historical data, with detailed resource estimations for key areas and buffers for flexibility. Scalability testing, including "faux" BFCM events like "Oktoberfest scale-up", is now a core part of the readiness program. This post details the extensive preparation for BFCM 2025, including year-round capacity planning, multi-region strategy on Google Cloud, risk assessments (WCGW exercises), and extensive Game Days (chaos engineering at scale) focusing on critical user journeys. It also highlights the development of a Resiliency Matrix for documenting vulnerabilities and recovery procedures, and the use of load testing tools like Genghis and Toxiproxy to simulate traffic and identify breaking points. The post also addresses new analytics challenges for systems launched post-BFCM 2024, requiring new testing methodologies. Finally, it describes the scale testing program, which validates the entire platform working together at BFCM volumes, including architectural scale-up, load tests, failover tests, and Game Day simulations, all coordinated with cloud providers and external partners like YouTube.

Shopify has developed and enhanced its search algorithms and the methodologies used to evaluate them. Initial efforts focused on improving search relevance within the Help Center. This post details a three-step framework for evaluating new search algorithms, encompassing data collection (Kafka events, annotation), offline metric evaluation (MAP, NDCG), and online metric evaluation (user interaction, scroll depth, search abandonment). The framework aims to provide a robust system for deploying ML models like transformers and gradient boosting algorithms at native C++ speed using RankFlow and TurboDSL, enabling rapid iteration and millisecond latency at scale.

Shopify has successfully migrated all its mobile applications to React Native, achieving significant productivity gains, talent portability, and faster feature delivery. The company has learned that React Native apps can be performant, leveraging hot reloading and TypeScript for developer efficiency. Native expertise and code remain crucial for cutting-edge features and background tasks. Challenges in debugging and React Native updates are being addressed, with a focus on adopting the New Architecture. The reliance on third-party libraries is managed through automated updates and scanning. Shared foundations are being extracted into common libraries to increase leverage and consistency across apps. The future of React Native is seen as bright, with ongoing contributions to the framework and community.

Shopify has developed and enhanced its use of React Native Skia, a library that exposes Skia graphics engine functionality to React Native. This involves leveraging Skia's primitives for custom drawings, animations, and visual effects, integrating tightly with react-native-reanimated for performant animations and react-native-gesture-handler for interactive gestures. The library enables the creation of advanced UI elements and dynamic user experiences beyond the capabilities of standard React Native. Recent advancements include the integration of WebGPU as a backend for Skia, enabling unified backend architecture, web ecosystem symmetry, and general-purpose GPU computing. This allows for the seamless integration of libraries like Three.js and TensorFlow.js into React Native applications. Architectural improvements to React Native Skia's reconciler and scene graph model have led to significant performance gains and expanded platform support to macOS, tvOS, and Node.js. Experimental work with Skia Graphite introduces automatic threading and seamless 2D/3D composition.

Shopify has established a robust API versioning strategy to ensure stability and predictability for its partners. This involves releasing new API versions quarterly with date-based names (e.g. 2020-01). The platform team builds the infrastructure and tooling to enforce this strategy, which includes managing breaking and non-breaking changes. Breaking changes require thorough impact analysis and stakeholder agreement, while non-breaking changes are tested for forward compatibility. Tools like `A

Shopify has addressed critical adoption issues with JavaScript import maps and module loading. Previously, modules could not load before the import map, and only one import map was allowed per document, causing conflicts and fragility. Shopify collaborated on a web platform-level solution that introduces a reconciliation mechanism. This allows for multiple import maps, modules to load before import maps, and resolves conflicts by merging new maps into a global representation while respecting alignment. This post details the integration of Subresource Integrity (SRI) for imported JavaScript modules by extending import maps, enabling dynamic imports with integrity hashes in Chromium and WebKit browsers.

Shopify has developed and enhanced its mobile applications using a hybrid approach, combining native iOS and Android development with web technologies through the Mobile Bridge framework. This framework reinvents WebViews to improve performance, appearance, and integration, making web content blend seamlessly with native screens. Key improvements include preloading and caching WebViews for faster load times (reducing P75 from 6s to 1.4s), adapting Polaris components and hiding unnecessary UI elements. This post details the extension of this approach to treat mobile apps as services, enforcing best practices and tooling usage through ServicesDB. ServicesDB tracks all production services, defines ownership expectations, and provides tools for owners to improve infrastructure quality. For mobile projects, it codifies best practices like service ownership, contact information, CI/deployment configuration, versioning (including dependency vulnerability checks), and monitoring. This contract ensures teams are highly aligned but loosely coupled, allowing the Mobile Tooling team to enforce tool usage and identify misconfigured projects, notifying owners and helping them fix issues. Future plans include defining service level objectives for mobile apps and developing tools for easier project creation and reduced build times.

Shopify has developed and enhanced its capabilities for creating dynamic and visually engaging real-time data visualizations for marketing events like BFCM. This includes optimizing 3D rendering techniques for large numbers of dynamic elements like order arcs and fireworks, implementing instancing for GPU-based rendering, and developing sophisticated camera animation and interactive features. The focus is on achieving high performance and visually rich experiences for showcasing global commerce. This post details the development of the 2020 BFCM Globe and Live View, focusing on realism and performance. Key technical contributions include the use of a layered architecture with three.js, physically based rendering with 32bit EXR Environment Maps for realistic lighting, custom materials extending MeshStandardMaterial with custom shaders for effects like bubble transparency, and a particle-based system using BufferGeometry and ShaderMaterial to render up to 150,000 orders animating on the globe at 60fps. Performance optimizations include using BufferAttribute's `_updateRange_` to efficiently update particle attributes.

Shopify has developed and enhanced its mobile release process with the introduction of Tophat, a desktop macOS application that streamlines the tophatting of mobile changes. Tophat provides a seamless and interactive experience for testing pull requests on real devices or simulators without requiring local compilation. It integrates with CI pipelines to provide links to build artifacts, and includes a 'Quick Launch' feature for developers to easily grab the latest native builds. For iOS, Tophat leverages the `ios-deploy` utility and Fastlane's `match` for signing, addressing the lack of direct Apple APIs for remote device installation and debugging. This tool aims to improve developer experience, increase testing frequency, and serve as a foundation for future mobile development tooling.

Shopify has developed and implemented Packwerk, a static analysis tool to enforce modularity and code boundaries within large-scale Rails applications. Packwerk analyzes constant references to help decouple code and organize it into well-defined packages, enforcing non-circular dependency relationships. This post details a further evolution of this approach by focusing on behavior over state to create more maintainable codebases. By wrapping domain objects and exposing behavior-oriented methods, the system reduces direct state inspection and conditional logic, leading to more robust and extensible code. This also includes strategies for translating these behavioral abstractions into a format suitable for data teams, ensuring that reporting remains insulated from internal code changes.

Shopify is exploring and implementing WebAssembly (Wasm) as a secure, performant, and flexible runtime for executing untrusted partner code on its own infrastructure, moving beyond browser-based use cases. This involves leveraging the Lucet Wasm runtime and compiling code from languages like AssemblyScript to enable extensibility for partners. This post details four approaches to debugging server-side WebAssembly, including compiling to a native architecture, using LLDB with Wasmtime, employing WebAssembly-specific debuggers, and leveraging browser developer tools. The post highlights the trade-offs of each approach, with native debugging currently offering the most refined experience for variable rendering and interpreted languages, while other methods are explored for WebAssembly-specific behaviors.

ShopifyQL is an accessible, commerce-focused querying language used on both the client and server, defined by an ANTLR grammar and used to generate code for multiple targets (Go, Typescript). It is used to power ShopifyQL Notebooks, providing merchants with a guided code editing experience. The language features are encapsulated into a TypeScript language server conforming to the Language Server Protocol (LSP). This enables interoperability with code editors like CodeMirror, which uses its own Lezer parser engine. A custom adapter was developed to translate between the LSP-compliant language server and Lezer, handling complex token offset calculations from ANTLR to a CodeMirror-compatible format. This integration allows for features like syntax highlighting, code completion, linting, and tooltips within the ShopifyQL code editor, enhancing the merchant experience for data analysis.

Shopify engineers worked on many cool and impactful projects in 2021. Exhibit A: YJIT. Created by a small team led by Staff Engineer Maxime Chevalier-Boisvert, YJIT is a new JIT (just-in-time) compiler built inside CRuby that has now been merged upstream and is part of the Ruby 3.1.0 release. In this post, Maxime writes about the importance of the project to Shopify and Ruby developers worldwide. This post details the process of contributing support for new WebAssembly instructions to Winch, Was. This year, YJIT gained the ability to generate machine code for ARM platforms, enabling its use on ARM servers and Apple Silicon development machines. The YJIT compiler was also rewritten in Rust for improved maintainability and performance.

Shopify is expanding its Shopify Functions capabilities to support custom routing rules for order fulfillment. This involves leveraging Rust for default routing rules and enabling merchants to create their own rules using JavaScript compiled to WebAssembly (Wasm). The system utilizes a pipeline pattern for processing and a reducer for tie-breaking among rules. Future work will focus on opening up order routing for third-party developer-created rules.

Shopify has evolved its Ruby execution models and threading strategies to optimize garbage collection performance within its monolithic application. Initial efforts focused on reducing the latency impact of the Ruby garbage collector (GC) by analyzing and tuning its behavior. Key developments include addressing major GC triggers like `:oldmalloc` and `:nofree` by adjusting GC limits and pre-allocating heap slots. The introduction of Ruby 3.2's variable width allocation necessitated a finely-grained approach to pre-allocating heap slots across different size pools, which was initially achieved through a 'hack' and later integrated as a proper Ruby feature. These optimizations have led to significant reductions in GC-related latency, particularly for p99 and tail-end requests.

Shopify's Retail Point of Sale app migrated from Vanilla Redux to Redux Toolkit (RTK) to reduce boilerplate, improve code readability, and streamline state management. The migration adopted a balanced approach, reusing existing logic while refactoring reducers and thunks to leverage RTK's features like immutable syntax and generated action creators. A hybrid approach was used for asynchronous logic due to GraphQL limitations with RTK Query, utilizing createAsyncThunk with ThunkAPI. Key learnings included explicit typing of hooks and the use of `unwrap` for awaited dispatch calls. The migration resulted in significant code reduction and improved developer experience.

Shopify developed a centralized authentication service called Identity, built on the OpenID Connect (OIDC) specification, to manage user identities and enable single sign-on (SSO) across multiple stores. This involved synchronizing user data, migrating legacy accounts to Identity accounts, prompting users to combine accounts, and enforcing two-factor authentication (2FA). The system handles account merging, password updates, and prevents the creation of new legacy accounts, significantly reducing security risks. The Shop team has now integrated Passkeys into Shop's authentication flows on the web and in the native app to replace email and SMS verification, leveraging the webauthn-ruby gem for registration and login flows, and exploring Conditional UI for a more seamless user experience.

Shopify has evolved its capabilities for handling international address formatting. Initial efforts focused on addressing the complexities of varying address structures, field ordering, wording, and translations across different countries. The solution involved creating a database of country-specific address formats, implementing dynamic form representation based on country selection, and developing a display representation that accounts for locale. This post details the challenges and best practices for front-end developers in internationalization (i18n), specifically addressing issues with hardcoded English grammar, interpolation, pluralization rules (including CLDR categories like zero, one, two, few, many, other), gendered nouns affecting articles and adjectives, and complex declensions. It emphasizes translating entire strings rather than using placeholders to accommodate linguistic variations.

Shopify has developed and enhanced its Ruby LSP (Language Server Protocol) to improve the developer experience for Ruby and Rails developers. This involves providing accurate, performant, and stable language features like Go to Definition and Auto Formatting without requiring type annotations. The Ruby LSP leverages the Syntax Tree gem for parsing Ruby code into Abstract Syntax Trees (ASTs) and employs the visitor pattern for analyzing these ASTs to implement features such as Folding Range. Document synchronization ensures the server maintains an up-to-date representation of open files. Future plans include adding framework-specific features for Rails development.

This feature thread tracks the development and enhancement of how equality is implemented and understood within Ruby at Shopify. Initial efforts focused on understanding the nuances of Ruby's equality operators (#==, #eql?, #equal?, #===) and how to correctly implement them for custom objects. This includes differentiating between entities and value objects, and handling type coercion. This post details the dangers of monkey patching, advocating for upstream contributions and avoiding it where possible, and also discusses refinements as a slower alternative. Future development may explore advanced equality concepts, performance implications, and best practices for ensuring safe and maintainable code.

This feature thread tracks the development and enhancement of strategies for managing and reducing technical debt within Shopify's engineering organization. Initial efforts focused on categorizing debt into Yearly, Monthly, Weekly, and Daily types. The '25 Percent Rule' was introduced as a framework to allocate engineering time: 10% for Daily Debt (improving code encountered during regular work), 10% for Weekly Debt (addressing smaller issues identified as separate tasks), and 5% for Monthly/Yearly Debt (planning and prioritization). The emphasis is on fostering a culture that values code improvement and refactoring, encouraging engineers to proactively address debt as part of their regular workflow rather than as an afterthought.

Shopify has evolved its systems and methodologies for calculating core e-commerce metrics. Initial efforts focused on defining and implementing foundational metrics like customer count, churn rate, and lifetime value. This post details the refinement of churn rate calculation, moving from simplistic and problematic accounting-based definitions to a more robust, timely, and current approach that uses daily customer counts and cancellations, weighted by probability of being active. The BG/NBD model is used to statistically determine customer behaviors, enabling targeted marketing campaigns, order fulfillment prioritization, and customer support. The analysis is sent to reporting infrastructure and trained on over 500K merchants using Apache Spark and a Python UDF with the lifetimes library.

This feature thread tracks the development and enhancement of Shopify's tax engine. Initial efforts focused on addressing the complexity and distributed nature of tax calculations within the codebase. The primary strategy involved componentizing the tax logic into a distinct, well-defined module with a clear interface and schema for requests and responses. This involved extensive code analysis, refactoring, and the establishment of explicit architectural boundaries. The release process was carefully managed. This post details the development of the Tax Insights feature, which involved significant data modeling work, including modifying existing data models and creating four new ones. It explains the architecture for processing and storing tax-related data, utilizing Google Cloud Storage, Spark jobs, and Google Cloud Bigtable. The post also covers the process of understanding complex tax laws, prototyping data models with SQL, implementing logic, productionizing jobs with Spark, and publishing insights to merchants.

This feature thread tracks the development of systems to optimize the testing process for large codebases. Initial efforts focused on reducing the time and resources required for continuous integration by intelligently selecting which tests to run. This involves implementing dynamic analysis to map code changes to relevant tests, thereby reducing the overall test suite execution time and improving developer feedback loops. The thread also explores strategies for handling intermittently failing tests and the implementation of a test budget to constrain test execution time, prioritizing tests based on historical failure rates and other heuristics to maximize early failure detection within a fixed time window.

This feature thread tracks the evolution of the Shopify CLI, from its Ruby origins to a modern Node.js-based application. Initial efforts focused on unifying theme and app development tools into a single Ruby CLI. The rewrite to Node.js was driven by a desire for fewer system requirements, improved developer experience, and better alignment with web technologies. Key developments include establishing a consistent foundation with shared NPM packages, ensuring cross-OS compatibility through dedicated modules and comprehensive testing, migrating to a monorepo for atomic contributions, and embracing functional programming principles to enhance code reasoning and testability. Future work aims to fully remove the Ruby dependency for theme developers.

This feature thread tracks the development and enhancement of tools and strategies for testing and debugging HTTP interactions. Initial efforts focused on addressing the challenges of managing complex Webmock stubs and testing intricate remote API operations. The introduction of the VCR gem provides a mechanism to record and playback HTTP interactions, offering a more accurate representation of API behavior compared to manual mocking. This enables more reliable and easier-to-debug tests, especially for scenarios with unpredictable API sequences, multiple API calls, or unreliable external services. The thread also explores best practices for using VCR, including when to use it versus Webmock, and how to manage sensitive data within recorded interactions.

This feature thread tracks the development and enhancement of Shopify's system for managing and automatically rotating GitHub Personal Access Tokens (PATs). Initial efforts focused on addressing the security and operational challenges associated with long-lived, manually managed PATs, such as vulnerability to developer churn, attribution difficulties, and lack of enforced rotation policies. The solution involves building a system that leverages GitHub Actions and GitHub Apps to transparently rotate tokens with short lifespans, ensuring improved security, audibility, and resilience against developer churn. The system aims to replicate the usage patterns of PATs while automating their rotation and management.

Shopify has adopted Rust as its official systems programming language, alongside its continued commitment to Ruby for business logic. This decision is driven by Rust's safety, performance, productivity, and strong community. The company is investing in supporting Rust in its development and deployment processes, and aims to contribute to the Rust community and ecosystem. This standardization covers high-performance servers, Ruby extensions, and WebAssembly compilation.

Shopify has evolved its data pipeline platform, initially built with PySpark (Starscream), to a more efficient SQL-based workflow using dbt (Seamster) for reporting and common data tasks. This involved creating a new system that modularizes data model development, implements unit testing with mock data and Great Expectations, and provides schema and DAG validation through temporary BigQuery datasets. The goal is to reduce development time and improve the consistency of business metrics by leverag

This feature thread tracks the development and enhancement of Shopify's capabilities for extracting and analyzing metrics from Datadog. Initial efforts focused on addressing the limitations of Datadog's built-in visualization and analysis tools, as well as the granularity issues with long-term metric data. This post details a step-by-step guide for extracting Datadog metrics using its REST API and performing in-depth analysis locally in Jupyter notebooks using Python. The process involves obtaining API credentials, defining metric queries, handling API rate limits, and processing the extracted data into a more granular format for advanced visualization and statistical modeling.

Shopify has successfully migrated its caching mechanisms away from Ruby's Marshal serialization to MessagePack. This involved developing custom extension types for common Ruby objects like Date, Symbol, Time, DateTime, BigDecimal, and ActiveRecord::Base. The migration was phased, running both formats in parallel for six months, logging unencodable objects to identify necessary extension types and potential cache misuse. The new system uses a version byte prefix to distinguish between MessagePack and Marshal payloads during the transition. The primary benefit is improved safety and reliability by mitigating risks associated with code changes breaking cache payloads.

This feature thread tracks the development and enhancement of Shopify's use and understanding of the Apollo GraphQL client cache. Initial efforts focused on migrating from Apollo Client 2 to 3, which revealed past bugs and misunderstandings related to the cache's behavior. This post details the internal workings of the InMemoryCache, including data fetching policies, normalization strategies (object breakdown, unique cache identifiers using `__typename` and `id` or custom `keyFields`, and flattened data structures), and the mechanisms for updating and merging data. It also explores scenarios where automatic UI updates fail and the need for custom update functions or explicit return data. Future development will likely involve further optimization, advanced cache strategies, and addressing complex data scenarios.

Shopify has evolved its engineering organization towards a Production Engineering model, inspired by industry practices like SRE and Facebook's Production Engineering. The goal is to unify development and operations responsibilities, reduce manual toil through automation and tooling, and improve self-service capabilities for feature teams. This involves building common infrastructure, shared operational responsibility, and world-class developer tools. The model aims to balance management support, system and domain expertise, and high-level roadmaps through strategies like temporary team assignments, whole-team contributions to other teams' goals, and the formation of tiger teams for specific projects. The model aims to balance management support, system and domain expertise, and high-level roadmaps through strategies like temporary team assignments, whole-team contributions to other teams' goals, and the formation of tiger teams for specific projects.

Shopify has developed Hubble, an internal tool designed to encapsulate and extend various security tools, providing a unified interface for IT staff to manage inventory, device management, and security. It standardizes data from multiple systems, offers granular access control, and centralizes auditing for incident response. For end-users, Hubble provides device health and compliance information with actionable steps for remediation. It also serves as a single source of truth for security data, enabling other teams to build automations and security processes more easily. The development of Hubble highlights the value of cross-functional collaboration and investment in proactive security solutions.

This feature thread tracks the development and enhancement of Shopify's Ruby Dependency Security initiatives, focusing on analyzing the relationships between Ruby packages and their dependencies to identify potential security vulnerabilities and improve the overall security posture of the Ruby ecosystem. This includes efforts related to Multi-Factor Authentication (MFA) rollout on RubyGems, dependency analysis to uncover indirect vulnerabilities, and the development of tools and visualizations to understand these relationships at scale.

Shopify has evolved its Git workflows and best practices, with this post detailing the importance of Pull Request (PR) discipline. It emphasizes structuring PRs effectively by focusing on PR titles, summaries, and commits to enhance code review, implementation, and maintenance. This builds upon previous efforts to improve the safety and speed of merging code into the main branch, such as the upgrade of the Merge Queue system.

Shopify has evolved its engineering practices by emphasizing techniques that minimize code reading and maximize reliance on automated checks like the type system and linters. This includes strategies for finding entry points, making initial breaking changes, climbing the callstack with build errors, and updating recommended code during the downswing. The focus is on codifying standards in linters, leveraging the type system for stricter argument validation and exhaustive switches, and optimizing unit tests for clarity and maintainability. The ultimate goal is to improve developer efficiency and reduce the cognitive load associated with code maintenance.

Shopify has engineered a new compensation system called Flex Comp, which allows employees to customize their total rewards allocation between base salary, RSUs, and stock options. This involved building a new user interface, establishing robust API contracts, and implementing advanced privacy controls for sensitive compensation data. Key technical challenges included secure eventing for analytics, managing large RSA key pairs for encryption, and designing an extensible front-end architecture to handle regional and employment-based regulatory differences. The system prioritizes data integrity, user accessibility, and parallel development.

Shopify has developed and enhanced its Android Point of Sale (POS) application's receipt printing capabilities. This involved implementing a multi-step printing pipeline to handle diverse receipt content combinations, including data aggregation, content generation, and dynamic layout selection. The system aims to provide merchants with flexible and comprehensive receipt printing options.

Shopify has developed and enhanced its UI performance across its admin interfaces. Initial efforts focused on understanding user flows and identifying performance bottlenecks through profiling. Key strategies include prioritizing component and data loading, memoizing expensive tasks, deferring non-essential components, prefetching resources based on user hints, and improving loading states with skeletons and stable content presentation. This post details the development of the Polaris Viz React component for UI performance optimization, and the implementation of a short-lived cache and decoder strategy for iOS widgets to improve data freshness and reduce network calls, as well as the use of SwiftUI for building iOS widgets to leverage platform optimizations.

Shopify has developed and open-sourced the react-form library within its Quilt repository. This library provides custom React hooks, primarily `useForm` and `useField`, to manage form state and validation efficiently and safely. The library aims to simplify form development by offering smart defaults and reducing the need for manual state management with `useState`. It supports features like form submission handling, resetting form fields, conditionally enabling submit buttons based on form 'dirtiness', and implementing custom validation rules, such as ensuring required fields. The library is designed to be integrated with Shopify's Polaris Design System for UI consistency.

This feature thread tracks the development and enhancement of Shopify's event processing capabilities, focusing on scaling server-side data sharing. Initial efforts involved addressing consumption lag in Kafka by processing events in batches. This post details the implementation of the Go worker pool pattern to control the concurrency of goroutines, significantly increasing event processing performance from 7.75 thousand events per second per pod to over 21 thousand events per second per pod, and ultimately handling up to 46 thousand events per second during peak loads like BFCM.

This feature thread tracks the development and enhancement of strategies for effective communication during code reviews, particularly in cross-cultural contexts. Initial efforts focused on understanding how different cultural backgrounds influence feedback delivery and reception. This post highlights the challenges posed by varying politeness norms, directness in feedback, and the perception of confrontation. It proposes a shift towards more explicit, curious, and context-driven communication, and emphasizes practical techniques like keeping pull requests small, using draft PRs, focusing on code not people, picking the right reviewers, and providing a clear map in the PR description to improve the quality and efficiency of code reviews.

This feature thread tracks the development and enhancement of Shopify's capabilities for predicting and managing delivery dates for merchants and buyers. Initial efforts focused on building a platform for third-party logistics (3PL) providers to upload their pre-computed delivery dates. This involved creating new GraphQL APIs for 3PLs to integrate with the core Shopify platform and displaying these dates on product detail pages and at checkout. The Shopify Fulfillment Network (SFN) team developed a system to calculate and upload SFN-specific delivery dates, considering factors like inventory proximity, product characteristics (heavy, bulky, dangerous goods), fulfillment center processing times (including operating days, holidays, and capacity), and shipping carrier transit times (including pickup schedules, capacity limits, and time in transit). The system generates multiple schedules based on fulfillment center availability and product characteristics, mapping SKUs to these schedules for efficient real-time lookups. The displayed delivery date is then used during order fulfillment to route orders to appropriate warehouses and shipping carriers.

This feature thread tracks the development and enhancement of understanding and implementing stable values in React, particularly within the context of functional components and hooks. Initial efforts focused on identifying common pitfalls related to unstable values in dependency arrays of hooks like `useEffect`, `useCallback`, and `useMemo`, as well as the performance implications of unstable inline functions used as components. The thread also delves into the nuances of memoization in React, clarifying the distinction between true memoization and caching mechanisms. The goal is to improve developer understanding and prevent performance regressions caused by incorrect handling of value stability.

This feature thread tracks the development and enhancement of Shopify's AI-powered message classification model for Shopify Inbox. Initial efforts focused on building a prototype using a data-centric approach and a pre-trained Universal Sentence Encoder model to understand buyer intent. This led to the identification of key areas like order status requests. Subsequent development involved creating a more robust taxonomy with over 40 topics, refining annotation processes, and fine-tuning a DistilBERT model to achieve higher accuracy and coverage. The focus remains on improving the model's ability to accurately categorize merchant-buyer communications to enhance the merchant experience and inform product development.

This feature thread tracks the development and enhancement of Shopify's approach to containerization, focusing on the use of Docker and CoreOS for building an internal cloud. Initial efforts involved containerizing existing applications, managing secrets, routing, monitoring, and bulletproofing the system. Subsequent development has focused on container resource management, addressing issues with systemd processes within containers not adhering to Kubernetes resource limits due to cgroup driver mismatches. The goal is to ensure efficient and predictable resource utilization for containerized applications.

Shopify has evolved its Bug bounty program and its role in managing security vulnerabilities. Initial efforts focused on establishing a security response program and evolving it into a bug bounty with HackerOne. The program emphasizes high minimum payouts, transparency in disclosure, and educational opportunities for researchers. It has led to numerous fixes and significant bounty payouts. This post details the program reaching the $1M USD milestone in 2021 and doubling the maximum bounty to $100K for CVSS 10.0 issues in 2022, moving Shopify Plus, Shop, and Shop Pay into the highest severity bracket as core assets. The program also saw an increase in valid reports and a decrease in non-applicable issues, indicating improved researcher guidance. Efforts to support open-source security through sponsorships and bonuses for accepted patches were also highlighted. Response times were tracked, with a decrease in time to bounty in 2021. The program continues to focus on improving response times through automation.

This feature thread tracks the development and enhancement of Shopify's cloud-based development environments. Initial efforts focused on addressing the limitations of local development tools like 'dev' and 'railgun' which struggled to keep up with the increasing complexity of projects and the resource demands of modern development workflows. Experiments with local Kubernetes clusters and Google Compute Engine (GCE) VMs were conducted. The GCE VM experiment showed promise, leading to a broader exploration of cloud-native solutions. The team is now focused on building a dynamic, scalable, and automated development environment that leverages cloud infrastructure, moving away from managing VMs and towards a more elastic relationship between compute power and storage, ultimately aiming to provide developers with cloud-native development environments that scale as well as production applications.

This feature thread tracks the development and enhancement of Shopify Flow's orchestration capabilities, focusing on the introduction and evolution of the Maestro language and its execution engine. Initial efforts involved decoupling the Flow editor and engine by designing and implementing Maestro, a domain-specific orchestration language tailored for efficient, fault-tolerant workflow execution. A new, horizontally scalable engine was built to execute Maestro orchestrations. A translation layer was created to enable existing Flow workflows to run on the new engine. Subsequently, a new visual experience and editor language were developed, which also translate to Maestro orchestrations. All Flow workflows have since been migrated to use the new editor and engine, ensuring a more flexible, expressive, and independently evolvable system for workflow automation across Shopify.

This feature thread tracks the development and enhancement of capabilities for interacting with and manipulating data within Shopify's systems via its API. Initial efforts focused on data retrieval and creation. This post details the process of modifying existing data entities, such as customer addresses and product tags, using HTTP PUT requests with specific URLs and formatted request bodies (XML or JSON). It establishes the general formula for updating data through the Shopify API. This installment details the creation of the Marketplace Kit v2, decoupling it from the Shopify CLI and separating it into distinct admin and buyer-facing applications. It highlights the use of Node.js, Express, React, Next.js, GraphQL, and App Bridge for authentication and data fetching, including custom client wrappers for both REST and GraphQL interactions with Shopify's APIs.

Shopify has adopted Terraform for managing infrastructure as code, enabling engineering best practices like testing, revision history, and automated rollbacks for third-party services. This initiative began with the Conversations team implementing a custom Terraform provider for Twilio TaskRouter, allowing business and support teams to directly manage routing configurations. The broader adoption of Terraform aims to empower non-developers with infrastructure management, free up developer time, and increase visibility into infrastructure changes.

This feature thread tracks the development and enhancement of Shopify's v8go library, focusing on integrating the V8 CPU Profiler. Initial efforts involved exposing core V8 functionality through Cgo bindings. This post details the addition of the V8 CPU Profiler, exploring the trade-offs between lazy and eager loading strategies for profile data to optimize performance and developer experience. The focus is on providing Go users with an idiomatic and performant way to measure JavaScript execution performance within V8 isolates.

This feature thread tracks the development and enhancement of strategies for managing polymorphic associations in Rails, focusing on reducing coupling between the codebase and the database. Initial efforts addressed the challenges of namespace changes breaking polymorphic associations. This post details a solution to stop storing class names as polymorphic types, instead using arbitrary strings. This allows for easier refactoring of class names and namespaces without database migrations. The solution involves overriding the `polymorphic_name` method and adjusting association scopes to handle both old and new storage formats during the transition, followed by a cleanup task to standardize on arbitrary strings.

Shopify has developed and enhanced its tools and strategies for managing code deprecations. Initial efforts focused on preventing developers from accidentally using deprecated code and addressing the challenges of complex dependency upgrades. The Deprecation Toolkit was introduced as an open-source Ruby gem that uses a whitelist approach to track and manage deprecations. It allows for gradual fixing of existing deprecations while enforcing stricter rules for new ones. This post details the iterative development of tooling to address the dependency confusion vulnerability in over 600 Ruby applications, including a Bundler plugin to automate lockfile migrations and a CLI tool built on the CI system for large-scale migration. The team also collaborated with the Bundler and Heroku Buildpack teams to address broader community issues and prevent future vulnerabilities by ensuring correct Bundler versions are used in local development and CI.

Shopify has evolved its DNS traffic management strategies to improve service resiliency and availability. This has evolved into a fully automated, self-served system utilizing two DNS providers with layered traffic management capabilities, including geographical fencing, endpoint status health checking, priority-based failover, and weighted selection. The system aims to prevent outages caused by issues like expired root certificates in embedded client stores. The incident described highlights how a stale `cacert.pem` in the `httpclient` library, used by various internal services, led to certificate validation errors after the expiration of Let's Encrypt's old root certificate. The emergency fix involved forking the `httpclient` repository, updating `cacert.pem` with a more recent bundle, and deploying this patched version. A long-term solution involves automating updates to this fork and potentially leveraging system root certificate stores.

This feature thread tracks the development of capabilities for integrating Shopify with external e-commerce platforms and marketplaces. Initial efforts focused on bridging differences in domain models, communication protocols, and messaging styles to enable seamless data synchronization and order management. Key challenges include handling asynchronous vs. synchronous operations, message ordering, and differing data granularity for entities like orders and refunds. The goal is to provide merchants with a unified view of their sales channels. This post introduces a critique of RESTful design principles when applied to transactional processes, advocating for explicit URIs for state transitions (e.g., POST /orders/42/pay) over generic updates (e.g., PATCH /orders/42). It highlights the importance of modeling processes as state machines and ensuring transactional integrity and security by associating specific methods with transactions rather than relying on generic update actions.

This feature thread tracks the development and application of machine learning techniques, specifically quantile regression, to assess merchant eligibility for cash advances. The goal is to accurately predict future sales and offer funding that balances risk for both merchants and Shopify. The technique allows for nuanced predictions that account for varying levels of sales volatility, ensuring that advances are structured for a high probability of repayment. This post details the use of propensity score matching (PSM) to evaluate the impact of Shopify Capital on merchant growth by comparing US merchants who received funding with a matched cohort of Canadian merchants who did not have access at the time. The methodology involved defining treatment and control groups, selecting relevant features for matching, employing a caliper matching algorithm, and assessing matching quality through standardized mean differences, visual diagnostics, and variance ratios.

This feature thread tracks the development and enhancement of Shopify's Trino query execution performance. Initial efforts focused on speeding up query execution time to meet the goal of p95 query results in five seconds or less. This involved tuning infrastructure and addressing the risks associated with changes. The development of custom verification, benchmarking, and profiling tooling was initiated to minimize the risk of various changes at scale. This tooling, built as a lightweight Python library, aims to extract undocumented tribal knowledge into code, providing familiar interfaces for different testing needs. The verification component leverages PyTest fixtures for structured unit testing of Trino clusters, allowing for correctness and performance assertions. The benchmarking component utilizes TPC-DS queries on sample datasets with varying scale factors to evaluate performance under standardized conditions. The profiling aspect is intended to provide deeper insights into query execution.

This feature thread tracks the development and application of AI and optimization techniques to solve complex routing problems like the Traveling Salesperson Problem (TSP). Initial efforts focused on addressing computationally hard problems and handling uncertainty in real-world scenarios. This post details Shopify's participation and victory in the AI4TSP Competition, showcasing the use of dialectic search for expected-value tours and deep reinforcement learning for on-the-fly tour construction. The work is relevant to optimizing Shopify's fulfillment system.

This feature thread tracks the development and implementation of GraphQL mutations for creating, updating, and deleting data within Shopify's systems. Initial efforts focused on understanding GraphQL queries for data fetching. This post introduces the concept of mutations as the mechanism for modifying data, detailing the process of generating mutation resolvers, defining input arguments, and handling the `resolve` method to interact with the database. It covers specific examples for creating, updating, and deleting data. This post also details the implementation of a calculated query cost method for rate limiting GraphQL APIs, which adapts to the data clients need and provides a more predictable load on servers compared to traditional request-based models. The cost is calculated based on the types and number of objects requested, with mutations having a higher base cost due to their side effects.

This feature thread tracks the development and enhancement of Shopify's application security program. Initial efforts focused on establishing a security mindset within the company culture and growing the security team from a single individual to dozens of engineers. The program is structured around scaling secure applications, scaling security teams, and scaling security interactions. Key strategies include establishing a homogeneous technical baseline (e.g., defaulting to Ruby on Rails), gaining programmatic control over certificate issuance and renewal through partnerships with Certificate Authorities like Let's Encrypt, and automating the deployment and management of TLS certificates across millions of domains to ensure a secure and consistent experience for merchants.

This feature thread tracks the development and implementation of relative cursor pagination across Shopify's APIs. Initially, the platform relied on offset-based pagination, which led to significant performance degradation for large datasets. This post details the introduction of relative cursor pagination as a solution, enabling faster and more scalable data retrieval by remembering the last record's ID or sortable field value. The initiative involves standardizing pagination patterns, providing faster and more scalable data retrieval. This post also explores strategies for optimizing GraphQL queries on the client-side, including techniques for handling large and slow queries, preparing for paginated lists, rolling out new features, and making queries faster. It details strategies like designing base queries, loading multiple product pages, controlling new field rollouts using `@include` and `@skip` tags, chaining queries, and using parallel queries to improve performance and scalability.

This feature thread tracks the development and enhancement of tools and strategies for managing and discovering visual assets, specifically illustrations, across Shopify's large and distributed codebase. Initial efforts focused on the challenges of updating illustration styles and the fragmentation of assets across thousands of repositories. This post details the creation of `@shopify/get-repo-images`, an open-source tool built in Go, which enables illustrators and developers to efficiently find illustrations, their references in code, and their usage across multiple repositories. The tool provides capabilities for sorting, filtering, and browsing assets, significantly improving the efficiency and accuracy of managing visual assets at scale.

This feature thread tracks the development of surfacing merchant analytics directly within their operational workflows in the Shopify admin. Initial efforts focused on establishing a dedicated Analytics section. This evolution brings key metrics and insights to contextually relevant pages, such as the Orders page, to enable faster, data-informed decision-making by reducing navigation between different sections of the admin. The process involves identifying relevant metrics, understanding data sh. This post details a product thinking approach to dashboard creation, emphasizing user-centered design, clear goals, and iterative development to ensure dashboards provide value and are actively used by stakeholders.

This feature thread tracks the development and enhancement of Shopify's storefront rendering performance. Initial efforts focused on rewriting the critical software that powers all online storefronts to offer the fastest online shopping experience possible. This involved extracting storefront traffic from the Rails monolith into a dedicated, single-purpose application. Key strategies included designing on top of an active-active replication setup, reading from dedicated read replicas, and building on top of an active-active replication setup. This post details the use of the Shopify Theme Inspector to analyze Liquid render performance, interpret flame graphs, and identify common performance bottlenecks such as heavy loops, nested loops, and excessive use of global scope variables. It provides guidance on what constitutes acceptable rendering times and how to optimize Liquid code for better performance.

This feature thread tracks the development and enhancement of Shopify's system for safely deploying new features using beta flags and phased rollouts. Initial efforts focused on reducing the blast radius of changes by rolling out features to a percentage of users, enabling instantaneous activation and rollback. The system defines core concepts like Subjects, BetaIdentifiers, BetaFlags (explicit per-Subject toggles), and BetaRollouts (percentage-based distribution). A key innovation is the performant `BetaRollout#enabled?` method using a digest modulo 100 to ensure consistent subject selection across rollout percentages. The thread also discusses the evolution to a higher-level 'Feature' abstraction to provide more flexibility, including escape hatches and kill switches, and highlights considerations for data compatibility, rollback disruption, and avoiding identifier reuse.

This feature thread tracks the development and enhancement of Shopify's approach to managing SSH keys within Google Cloud Platform project-wide metadata. Initial efforts focused on addressing the security risks associated with persistent, unmanaged SSH keys, which could lead to unauthorized access and security vulnerabilities. The development of SSH-Pruner aims to automate the management of these keys, ensuring they do not persist longer than necessary and distinguishing between Shopify's keys and Google's infrastructure keys. The solution involves iterating through projects, identifying compute-enabled projects, and pruning SSH keys that lack valid expiration dates, while preserving essential Google infrastructure keys and unexpired Shopify keys. This initiative aims to enhance security posture without negatively impacting developer workflows.

Shopify has evolved its container security and governance practices by implementing Binary Authorization for Kubernetes, leveraging Voucher for signing Docker images after security checks. This builds upon previous efforts to manage application secrets in containerized environments and the overall containerization strategy, moving away from less secure methods and towards automated, verifiable image deployment.

This feature thread tracks the development and enhancement of Shopify's media management system, focusing on the addition of video and 3D model support alongside existing image capabilities. Initial efforts focused on supporting legacy image infrastructure while introducing new media types. A key challenge was deciding between a callback-based approach and a pipeline approach for handling complex data operations and ensuring transactional integrity. The pipeline approach was adopted to improve code control, maintainability, and separation of concerns, with individual handlers for each media type and a structured `before_transaction`, `during_transaction`, `after_transaction` pattern. This evolution aims to provide merchants with richer product showcasing capabilities while maintaining system stability and developer efficiency.

This feature thread tracks the development and enhancement of Shopify's use of Nix for developer tooling. Initial efforts focused on rebuilding developer tooling using Nix, including managing dependencies and creating a more consistent development environment. This post details the practical application of Nix for managing Ruby gems and exploring its potential for Node.js dependencies, as well as addressing challenges related to build workflows and dependency updates in a remote work environment. The post introduces the core concepts of Nix: the Nix Store, Derivations, Sandboxing, and the Nix Language, explaining how they work together to make software dependency graphs explicit and immutable.

This feature thread tracks the development and enhancement of security measures to protect user sessions from hijacking attacks. Initial efforts focused on addressing the threat posed by tools like Firesheep, which exploit insecure connections to steal session cookies. The primary solution implemented was the widespread adoption of SSL/TLS for all traffic within the Shopify admin area. This post details the evolution from manually provisioning TLS certificates to a fully automated system that supports over 1M business owners, including the development of the Notary system for automated provisioning and scaling, the integration of Let's Encrypt, and the migration to a cloud provider while maintaining the Notary infrastructure for resiliency.

This feature thread tracks the development and enhancement of Ruby application boot time optimization at Shopify. Initial efforts focused on addressing the substantial time spent waiting for Rails to boot in the monolithic application, impacting developer iteration speed. This post details the introduction of Bootsnap, a gem that optimizes and caches expensive computations by implementing path pre-scanning for `Kernel#require` and `ActiveSupport` autoloads, and compilation caching for Ruby bytec. This post further details approaches to identifying and fixing slow code through profiling (using tools like rbspy, stackprof, rack-mini-profiler, and app_profiler) and benchmarking (using Benchmark module, bm, bmbm, and benchmark-ips). It provides case studies on using these techniques to address performance regressions, such as unnecessary GC cycles caused by object allocation issues.

Shopify has established a robust API versioning strategy to ensure stability and predictability for its partners. This involves releasing new API versions quarterly with date-based names (e.g., 2020-01). The platform team builds the infrastructure and tooling to enforce this strategy, which includes managing breaking and non-breaking changes. Breaking changes require thorough impact analysis and stakeholder agreement, while non-breaking changes are tested for forward compatibility. Tools like `ApiChange.in_effect?`, `mark_breaking`, and `mark_possibly_breaking` are used to track changes, measure their impact, and manage adoption across the ecosystem. This system aims to build trust by respecting the impact of API decisions on third-party developers and merchants.

Shopify has redesigned its MySQL backup and restore tooling to leverage GCP's Persistent Disk (PD) snapshot feature. This has significantly reduced the Recovery Time Objective (RTO) to under 30 minutes. The new system uses Kubernetes CronJobs for frequent, automated PD snapshots across all regions, minimizing data loss and enabling faster replica rebuilds. A robust retention policy manages snapshot storage costs. A verification process ensures data integrity by restoring and checking daily snapshots for instance startup, replication, and corruption. For offsite disaster recovery, data from snapshots is compressed, encrypted, and archived with a third-party provider. While PD snapshots offer speed and efficiency, they are more expensive for storage and require additional tooling for integrity checks and offsite archiving.

This feature thread tracks the strategy and implementation of continuously integrating the latest upstream commits of critical frameworks, such as Rails, into production. Initial efforts focused on upgrading major framework versions, but evolved to a proactive approach of living on the 'edge' of the framework's master branch. This involves automated tools for weekly integration, extensive CI testing against a large test suite, and a phased deployment strategy using canaries to mitigate risks. The introduction of Shipit's merge queue and unsafe commit detection further enhances this by providing automated checks and management of the merge process, ensuring safer and faster deployments by preventing problematic commits from reaching production and automating the handling of reverts.

This feature thread tracks the development and enhancement of tools and strategies for managing and debugging Kubernetes Ingress controllers, specifically ingress-nginx. Initial efforts focused on improving the quality of the debugging experience for ingress-nginx, which is critical for Shopify's infrastructure. This post introduces a new kubectl plugin that simplifies upgrades by identifying breaking changes, provides a more detailed listing of ingress configurations, and streamlines common debugging tasks. It also details the contribution of dynamic configuration to ingress-nginx, allowing for in-memory endpoint updates and eliminating the need for NGINX reloads during application deployments, significantly improving request latencies, especially during deploys.

This feature thread tracks the development and enhancement of Shopify's Android Point of Sale (POS) application architecture. Initial efforts focused on addressing architectural inconsistencies by migrating from competing MVC and MVP patterns to a unified Model-View-ViewModel (MVVM) approach. This adoption leverages Google's Android Architecture Components, including ViewModel and LiveData, to ensure consistent architecture, automatic state retention across configuration changes, and a clearer separation of concerns for improved testability and developer onboarding. The strategy involves adopting MVVM for all new screens and converting older screens incrementally.

This feature thread tracks the development and enhancement of Shopify's capabilities for locale-aware number and currency condensing. Initial efforts focused on addressing the challenges of displaying long numbers in graphs within the Shopify admin across multiple languages. The solution involved leveraging Unicode CLDR data and Intl.js to create a library that abbreviates numbers and currencies according to the user's locale, ensuring consistent design and improved readability for merchants worldwide. The library was open-sourced as the `@shopify/condense-number` npm package.

This feature thread tracks the development and enhancement of integrating new browser payment APIs to streamline the checkout experience. Initial efforts focused on evaluating the Payment Request API, including its `canMakePayment()` method and the payment sheet UI. Experiments were conducted to measure its impact on checkout completion rates and purchase speed, comparing it against traditional checkout flows. The integration involved building an isolated Ruby app leveraging Shopify's existing APIs and app ecosystem, injecting script tags onto merchant storefronts, and overriding the checkout button to initiate the Payment Request flow. Considerations for alternative payment flows, such as discount codes and gift cards, were also addressed. Future work will involve analyzing experiment data and exploring broader adoption.

This feature thread tracks the development and enhancement of Shopify's Continuous Integration and Continuous Delivery (CI/CD) infrastructure for iOS applications. Initial efforts focused on building a unified build system, Shopify Build, based on Buildkite for Linux and Android projects. This post details the significant challenges and solutions for extending this unified system to iOS, which requires macOS environments. Key developments include the transition from a VMware ESXi-based system with a Storage Area Network (SAN) to a more scalable and maintainable infrastructure using Anka virtualization on Mac Minis with local storage. This evolution addresses issues of maintenance burden, single points of failure, and performance degradation, ultimately improving build stability, speed, and developer experience for iOS developers by providing disposable, on-demand build environments.

Shopify has evolved its approach to managing cloud storage usage, specifically Amazon S3, to address increasing error rates and latencies caused by rapid merchant growth. Initial efforts focused on understanding the impact of S3's internal partitioning mechanism on request rates and rate limits. The core problem identified was that a spike in activity from a single shop could trigger a 'SlowDown' exception for an entire partition, affecting multiple shops. To mitigate this, Shopify implemented a strategy of prefixing asset operations with a hash-generated digest derived from the shop ID and asset path. This injection of entropy distributes writes across more partitions, reducing the likelihood of hitting a single partition's rate limit and causing widespread failures. This change resulted in a significant reduction in SlowDown events and a notable decrease in read latencies for digested assets, improving overall platform reliability and performance. The lessons learned are applicable to other cloud storage providers as well.

This feature thread tracks the development and integration of ChatOps into Shopify's incident management procedures. Initial efforts focused on establishing a structured incident response process based on the Incident Command System (ICS). This post details the implementation of 'Spy', a chatbot built on Lita, to assist Incident Managers on Call (IMOCs) by automating and streamlining incident detection, communication, mitigation, and documentation. Spy integrates with tools like PagerDuty, StatusPage, and GitHub, providing commands for incident initiation, status updates, and action execution directly within Slack. It also includes features for IMOC well-being, such as reminders and fatigue prevention mechanisms.

This post details the evolution of code style consistency at Shopify, starting with the introduction of RuboCop and the development of Policial, a tool to enforce these standards on pull requests. It covers the challenges of managing a large, decade-old codebase, the iterative improvements to Policial (including using GitHub's commit status API), and strategies for fixing existing violations and preventing future ones by integrating RuboCop into CI pipelines. The goal is to achieve full compliance with the Shopify Ruby style guide.

This feature thread tracks the development and enhancement of the Facebook channel integration, focusing on scaling product synchronization capabilities. Initial efforts involved establishing the Facebook channel and growing its merchant base. Recent optimizations have addressed the serial nature of the product synchronization backend by introducing concurrent processing of product updates per shop and per product. This was achieved by leveraging Sidekiq Pro's batching functionality, where a supervisor job enqueues concurrent product sync jobs and manages their completion. The supervisor also implements a lock on the shop's domain to prevent race conditions at the product record level. Immediate re-enqueueing of supervisor jobs after completion further enhances worker utilization. These changes have significantly improved throughput, enabling horizontal scalability by adding more servers, and have shifted the system from a state of increasing backlog to actively decreasing it.

This feature thread tracks the development and enhancement of tools and strategies for debugging production Rails applications. Initial efforts focused on addressing the challenges of quickly identifying and resolving issues without extensive code changes or deployments. This post details the use of Ruby's introspection methods (`#class`, `#is_a?`, `#instance_methods`, `#method`, `#caller`, `#instance_variables`, etc.) and class/module methods (`::name`, `::class_variables`, `::instance_methods`, `::ancestors`, `::constants`, etc.) to inspect object types, inheritance, available methods, instance and class variables, and constants. It also covers the use of debuggers like `debugger` and `byebug` with commands like `var`, `backtrace`, and `frame` for live code inspection. The thread also touches upon the broader topic of production debugging tools, including the use of `gdb` for core dumps and inspecting Ruby object representations.

This feature thread tracks the development and enhancement of Shopify's approach to managing application secrets within containerized environments. Initial efforts focused on moving away from Chef's encrypted data bags due to limitations in security, auditing, and compatibility with containerization. The introduction of EJSON provided a solution for encrypting secrets with asymmetric encryption, allowing developers to manage secrets alongside their code. Subsequent development focused on secure key management and decryption processes within Docker containers, including the use of custom init processes and capabilities like CAP_SYS_ADMIN. Recent efforts involve attaching metadata to secrets for easier auditing and rotation, and exploring schema enforcement via CI checks.

This feature thread tracks the development and enhancement of Shopify's implementation of the Lua programming language in Go, known as go-lua. Initial efforts focused on creating a load generation tool that could simulate heavy traffic on Shopify's platform. To enable writing and executing new flows without full tool redeployments, a lightweight scripting language was needed. Lua was chosen for its simplicity and existing use within Shopify. The decision was made to implement Lua purely in Go (go-lua) rather than binding to the C implementation, offering advantages such as leveraging Go's garbage collector, reduced overhead for execution contexts, and easier mapping of concurrency and error handling semantics. The implementation involved manually porting the Lua 5.2 compiler and virtual machine from C to Go. Challenges encountered included performance optimizations for the interpreter's inner loop, particularly concerning Go's switch statement implementation and function call inlining, leading to the exploration of function table dispatch and careful rewriting of performance-critical code. Benchmarks showed go-lua performing slower than C Lua for certain operations, with differences attributed to debug hooks and call stack implementation. Limitations include the lack of support for coroutines, string pattern matching, string.dump, and weak reference tables.

Shopify's client-side JavaScript framework, Batman.js, is written in CoffeeScript. This post details the benefits of using CoffeeScript for writing JavaScript, including more concise code, fewer errors, and improved debugging workflows. It highlights how CoffeeScript compiles to JavaScript, allowing for standard minification and debugging processes. The workflow involves using a `coffee --watch --compile` command to automatically convert .coffee files to .js files, which are then committed to git.

This feature thread tracks the development and enhancement of tools and strategies for testing and developing webhooks. Initial efforts focused on addressing the challenges of testing webhooks during development, particularly the need for a publicly accessible URL. This post introduces PostCatcher, a web application that generates unique URLs for receiving and examining POST requests, and LocalTunnel, a service that exposes local machine ports to the internet, allowing developers to use localhost. This post details best practices for handling webhook requests, including ensuring quick responses, deferring processing, and implementing retry mechanisms. It also outlines strategies for recovering from extended outages and re-establishing webhook subscriptions.

This feature thread tracks the development and enhancement of Shopify's Rack middleware for input sanitization. Initial efforts focused on addressing issues with invalid form submissions during user login, specifically when users provided malformed shop URLs. The solution involved creating a custom Rack middleware that runs before OmniAuth to sanitize and recover bad form data, preventing errors caused by invalid URIs. This middleware utilizes a strategy pattern for different sanitization techniques, such as cleaning whitespace and handling domain name conventions.

This feature thread tracks the development and enhancement of real-time debugging tools for iOS applications, focusing on interactive object exploration without traditional breakpoints. Initial efforts introduced the Super Debugger, a wireless tool that allows developers to inspect and manipulate objects in running iOS apps (on devices or simulators) from a Mac application. This enables rapid iteration and immediate visual feedback by sending messages to objects and observing changes in real-time, bypassing the build-compile-wait cycle of traditional debugging.

This feature thread tracks the development and enhancement of strategies and tools for detecting and debugging memory leaks in production systems. Initial efforts focused on understanding the nature of memory leaks, differentiating between 'good' (intentional global allocations) and 'bad' (unintentional, growing allocations) leaks. The thread details the challenges of reproducing leaks, the limitations of general-purpose tools, and the exploration of language-specific profilers like memprof. It highlights the importance of teamwork, hunches, and iterative debugging in identifying root causes, including issues within C extensions and external libraries like memcached clients. The ultimate goal is to improve system stability and prevent crashes due to memory exhaustion.

This feature thread tracks the development and enhancement of tools and libraries for building Shopify applications using various programming languages and frameworks. Initial efforts focused on providing core API access. This post details the integration of the Python Shopify API with the Django web framework and Google App Engine, providing developers with a robust stack for building and deploying applications. It includes reusable Django app code for authentication, session management, and API access, along with deployment instructions for both regular Django applications and Google App Engine.