Production Engineering Model

7 posts · Oct 2016 – Oct 2022

What this trail covers

Shopify has evolved its engineering organization towards a Production Engineering model, inspired by industry practices like SRE and Facebook's Production Engineering. The goal is to unify development and operations responsibilities, reduce manual toil through automation and tooling, and improve self-service capabilities for feature teams. This involves building common infrastructure, shared operational responsibility, and world-class developer tools. The model aims to balance management support and empower feature teams with ownership and operational excellence.

All Shopify trails

Oct 2022The Management Poles of Developer Infrastructure Teams - ShopifyThis post introduces a management model for developer infrastructure teams that balances three poles: management support, system and domain expertise, and roadmaps. It discusses challenges specific to Dev Infra teams, such as lack of dedicated PMs and UX experts, and the need for extensive domain expertise. Strategies for balancing these poles are explored, including temporarily assigning people from one team to another, having an entire team contribute to another team's goals, and forming tiger teams for specific projects. The post emphasizes that there are no perfect solutions and trade-offs are necessary.
Aug 2022Spin Cycle: Shopify’s SFN Team Overcomes a Cloud-Development Spiral - ShopifyThis post details the challenges and strategies employed by the Shopify Fulfillment Network (SFN) team to migrate their monolithic Rails application to Spin, Shopify's cloud-based development tool. It describes the technical complexities of handling numerous external dependencies by either running them in the same container, mocking them, or using shared environments. The post also highlights the human systems aspect, detailing the 'Spiral of Struggle' feedback loop where development friction discourages Spin adoption, leading to regressions. The SFN team's solution involved focusing on core functionality, prioritizing bug fixes that impacted common flows, and encouraging other developers to use and contribute to Spin's development, aiming to transition to a 'Loop of Happy Equilibrium'.
Jan 2022Cloud, Load, and Modular Code: What 2022 Looks Like for Shopify - ShopifyThis post outlines Shopify's strategic focus for 2022, emphasizing the evolution towards a Production Engineering model. Key initiatives include building more modular code by further deconstructing the Rails monolith, exemplified by splitting out storefront rendering for performance gains. It also highlights the adoption of cloud development environments like Spin to accelerate developer productivity and the integration of load testing as a natural part of the development process to handle massive traffic surges. The post also mentions the impact of YJIT on Ruby performance and reaffirms the commitment to Rails and React Native.
Aug 2021Modelling Developer Infrastructure Teams - ShopifyThis post introduces a model for organizing Developer Infrastructure teams by categorizing them into horizontally integrated teams (supporting the entire develop-validate-deploy cycle) and vertically integrated teams (specializing in specific technology stacks across multiple workflow phases). It analyzes the implications of these structures on scale, scope, and collaboration, and proposes avenues for future development (Concentration, Expansion, Interfacing) to inform organizational strategy.
Aug 2018E-Commerce at Scale: Inside Shopify's Tech Stack - Stackshare.io - ShopifyThis post details the evolution of Shopify's engineering organization towards a Production Engineering model, merging Operations and Performance teams into a Production Engineering department. It highlights the shared responsibility for ongoing operation of end-user applications, including monitoring and incident response. The post also introduces ServicesDB, an internal app for tracking production services, and discusses the shift from shards to isolated 'pods' for improved reliability.
Advertisement
Mar 2018Shopify’s Infrastructure Collaboration with Google - ShopifyThis post details Shopify's strategic decision to build its cloud infrastructure with Google Cloud Platform (GCP), specifically leveraging Google Kubernetes Engine (GKE). It outlines the company's journey from its early days as a cloud company, through pragmatic infrastructure evolution including database and application sharding, to the adoption of Docker and then Kubernetes for container orchestration. The post highlights the migration of over 50% of data center workloads to GCP, the development of a Shop Mover tool for database shard rebalancing, and contributions to Grafeas for container governance. It also mentions the growth to over 400 production services and the development of a Platform as a Service (PaaS) on Kubernetes.
Oct 2016Why Shopify Moved to The Production Engineering Model - ShopifyThis post details the organizational shift at Shopify to a Production Engineering model, moving away from a traditional Development/Operations split. It outlines the motivations for this change, including the pain points of frequent changes in large-scale web applications and the need to balance innovation speed with operational expertise. The new model integrates developers into a dedicated Production Engineering team, focusing on automation, reducing disconnected teams, and providing tools for feature teams. Key outcomes include increased self-service for feature teams, shared operational responsibility, and a focus on building developer infrastructure. The post highlights a threefold improvement in deploy speed and release frequency within the first year.