TechPrufer
Get started
Shopify
Shopify logoShopify · Feature trail

Continuous Integration of Framework Head

4 posts · Oct 2016 – Aug 2021

What this trail covers

Shopify has evolved its strategy for continuously integrating the latest upstream commits of critical frameworks, such as Rails, into production. This proactive approach involves living on the 'edge' of the framework's master branch, with automated tools for weekly integration, extensive CI testing against a large test suite, and a phased deployment strategy using canaries to mitigate risks. This builds upon previous efforts to upgrade major framework versions and manage the risks associated with long-running upgrades.

All Shopify trails
  1. Modelling Developer Infrastructure Teams - ShopifyThis post introduces a mental model for Developer Infrastructure teams, categorizing them by their role in the develop-validate-deploy cycle. It describes how teams like Local Environments, Cloud Environments, Test Infrastructure, and Deploys map to specific phases (horizontal integration), while teams like Web Foundations and React Native Foundations span multiple phases (vertical integration). The analysis of this model informs strategic decisions about work avenues (Concentration, Expansion, Interfacing) and organizational structure, leading to the creation of Client Foundations teams.
  2. Living on the Edge of Rails - ShopifyThis post details the strategy of 'Living on the Edge of Rails' by continuously integrating the Rails master branch into production. It introduces the Solid Track bot for automated weekly upgrades, the use of a large test suite for early detection of issues, and canary deployments for risk mitigation. It also highlights the benefits of this approach for both Shopify and the Rails community, including faster bug fixes, quicker feature adoption, and contributions to upstream projects like Zeitwerk.
  3. Introducing the Merge Queue - ShopifyIntroduced Shipit's merge queue to automate the process of merging pull requests into the main branch, reducing manual intervention and improving developer workflow. Implemented a browser extension to integrate the merge queue into the GitHub pull request interface, making it the default merging mechanism. Added configurable thresholds for branch divergence to discourage long-lived branches. Enhanced the merge queue to automatically handle pending CI statuses, merging or rejecting pull requests based on results. Developed features for marking commits as unsafe for deploy and automatic revert detection to improve the safety and automation of the deployment pipeline, especially in the presence of bad commits or reverts.
  4. Upgrading Shopify to Rails 5 - ShopifyThis post details Shopify's experience upgrading its monolith from Rails 4.2 to Rails 5.0. Key contributions include the strategy of 'dual boot' and 'dual CI' to maintain compatibility with both versions during the upgrade process, the use of conditional logic to manage version-specific code, and the discovery of issues related to serialized data in cache and deprecation warnings impacting performance. It also highlights the importance of keeping gems upgraded and the challenges of fixing tests and dealing with legacy code. The post concludes with lessons learned, emphasizing early removal of deprecated features, proactive gem upgrades, and the necessity of dual boot/CI for large applications.
  5. Automatic Deployment at Shopify - ShopifyThis post details the automation of the final manual deploy trigger, removing the "Hit deploy button" step and enabling automatic batching and deployment of merges via the GitHub merge button. It introduces the Shipit deploy robot for automated batching and cadence-based deploys, and Spy, a ChatOps robot for visibility and notifications. The post also discusses the performance impact of containerized deployments and the human impact of removing manual steps, leading to the development of automated reminders via Spy.