TechPrufer
Get started
Shopify
Shopify logoShopify · Feature trail

DNS Traffic Management for Resiliency

4 posts · Feb 2019 – Jan 2022

What this trail covers

Shopify has evolved its DNS traffic management strategies to improve service resiliency and availability. This has evolved into a fully automated, self-served system utilizing two DNS providers with layered traffic management capabilities, including geographical fencing, endpoint status health checking, priority-based failover, and weighted selection. The system aims to prevent outages caused by issues like expired root certificates in embedded client stores. The incident described highlights how the system has evolved from manual TLS certificate provisioning to a fully automated, cloud-integrated system that supports over 1M merchants, including the adoption of Let's Encrypt and the integration with cloud provider managed certificates, before returning to a unified approach using their own Notary system for both edge networks.

All Shopify trails
  1. That Old Certificate Expired and Started an Outage. This is What Happened Next - ShopifyThis post details an incident where the expiration of Let's Encrypt's old root certificate caused internal certificate errors due to a stale `cacert.pem` file within the Ruby `httpclient` library. The emergency fix involved forking the `httpclient` repository, updating `cacert.pem` with a current bundle, and using this fork. The post also discusses the delay in observing the issue due to HTTP keep-alive connections and the impact on various services, including product recommendations, marketing campaigns, and order confirmations. It highlights the importance of monitoring internal certificate stores and the potential for using system root certificates as a long-term solution.
  2. Let’s Encrypt x Shopify: Securing the Web 4.5 Million Domains at a Time - ShopifyThis post details Shopify's partnership with Let's Encrypt for automated SSL certificate provisioning and renewal, which is a critical component of their DNS traffic management and overall service resiliency. It highlights the challenges of manual certificate management, the benefits of the ACME protocol for automation, and how Let's Encrypt's open and programmatic approach enabled Shopify to provision and renew certificates for millions of domains efficiently. The post also touches upon the initial scalability issues encountered with other providers and how Let's Encrypt's open specification and implementation facilitated debugging and faster rollout, ultimately leading to Let's Encrypt becoming Shopify's primary certificate authority for enhanced reliability and incident response.
  3. An Introduction to DNS Traffic Management - ShopifyThis post provides an overview of DNS and its role in traffic management, explaining concepts like TTL, negative TTL, and the hierarchical structure of DNS. It details four use cases for DNS traffic management: easy failover (active-passive), sharing traffic between endpoints (active-active), and progressively deploying changes. It also discusses the trade-offs between low TTL for accuracy and DNS resolution time for performance, and acknowledges that DNS traffic management is not instantaneous due to caching.
  4. Using DNS Traffic Management to Add Resiliency to Shopify’s Services - ShopifyThis post details Shopify's journey from various manual and semi-automated DNS traffic management approaches to a consolidated, fully automated, and self-served system. It outlines the requirements for reliability and redundancy, leading to the adoption of two DNS providers. The post defines a layered traffic management approach: Geographical Fencing, Endpoint Status (health checking), Endpoint Priority, and Weighted Selection. It also describes the deployment and maintenance process using a git repository and spy commands for configuration.
  5. A Brief History of TLS Certificates at Shopify - ShopifyThis post details Shopify's journey from manually provisioning TLS certificates to a fully automated system. Key contributions include the development of the 'Notary System' for automated certificate provisioning via API calls to certificate authorities, handling domain ownership verification, and certificate/private key delivery. It also covers scaling certificate provisioning by dynamically loading TLS certificates from a database in Lua middleware via the `ssl_certificate_by_lua` module, using an in-memory LRU cache to reduce latency. The post also discusses solving mixed content warnings by processing shop themes to replace HTTP with HTTPS and the migration to Let's Encrypt. Finally, it details the move to a cloud provider, initially using managed certificates, and then adapting the Notary system to upload certificates to the cloud provider for both edge networks.
  7. Engineering a Historic Moment: Shopify Gets Ready for Cannabis in Canada - ShopifyThis post details the creation of a new regional infrastructure in Montreal to support regulated industries like cannabis sales in Canada. This involved setting up new Google Cloud Platform projects, utilizing Google Kubernetes Engine and Google Compute Engine, and configuring infrastructure with Terraform. It also describes the extensive load testing and pre-mortem discussions undertaken to prepare for high traffic volumes and potential security threats. Furthermore, it outlines the technical measures implemented to ensure personal information is stored within Canada, including application-level libraries, network firewall rules, container security, and SSL certificate management.