TechPrufer
Sign inGet started

Blogs›Shopify›DNS Traffic Management for Resiliency

DNS Traffic Management for Resiliency

DNS Traffic Management for Resiliency

4
posts
2019–2022

Shopify has evolved its DNS traffic management strategies to improve service resiliency and availability. This has evolved into a fully automated, self-served system utilizing two DNS providers with layered traffic management capabilities, including geographical fencing, endpoint status health checking, priority-based failover, and weighted selection. The system aims to prevent outages caused by issues like expired root certificates in embedded client stores. The incident described highlights how a stale `cacert.pem` in the `httpclient` library, used by various internal services, led to certificate validation errors after the expiration of Let's Encrypt's old root certificate. The emergency fix involved forking the `httpclient` repository, updating `cacert.pem` with a more recent bundle, and deploying this patched version. A long-term solution involves automating updates to this fork and potentially leveraging system root certificate stores.

2022

That Old Certificate Expired and Started an Outage. This is What Happened Next - Shopify

1/12/2022

This post details an incident where the expiration of Let's Encrypt's old root certificate caused internal service outages due to a stale `cacert.pem` file embedded in the `httpclient` Ruby gem. The fix involved forking the `httpclient` repository, updating `cacert.pem` with a current root certificate bundle, and deploying the patched version. This incident also revealed that TLS validation errors were triggered by Kubernetes pod recreations during deployments, due to HTTP keep-alive connections being re-established. The post also touches upon the impact on various services and the role of canaries in mitigating the full impact on the monolith.

2020

An Introduction to DNS Traffic Management - Shopify

8/5/2020

This post introduces the fundamental concepts of DNS and traffic management, explaining their combination into DNS traffic management. It provides a historical overview of DNS, its architecture (root servers, TLDs, nameservers, zones, records), and the role of TTL and caching. It also defines traffic management and its application to DNS, highlighting the trade-offs between low TTL for rapid updates and the impact on DNS resolution time. The post then details four use cases for DNS traffic management: easy failover (active-passive), sharing traffic between endpoints (active-active), progressive deployment of changes, and geographical distribution of traffic. This post sets the stage for a deeper dive into Shopify's specific implementation in a subsequent article.

Using DNS Traffic Management to Add Resiliency to Shopify’s Services - Shopify

8/5/2020

This post details the evolution of Shopify's DNS traffic management from fragmented, manual approaches to a consolidated, automated, and self-served system. It introduces a layered traffic management model comprising Geographical Fencing, Endpoint Status, Endpoint Priority, and Weighted Selection. The system leverages two DNS providers for redundancy and utilizes a git repository for configuration management, with spy commands for efficient updates and failovers. This post focuses on the design and implementation of this unified system to enhance service resiliency.

2019

Engineering a Historic Moment: Shopify Gets Ready for Cannabis in Canada - Shopify

2/7/2019

This post details the creation of a new regional infrastructure in Montreal to support regulated industries like cannabis sales in Canada. This involved setting up new Google Cloud Platform projects, utilizing Google Kubernetes Engine and Google Compute Engine, and configuring infrastructure with Terraform. It also describes the extensive load testing and pre-mortem discussions undertaken to prepare for high traffic volumes and potential security threats. Furthermore, it outlines the technical measures implemented to ensure personal information is stored within Canada, including application-level libraries, network firewall rules, container security, and SSL certificate management.