That Old Certificate Expired and Started an Outage. This is What Happened Next - Shopify

What this post added

This post details an incident where the expiration of Let's Encrypt's old root certificate caused internal service outages due to a stale `cacert.pem` file embedded in the `httpclient` Ruby gem. The fix involved forking the `httpclient` repository, updating `cacert.pem` with a current root certificate bundle, and deploying the patched version. This incident also revealed that TLS validation errors were triggered by Kubernetes pod recreations during deployments, due to HTTP keep-alive connections being re-established. The post also touches upon the impact on various services and the role of canaries in mitigating the full impact on the monolith.