That Old Certificate Expired and Started an Outage. This is What Happened Next - Shopify

This post details an incident where the expiration of Let's Encrypt's old root certificate caused internal certificate errors due to a stale `cacert.pem` file within the Ruby `httpclient` library. The emergency fix involved forking the `httpclient` repository, updating `cacert.pem` with a current bundle, and using this fork. The post also discusses the delay in observing the issue due to HTTP keep-alive connections and the impact on various services, including product recommendations, marketing campaigns, and order confirmations. It highlights the importance of monitoring internal certificate stores and the potential for using system root certificates as a long-term solution.