How and why the leap second affected Cloudflare DNS

What this post added

This post details a critical incident where a leap second caused a negative time value in Cloudflare's DNS resolution software (RRDNS), leading to a temporary outage for customers using CNAME records. It highlights a specific bug in the Go programming language's time handling and how Cloudflare's internal systems, particularly the weighted selection algorithm for upstream DNS resolvers, were affected. The post contributes to the understanding of Cloudflare's data infrastructure by showcasing the challenges of timekeeping in distributed systems and the engineering effort involved in identifying and fixing such subtle, yet impactful, issues.