1.1.1.1 lookup failures on October 4, 2023

What this post added

This post details a specific incident where a new DNS record type (ZONEMD) caused parsing failures in Cloudflare's static_zone application, preventing the update of the root zone cache. This led to the use of a stale root zone with expired DNSSEC signatures, resulting in an increase in SERVFAIL responses. The post explains the technical details of the failure, including the binary DNS format, RDLENGTH handling, and the behavior of the static_zone application when parsing fails. It also describes the incident timeline, impact, and the steps taken to resolve the issue, including disabling the static_zone app and the partial impact due to rolling reboots and serve stale mechanisms.