1.1.1.1 lookup failures on October 4, 2023

This post details an incident where a new ZONEMD record in the root zone caused parsing failures in Cloudflare's static_zone WebAssembly app, preventing updates to the root zone. This led to the use of a stale root zone with expired DNSSEC signatures, resulting in SERVFAIL responses for some 1.1.1.1 users. The incident was resolved by disabling the static_zone app and implementing a new override rule. The post also explains the binary DNS format, the role of RDLENGTH, and the challenges of handling unknown resource record types.