Remediating new DNSSEC resource exhaustion vulnerabilities

This post details the implementation of specific resource exhaustion mitigations for DNSSEC vulnerabilities Keytrap and NSEC3 iteration count. For Keytrap, Cloudflare implemented limits on the number of keys accepted at a zone cut, and added signature validation limits per RRSET and per resolution task. For NSEC3, a limit on total hash calculations per resolution task was added. Both mitigations involve failing resolution for queries exceeding these limits and returning an Extended DNS Error (EDE) code. Metrics were also added to detect exploitation attempts.