TechPrufer
Get started
Cloudflare
Cloudflare logoCloudflare · Feature trail

Internal DNS Resolution Reliability

1 post · Dec 2016

What this trail covers

Cloudflare's internal DNS resolution has evolved to address transient network issues, specifically packet loss, that could lead to spurious NXDOMAIN errors. The system now incorporates improved client-side handling of timeouts and retries, and a more robust understanding of how search domains interact with DNS failures. This ensures more reliable DNS lookups within the internal cluster, even under adverse network conditions.

All Cloudflare trails
  1. Debugging war story: the mystery of NXDOMAINThis post details a debugging war story where transient packet loss on the internal network caused DNS lookups to fail with NXDOMAIN errors, despite the domains existing. The investigation involved load testing, analyzing network captures, and simulating packet loss. The root cause was identified as a combination of lost DNS requests, the default `resolv.conf` retry mechanism, and the interaction of the `search` domain with failed lookups. Solutions proposed include increasing DNS timeouts and retries, and a proof-of-concept DNS server to reliably reproduce the issue.