Debugging war story: the mystery of NXDOMAIN

This post details a debugging war story where transient packet loss on the internal network caused DNS lookups to fail with NXDOMAIN errors. The root cause was identified as the interaction between DNS query timeouts, the `search` option in `/etc/resolv.conf`, and a custom DNS server that dropped specific requests while returning NXDOMAIN for others. The post outlines the investigation process, including load testing, network packet analysis, and the creation of a proof-of-concept DNS server to replicate the issue. Solutions proposed include tuning `timeout` and `retries` in `resolv.conf` to better handle transient network issues.