How recurring network maintenance exposed 6 bugs

This post details the discovery and resolution of six bugs encountered during recurring network maintenance. It specifically addresses: 1. An eleven-year-old glibc bug in the DNS resolver leading to segfaults under specific error conditions. 2. A UDP socket leak in the OCaml socket library when `bind` fails under ephemeral-port pressure. 3. A fourteen-year-old regression in Async where `Tcp.connect_sock`'s timeout no longer covers DNS resolution. 4. A Kafka client request retry loop that times out without aborting the underlying attempt. 5. A bug in the Kafka client's bootstrapping logic that can duplicate concurrent lookups for broker discovery. 6. A job runner migration that silently reduced file-descriptor limits. The post outlines the investigation process, the root causes, and the fixes for each bug, contributing to the ongoing effort of identifying and mitigating complex system failures.