The revenge of the listening sockets

What this post added

This post details a specific latency issue encountered within Cloudflare's datacenter related to DNS services. It identifies the root cause as the Linux kernel's listening hash table (LHTABLE) becoming a bottleneck when a large number of TCP sockets (16k) were bound to specific IP addresses on port 53. The post explains how this led to slow lookups in the LHTABLE, impacting ping times. It then describes the implemented solutions: changing TCP binding to 'bind to star' (0.0.0.0:53) and increasing the LHTABLE size, which resolved the latency problem and ensured consistent performance.