TCP Server Connection Acceptance & Load Balancing

12 posts · Oct 2011 – Mar 2020

What this trail covers

Cloudflare's TCP connection handling has evolved to utilize end-to-end Keep Alives, reducing TCP overhead and latency by maintaining persistent connections to origin servers for multiple requests. This optimization improves the time to first byte and overall site snappiness, especially for sites with consistent traffic. The implementation benefits sites whose origin servers support Keep Alive connections. This post details how disabling keepalives for internal connections between SSL and FL proc

All Cloudflare trails

Mar 2020Keepalives considered harmfulThis post details an optimization where disabling keepalives for internal connections between SSL and FL processes, by forcing a new connection per request, significantly reduced latency. This was achieved by ensuring requests were always picked up by idle workers, thereby avoiding the load balancing issues caused by busy workers holding onto connections. The change resulted in a 4x reduction in the 99.9th latency percentile.
Jan 2018SYN packet handling in the wildThis post details the inner workings of Linux's TCP SYN and Accept queues, explaining their roles in connection establishment and handling SYN floods. It covers queue size limits, the impact of slow applications on Accept Queue overflows, and the mechanism of SYN Cookies for stateless SYN+ACK generation. It also touches upon the interaction between SYN Cookies and TCP Timestamps.
Nov 2017Perfect locality and three epic SystemTap scriptsThis post details the investigation into the SO_REUSEPORT socket option and its ability to improve packet locality. It explains the evolution from the standard BSD socket API to REUSEPORT and then to SO_INCOMING_CPU and SO_ATTACH_REUSEPORT_CBPF/EBPF for more advanced load balancing. The post introduces three SystemTap scripts: one to set CBPF on NGINX sockets, another to measure accept() calls and their CPU locality, and a third to measure packet locality (allocation/free on the same CPU). While end-to-end performance gains were not definitively proven for high-level HTTP servers, the scripts demonstrated near-perfect packet locality (99%) after extensive tuning.
Oct 2017Why does one NGINX worker take all the load?This post details the subtle differences in Linux's load balancing behavior between a blocking accept() model and an epoll-based accept() model. It demonstrates how the epoll model with EPOLLEXCLUSIVE can lead to uneven load distribution where one worker process receives a disproportionate amount of traffic due to LIFO-like behavior. It then introduces SO_REUSEPORT as a solution for better load balancing but highlights potential latency degradation issues with separate queues under high load.
Aug 2016This is strictly a violation of the TCP specificationThis post details a specific instance of a TCP connection timeout issue occurring between Cloudflare's edge and origin servers, which was traced back to a socket leak in an internal application. The leak caused sockets to remain in the CLOSE_WAIT state indefinitely. This prevented the proper cleanup of TCP connections, leading to a scenario where new connection attempts would time out because the kernel could not advance the connection state. The investigation involved analyzing tcpdump output, `ss` command results, and understanding the lifecycle of TCP states like FIN_WAIT_2 and CLOSE_WAIT, ultimately identifying the root cause as a failure to call `close()` on sockets in the listening application.
Advertisement
Jul 2016Why we use the Linux kernel's TCP stackThis post explains why Cloudflare primarily uses the Linux kernel's TCP stack for its general-purpose networking needs, citing its hardware independence, API usability, time-sharing capabilities, rich feature set (e.g., PMTU discovery, hashlimits, ipsets), and extensive debugging tools. It contrasts this with kernel bypass technologies (PF_RING, Snabbswitch, DPDK, Netmap) which offer performance benefits but restrict applications to one process per network card. Cloudflare utilizes a 'partial kernel bypass' for DDoS mitigation to handle high packet rates (up to 3M pps) by offloading iptables to userspace via Solarflare EFVI or Netmap patches, thus avoiding IRQ storms. The post argues against full kernel bypass for applications like NGINX due to interference with debugging tools, firewall features, and the difficulty of maintaining custom TCP stacks.
Apr 2016The curious case of slow downloadsThis post details a specific bug encountered in Cloudflare's stack related to slow downloads, particularly affecting mobile users. The issue was traced to a combination of NGINX's `send_timeout` (defaulting to 60 seconds) and `reset_timedout_connection` settings, exacerbated by a subtle asymmetry in how the Linux kernel reports socket writeability. The kernel's 'writeable' state requires free send buffer space to be greater than half of the used send buffer space, which, when combined with slow client download speeds, prevented NGINX from refilling its send buffer within the timeout period, leading to connection resets. The post describes the debugging process using `tcpdump` and `strace`, reproduces the issue with `curl --limit-rate`, and proposes solutions including increasing `send_timeout`, reducing `tcp_wmem`, or a custom NGINX patch for `send_minimum_rate`.
Apr 2016The revenge of the listening socketsThis post details an issue where binding 16k TCP sockets to specific IP addresses on port 53 caused significant latency due to the Linux kernel's fixed-size, destination-port-only hashed listening hash table (LHTABLE). The problem was diagnosed using system tap scripts to measure soft IRQ handling (`net_rx_action`) and specifically the `__inet_lookup_listener` function within `tcp_v4_rcv`. The solution involved changing the DNS server to bind to ANY_IP for TCP traffic and increasing the LHTABLE size.
Dec 2015Partial kernel bypass merged into netmap mainThis post details the merging of a new netmap mode into mainline netmap that supports partial kernel bypass. Previously, netmap was an all-or-nothing deal, requiring all queues to be detached from the host network stack. The new patch allows netmap to leave unrequested queues attached to the host stack, and packets destined for netmap TX rings are moved to the host RX ring. New flags (`NR_TX_RINGS_ONLY` and `NR_RX_RINGS_ONLY`) were introduced to request only TX or RX rings. The post also provides instructions and source code for building and running a test program that utilizes this new functionality.
Nov 2015The story of one latency spikeThis post details the debugging of latency spikes in Cloudflare's CDN, identified as originating from the `tcp_collapse` function within the Linux kernel's TCP stack. The investigation used System Tap and flame graphs to pinpoint the issue, which was exacerbated by large TCP receive buffer sizes (`net.ipv4.tcp_rmem`). Tuning the `tcp_rmem` sysctl to a lower maximum value resolved the latency spikes by reducing the time spent in `tcp_collapse` operations.
Oct 2015Single RX queue kernel bypass in Netmap for high packet rate networkingIntroduced a 'single RX queue mode' for the Netmap project, enabling user-space applications to selectively receive packets from a specific RX queue on Intel IXGBE NICs, while leaving other RX and TX queues to the Linux kernel network stack. This allows for bifurcated driver functionality, where high-volume traffic can be offloaded to user-space for filtering during floods, while normal traffic continues through the kernel. The post details the implementation, configuration steps, and performance benefits observed.
May 2015CloudFlare "Interview Questions"This post explores various TCP/IP quirks and low-level network stack details, including checksumming algorithms (IPv4 vs. IPv6), URG pointer usage, RST packet payloads, IPv6 flow fields, IP_FREEBIND socket option, PSH flag functionality, SYN cookies, UDP checksums, simultaneous TCP opens, stupid window syndrome, CWE/ECE flags, IP ID field behavior, SYN packet payloads, ICMP Path MTU handling, Linux TCP configuration (SYN queue length, overflow behavior), BGP bogons, and TCP MD5 extensions. It aims to encourage a deeper understanding of network protocols.
Oct 2011Stayin' AliveIntroduced end-to-end Keep Alives for TCP connections between Cloudflare's network and origin servers. This change reduces TCP overhead and latency by maintaining persistent connections, leading to improved time to first byte and overall site snappiness, with an average performance improvement of approximately 10%.