TCP Server Connection Acceptance & Load Balancing

1/15/2018

This post details the inner workings of Linux's TCP SYN and Accept queues, explaining their roles in connection establishment and handling SYN floods. It covers queue size limits, the impact of slow applications on Accept Queue overflows, and the mechanism of SYN Cookies for stateless SYN+ACK generation. It also touches upon the interaction between SYN Cookies and TCP Timestamps.

11/7/2017

This post details the investigation into the SO_REUSEPORT socket option and its ability to improve packet locality. It explains the evolution from the standard BSD socket API to REUSEPORT and then to SO_INCOMING_CPU and SO_ATTACH_REUSEPORT_CBPF/EBPF for more advanced load balancing. The post introduces three SystemTap scripts: one to set CBPF on NGINX sockets, another to measure accept() calls and their CPU locality, and a third to measure packet locality (allocation/free on the same CPU). While end-to-end performance gains were not definitively proven for high-level HTTP servers, the scripts demonstrated near-perfect packet locality (99%) after extensive tuning.

10/23/2017

This post details the subtle differences in Linux's load balancing behavior between a blocking accept() model and an epoll-based accept() model. It demonstrates how the epoll model with EPOLLEXCLUSIVE can lead to uneven load distribution where one worker process receives a disproportionate amount of traffic due to LIFO-like behavior. It then introduces SO_REUSEPORT as a solution for better load balancing but highlights potential latency degradation issues with separate queues under high load.

8/12/2016

This post details a specific instance of a TCP connection timeout issue occurring between Cloudflare's edge and origin servers, which was traced back to a socket leak in an internal application. The leak caused sockets to remain in the CLOSE_WAIT state indefinitely. This prevented the proper cleanup of TCP connections, leading to a scenario where new connection attempts would time out because the kernel could not advance the connection state. The investigation involved analyzing tcpdump output, `ss` command results, and understanding the lifecycle of TCP states like FIN_WAIT_2 and CLOSE_WAIT, ultimately identifying the root cause as a failure to call `close()` on sockets in the listening application.

7/7/2016

This post explains why Cloudflare primarily uses the Linux kernel's TCP stack for its general-purpose networking needs, citing its hardware independence, API usability, time-sharing capabilities, rich feature set (e.g., PMTU discovery, hashlimits, ipsets), and extensive debugging tools. It contrasts this with kernel bypass technologies (PF_RING, Snabbswitch, DPDK, Netmap) which offer performance benefits but restrict applications to one process per network card. Cloudflare utilizes a 'partial kernel bypass' for DDoS mitigation to handle high packet rates (up to 3M pps) by offloading iptables to userspace via Solarflare EFVI or Netmap patches, thus avoiding IRQ storms. The post argues against full kernel bypass for applications like NGINX due to interference with debugging tools, firewall features, and the difficulty of maintaining custom TCP stacks.

4/11/2016

This post details a specific bug encountered in Cloudflare's stack related to slow downloads, particularly affecting mobile users. The issue was traced to a combination of NGINX's `send_timeout` (defaulting to 60 seconds) and `reset_timedout_connection` settings, exacerbated by a subtle asymmetry in how the Linux kernel reports socket writeability. The kernel's 'writeable' state requires free send buffer space to be greater than half of the used send buffer space, which, when combined with slow client download speeds, prevented NGINX from refilling its send buffer within the timeout period, leading to connection resets. The post describes the debugging process using `tcpdump` and `strace`, reproduces the issue with `curl --limit-rate`, and proposes solutions including increasing `send_timeout`, reducing `tcp_wmem`, or a custom NGINX patch for `send_minimum_rate`.

4/5/2016

This post details an issue where binding 16k TCP sockets to specific IP addresses on port 53 caused significant latency due to the Linux kernel's fixed-size, destination-port-only hashed listening hash table (LHTABLE). The problem was diagnosed using system tap scripts to measure soft IRQ handling (`net_rx_action`) and specifically the `__inet_lookup_listener` function within `tcp_v4_rcv`. The solution involved changing the DNS server to bind to ANY_IP for TCP traffic and increasing the LHTABLE size.

12/17/2015

This post details the merging of a new netmap mode into mainline netmap that supports partial kernel bypass. Previously, netmap was an all-or-nothing deal, requiring all queues to be detached from the host network stack. The new patch allows netmap to leave unrequested queues attached to the host stack, and packets destined for netmap TX rings are moved to the host RX ring. New flags (`NR_TX_RINGS_ONLY` and `NR_RX_RINGS_ONLY`) were introduced to request only TX or RX rings. The post also provides instructions and source code for building and running a test program that utilizes this new functionality.

11/19/2015

This post details the debugging of latency spikes in Cloudflare's CDN, identified as originating from the `tcp_collapse` function within the Linux kernel's TCP stack. The investigation used System Tap and flame graphs to pinpoint the issue, which was exacerbated by large TCP receive buffer sizes (`net.ipv4.tcp_rmem`). Tuning the `tcp_rmem` sysctl to a lower maximum value resolved the latency spikes by reducing the time spent in `tcp_collapse` operations.

10/9/2015

Introduced a 'single RX queue mode' for the Netmap project, enabling user-space applications to selectively receive packets from a specific RX queue on Intel IXGBE NICs, while leaving other RX and TX queues to the Linux kernel network stack. This allows for bifurcated driver functionality, where high-volume traffic can be offloaded to user-space for filtering during floods, while normal traffic continues through the kernel. The post details the implementation, configuration steps, and performance benefits observed.

5/11/2015

This post explores various TCP/IP quirks and low-level network stack details, including checksumming algorithms (IPv4 vs. IPv6), URG pointer usage, RST packet payloads, IPv6 flow fields, IP_FREEBIND socket option, PSH flag functionality, SYN cookies, UDP checksums, simultaneous TCP opens, stupid window syndrome, CWE/ECE flags, IP ID field behavior, SYN packet payloads, ICMP Path MTU handling, Linux TCP configuration (SYN queue length, overflow behavior), BGP bogons, and TCP MD5 extensions. It aims to encourage a deeper understanding of network protocols.

10/19/2011

Introduced end-to-end Keep Alives for TCP connections to origin servers. This change allows Cloudflare to reuse existing connections for multiple resource requests from a single visitor, reducing the overhead associated with establishing new TCP connections for each item. This optimization is expected to improve time to first byte and overall site performance by approximately 10%.