TCP Port Sharing Behavior

7 posts · Dec 2012 – Sep 2024

What this trail covers

Cloudflare's understanding and utilization of Linux's conntrack subsystem has evolved from avoiding it for simplicity to actively investigating its intricacies for new product needs. This includes understanding its state management, capacity limits, and the implications of its behavior when full, such as implicit packet drops and EPERM errors for UDP flows. The company is exploring its use in containerized environments and the nuances of its strict vs. loose modes. This post details how conntrack is used to manage TCP port sharing behavior, particularly in scenarios with a large number of bound TCP sockets, and how increasing the LHTABLE size and binding to ANY_IP addresses for TCP traffic resolved latency issues.

All Cloudflare trails

Sep 2024Bringing insights into TCP resets and timeouts to Cloudflare RadarIntroduced new dashboard and API endpoint on Cloudflare Radar to display near real-time data on anomalous TCP connections (resets or timeouts within the first 10 ingress packets). Detailed the stages of TCP connection lifecycle (establishment, data transfer, close) and defined anomalous connections. Discussed potential sources of anomalous connections including scanners, application shutdowns, network errors, attacks, and tampering. Provided guidance on interpreting the dataset by connection stage (Post-SYN, Post-ACK, Post-PSH, Later) and use cases for self-evaluation, exploring new targets, and longitudinal studies.
May 2023Unbounded memory usage by TCP for receive buffers, and how we fixed itThis post details the discovery and root cause analysis of unbounded memory usage in TCP receive buffers on Cloudflare's production network. It identifies a specific scenario where Linux kernel TCP sessions can ignore memory allocation limits, leading to excessive memory consumption and impacting overall server performance. The post provides a minimal reproducible example using Python scripts and analyzes kernel code paths (tcp_clamp_window) to pinpoint the issue. It also discusses the observed behavior in production traffic, including scenarios where the user-space process stops reading data.
Mar 2023The quantum state of a TCP portThis post delves into the intricate rules governing local address sharing between TCP sockets under Linux, specifically focusing on when two TCP sockets can share a local port. It uses a series of quiz questions to explore scenarios involving different local IPs, ports, and remote addresses, and the impact of `bind()` and `SO_REUSEADDR`. The underlying Linux kernel mechanisms, including the `bhash` and `ehash` tables and bind buckets, are explained to clarify the tri-state life of a local TCP port.
Jul 2022When the window is not fully open, your TCP stack is doing more than you thinkThis post details the complexities of Linux TCP receive buffer and window management, specifically focusing on how the `skmem_rb` (sk_rcvbuf) budget is divided between application data and metadata. It explains how `tcp_adv_win_scale` influences the advertised window size and the impact of exceeding the memory budget, leading to packet drops and sender backoff. The post also highlights the roles of TCP Coalesce and TCP Collapse in optimizing metadata memory footprint and buffer utilization.
Jul 2022Optimizing TCP for high WAN throughput while preserving low latencyThis post details the optimization of TCP receive window sizes and sysctl parameters to balance high WAN throughput with low latency. It explains the trade-offs between TCP collapse processing (which causes latency spikes) and the need for large receive windows for high Bandwidth-Delay Product (BDP) connections. The post introduces Linux autotuning, explains key sysctl parameters like `tcp_rmem`, `tcp_wmem`, and `tcp_adv_win_scale`, and details the calculation and selection of specific sysctl values (e.g., `net.ipv4.tcp_rmem = 8192 262144 536870912`, `net.ipv4.tcp_wmem = 4096 16384 536870912`, `net.ipv4.tcp_adv_win_scale = -2`) to achieve the desired balance for products like Magic WAN.
Advertisement
Sep 2019When TCP sockets refuse to dieThis post details the behavior of TCP sockets during connection establishment and idle states, focusing on the interaction between TCP keepalives and the TCP_USER_TIMEOUT socket option. It explains how SYN-SENT and SYN-RECV states handle retries and timeouts, and how idle ESTABLISHED connections can be monitored using keepalives. The post highlights the subtle and sometimes confusing interaction between TCP_USER_TIMEOUT and TCP keepalives, particularly how TCP_USER_TIMEOUT only takes effect after the first keepalive probe has been sent.
Aug 2016This is strictly a violation of the TCP specificationThis post details a specific instance of a TCP connection timeout occurring on the loopback interface due to a listening application leaking sockets and getting stuck in the CLOSE_WAIT state. It explains how this prevents the proper cleanup of TCP connections, leading to SYN packets being dropped and connection attempts timing out with ETIMEDOUT. The post provides reproduction steps and diagnostic commands to identify the issue.
Apr 2016The revenge of the listening socketsThis post details the investigation and resolution of high latency within Cloudflare's datacenter by analyzing Linux kernel behavior related to TCP socket lookups. It identifies a performance bottleneck in the `__inet_lookup_listener` function, specifically within the fixed-size `LHTABLE` (32 buckets) which is hashed by destination port. The issue was exacerbated by Cloudflare's DNS setup, which bound 16k TCP sockets to port 53 on different IP addresses, causing all these sockets to hash into a single `LHTABLE` bucket. This resulted in slow linear traversals of the bucket for incoming SYN and RST packets. The fixes deployed were: 1) binding the DNS server's TCP sockets to ANY_IP (0.0.0.0:53) instead of specific IPs, and 2) increasing the `LHTABLE` size in the kernel. These changes resolved the latency spikes, leading to flat ping times within the datacenter.
Dec 2012Optimizing Your Linux Stack for Maximum Mobile Web PerformanceThis post details how to optimize the Linux TCP stack for mobile connections by adjusting TCP congestion control algorithms and kernel parameters. It explains the challenges of mobile networks (inherent packet loss) and how default TCP configurations, designed for wired networks, can misinterpret this loss as congestion, leading to suboptimal performance. The article discusses the analogy of TCP congestion control as managing water pressure in pipes and highlights the need for algorithms that can differentiate between congestive loss and random loss. It recommends upgrading Linux kernels to newer versions (e.g., 2.6.38+) to benefit from improved default settings like a larger initial congestion window (initcwnd) and initial receive window (initrwnd), and suggests exploring bandwidth estimation algorithms like Vegas, Veno, and Westwood+ for more nuanced loss handling.