TCP Port Sharing Behavior

5/25/2023

This post details the discovery and root cause analysis of unbounded memory usage in TCP receive buffers on Cloudflare's production network. It identifies a specific scenario where Linux kernel TCP sessions can ignore memory allocation limits, leading to excessive memory consumption and impacting overall server performance. The post provides a minimal reproducible example using Python scripts and analyzes kernel code paths (tcp_clamp_window) to pinpoint the issue. It also discusses the observed behavior in production traffic, including scenarios where the user-space process stops reading data.

3/20/2023

This post introduces a new feature thread focused on TCP port sharing behavior in Linux. It details a series of quizzes and explanations designed to illuminate the complex rules governing when two TCP sockets can share a local address and port. The post breaks down the scenarios based on local IP and port uniqueness relative to remote addresses, and introduces the concepts of bind buckets and the kernel's internal tracking mechanisms for ports and sockets. It also explores the impact of `SO_REUSEADDR` on port sharing.

7/26/2022

This post details the complexities of Linux TCP receive buffer and window management, specifically focusing on how the `skmem_rb` (sk_rcvbuf) budget is divided between application data and metadata. It explains how `tcp_adv_win_scale` influences the advertised window size and the impact of exceeding the memory budget, leading to packet drops and sender backoff. The post also highlights the roles of TCP Coalesce and TCP Collapse in optimizing metadata memory footprint and buffer utilization.

7/1/2022

This post details the optimization of TCP receive window sizes and sysctl parameters to balance high WAN throughput with low latency. It explains the trade-offs between TCP collapse processing (which causes latency spikes) and the need for large receive windows for high Bandwidth-Delay Product (BDP) connections. The post introduces Linux autotuning, explains key sysctl parameters like `tcp_rmem`, `tcp_wmem`, and `tcp_adv_win_scale`, and details the calculation and selection of specific sysctl values (e.g., `net.ipv4.tcp_rmem = 8192 262144 536870912`, `net.ipv4.tcp_wmem = 4096 16384 536870912`, `net.ipv4.tcp_adv_win_scale = -2`) to achieve the desired balance for products like Magic WAN.

9/20/2019

This post details the behavior of TCP sockets during connection establishment and idle states, focusing on the interaction between TCP keepalives and the TCP_USER_TIMEOUT socket option. It explains how SYN-SENT and SYN-RECV states handle retries and timeouts, and how idle ESTABLISHED connections can be monitored using keepalives. The post highlights the subtle and sometimes confusing interaction between TCP_USER_TIMEOUT and TCP keepalives, particularly how TCP_USER_TIMEOUT only takes effect after the first keepalive probe has been sent.

12/31/2012

This post details how to optimize the Linux TCP stack for mobile connections by adjusting TCP congestion control algorithms and kernel parameters. It explains the challenges of mobile networks (inherent packet loss) and how default TCP configurations, designed for wired networks, can misinterpret this loss as congestion, leading to suboptimal performance. The article discusses the analogy of TCP congestion control as managing water pressure in pipes and highlights the need for algorithms that can differentiate between congestive loss and random loss. It recommends upgrading Linux kernels to newer versions (e.g., 2.6.38+) to benefit from improved default settings like a larger initial congestion window (initcwnd) and initial receive window (initrwnd), and suggests exploring bandwidth estimation algorithms like Vegas, Veno, and Westwood+ for more nuanced loss handling.