Unbounded memory usage by TCP for receive buffers, and how we fixed it
May 25, 2023Mike Freemon

Summary
This post details a specific, critical issue discovered in the Linux kernel's TCP receive buffer memory management. It explains how certain conditions can lead to unbounded memory allocation, impacting overall server performance and causing packet loss. The post outlines the root cause, provides a reproducible lab setup, and discusses the implications for Cloudflare's production network, highlighting the need for kernel-level fixes to maintain optimal performance and stability.
More from the Hardware Infrastructure & Performance Optimization trail
- How we reduced core unit boot time from hours to minutes
- A one-line Kubernetes fix that saved 600 hours a year
- Inside Gen 13- how we built our most powerful server yet
- Launching Cloudflare’s Gen 13 servers- trading cache for cores for 2x edge compute performance
- So long, and thanks for all the fish- how to escape the Linux networking stack