Linux Kernel Traffic Control Vulnerability Analysis

8 posts · Dec 2015 – Jun 2023

What this trail covers

Cloudflare's Linux kernel traffic control capabilities have evolved to address vulnerabilities, particularly those exploitable via USER namespaces. The 'noqueue' qdisc, when misapplied to classful qdiscs, was found to cause NULL pointer dereferences and kernel panics. The primary mitigation involved disallowing 'noqueue' for qdisc classes, aligning with documentation and preventing misuse. This evolution focuses on hardening the kernel's traffic control subsystem against unintended behavior and.

All Cloudflare trails

Jun 2023Lost in transit: debugging dropped packets from negative header lengthsThis post details the debugging of dropped packets after switching from Foo-over-UDP (FOU) encapsulation with virtual interfaces to IPVS native support for Generic UDP Encapsulation (GUE). The root cause was identified as packets exceeding the Internet MTU of 1500 bytes being dropped by the traffic directors, despite being smaller than internal network MTUs. The debugging process involved using Cilium's `pwru` tool and Linux `perf-probe` to trace packet flow through `validate_xmit_skb` and `ip_vs_tunnel_xmit`, revealing that the `kfree_skb_reason` was called due to a negative header length issue within the GUE encapsulation processing.
May 2023Unbounded memory usage by TCP for receive buffers, and how we fixed itThis post details the discovery and resolution of a Linux kernel bug causing unbounded memory usage in TCP receive buffers. The problem was identified by observing excessive memory allocation in TCP sessions, exceeding configured limits and impacting server-wide performance. The root cause was traced to a kernel issue where `tcp_clamp_window` inappropriately increased `sk_rcvbuf` when `rmem_alloc` exceeded `sk_rcvbuf`, particularly under specific sending and receiving patterns. The solution involved addressing this logic within the kernel's TCP congestion control mechanism.
Jan 2023CVE-2022-47929: traffic control noqueue no problem?This post details the discovery and analysis of a Linux kernel bug (CVE-2022-47929) where assigning the 'noqueue' traffic control queue discipline to a classful qdisc (like HTB) leads to a NULL pointer dereference and kernel panic. It explains how USER namespaces can be leveraged to trigger this bug, enabling unprivileged denial-of-service attacks. The post also discusses the root cause in the kernel's handling of qdisc enqueue functions and the chosen mitigation strategy: disallowing 'noqueue' for qdisc classes.
Jan 2023A debugging story: corrupt packets in AF_XDP; a kernel bug or user error?This post details a debugging investigation into corrupt packets observed in Cloudflare's flowtrackd system, which uses AF_XDP for high-performance packet processing. The root cause was identified as a descriptor being inserted into the AF_XDP rings twice, leading to packet corruption. The investigation involved logging packet lengths and descriptor addresses, revealing that descriptor lengths from the RX ring did not match those in the descriptor, and that corrupt packets often shared the same descriptor address. The team developed a state machine to track descriptor transitions to pinpoint how a descriptor could end up in the rings twice, ultimately identifying a potential user-space error rather than a kernel bug.
Jul 2022A story about AF_XDP, network namespaces and a cookieThis post details a debugging journey and fix for a crash occurring when creating multiple AF_XDP sockets within different network namespaces. The root cause was identified as AF_XDP not being network namespace aware, specifically when interface index numbers conflicted. A fix was implemented by ensuring unique interface index numbers, allowing the shared UMEM feature to function correctly.
Advertisement
Nov 2021The tale of a single register valueThis post details the debugging process of a Linux kernel crash triggered by a single malformed network packet. It focuses on the `skb_gso_transport_seglen` function and the calculation of inner TCP header lengths, identifying a potential issue with the `doff` field in the inner TCP header. The analysis involves decoding kernel oops reports, disassembling machine code, and examining CPU register states to pinpoint the exact memory access that caused the page fault. The investigation highlights the complexities of Generic Segmentation Offload (GSO) packets and encapsulated traffic, emphasizing the need for robust error handling and data validation within the network stack.
May 2019Cloudflare architecture and how BPF eats the worldThis post details Cloudflare's use of eBPF and XDP for DoS mitigation and layer 4 load balancing on their edge servers. It highlights the challenges encountered, such as the lack of concurrency primitives (now addressed by bpf_spin_lock) and limitations in eBPF verifier for socket filters. It also discusses the use of iptables with xt_bpf for application layer mitigations and the use of SO_ATTACH_BPF for single UDP socket filters. A new project involving socket lookup from XDP for load balancing is also introduced, along with a corner case related to SYN-cookies and ACK validation.
May 2019eBPF can't count?!This post details a bug in the eBPF verifier that caused 64-bit subtractions to produce incorrect results when run by unprivileged users. The bug was introduced as a security fix for CVE-2019-7308 (Spectre variant 1) but incorrectly applied to eBPF arithmetic operations. The verifier rewrites eBPF code to prevent out-of-bounds memory access, but in this case, it misidentified scalar subtractions as pointer arithmetic. The issue was fixed in Linux kernel v5.0 and backported to stable kernels. A workaround using 32-bit ALU operations was also explored.
Feb 2019SOCKMAP - TCP splicing of the futureThis post introduces SOCKMAP, a new eBPF machinery in the Linux kernel (v4.14+) for TCP socket splicing. It details the challenges of traditional L7 proxies (syscall cost, wakeup latency, copying cost) and compares naive read-write loops, splice(2), io_submit, and SOCKMAP. While theoretically promising for zero-copy and zero-wakeup, benchmarks showed SOCKMAP to be the slowest due to bugs and overhead. The post includes code examples for each approach and discusses the importance of publishing negative results.
Mar 2018eBPF, Sockets, Hop Distance and manually writing eBPF assemblyThis post details the use of eBPF and eBPF maps to extract IP TTL values from TCP SYN+ACK packets. It explains how to create an eBPF map using the `bpf()` syscall, write an eBPF program in raw bytecode to inspect packet data and store TTL values in the map, and then read these values from userspace. The post highlights the challenges of accessing packet data with classical BPF and the advantages of eBPF with maps for this use case. It also covers the eBPF calling convention and accessing packet payload.
Nov 2017Perfect locality and three epic SystemTap scriptsThis post details the investigation into the REUSEPORT socket option and its impact on packet locality. It introduces three SystemTap scripts used to: 1) set the SO_ATTACH_REUSEPORT_CBPF option on NGINX sockets, 2) measure accept() calls to verify CPU affinity, and 3) measure packet locality (allocation/freeing on the same CPU). While perfect packet locality was achieved, no significant end-to-end performance improvement was observed for the high-level HTTP server. The post concludes that for such applications, the overhead of packet locality improvements may be negligible compared to other costs.
Dec 2015Partial kernel bypass merged into netmap mainThis post details the merging of a new netmap mode into the mainline netmap repository. The new mode allows for partial kernel bypass, enabling users to select specific RX/TX rings for netmap mode while leaving others attached to the host network stack. It also introduces flags to request only RX or TX rings and a mechanism to move packets destined for netmap TX rings to the RX host ring. A test program and build instructions are provided.