TechPrufer
Get started
Cloudflare
Cloudflare logoCloudflare · Feature trail

Linux Connection Tracking (conntrack)

2 posts · Apr 2020 – Oct 2025

What this trail covers

Cloudflare's understanding and utilization of Linux's conntrack subsystem has evolved from avoiding it for simplicity to actively investigating its intricacies for new product needs. This includes understanding its state management, capacity limits, and the implications of its behavior when full, such as implicit packet drops and EPERM errors for UDP flows. The company is exploring its use in containerized environments and the nuances of its strict vs. loose modes. This post details how conntrack's interaction with socket binding can lead to unexpected source port rewriting, necessitating solutions like TCP_REPAIR or TCP Fast Open for scenarios involving packet rewriting and socket usage, particularly for soft-unicast address spaces.

All Cloudflare trails
  1. So long, and thanks for all the fish- how to escape the Linux networking stackThis post details a specific failure mode in the Linux networking stack where Netfilter's conntrack module, when used for SNAT, can silently rewrite the source port of TCP connections established via socket binding to a port outside of the intended range. This occurs because Netfilter is unaware of the socket subsystem's state. The post explores two solutions: terminating TCP connections locally (which adds overhead) and abusing kernel features like TCP_REPAIR or TCP Fast Open to create 'connected' sockets that prevent other processes from binding to the same 5-tuple, thereby ensuring consistency between packet forwarding and socket usage.
  2. Conntrack turns a blind eye to dropped SYNsThis post details how conntrack, by default, does not track dropped SYN packets. It explains that conntrack relies on Netfilter hooks and that the `enable_hooks` module parameter (introduced in Linux v5.1) is required to register conntrack callbacks with Netfilter at module load time. This enables conntrack to track all packets, including dropped SYNs, which are not recorded by default. The post uses drgn and bpftrace to demonstrate how to inspect Netfilter hooks and trace conntrack callbacks.
  3. Conntrack tales - one thousand and one flowsThis post details the behavior of Linux's conntrack subsystem when its state table fills up. It demonstrates through experiments that when the conntrack table exceeds its limit, new connection flows (e.g., SYN packets for TCP, or new UDP flows) are implicitly dropped without explicit iptables rules. It also shows that this can lead to EPERM errors for UDP sendto() syscalls. The post highlights the importance of ensuring the conntrack table does not fill up and discusses the strict vs. loose modes and situations where conntrack entries are not created.