TechPrufer
Sign inGet started

Blogs›Cloudflare›IP Fragmentation Handling

IP Fragmentation Handling

IP Fragmentation Handling

2
posts
2015–2017

Cloudflare's platform has evolved to address the complexities and pitfalls of IP fragmentation. This includes understanding the limitations of IPv4 and IPv6 fragmentation, the critical role of Path MTU Discovery (PMTUD) and the 'Don't Fragment' (DF) flag, and the challenges posed by network middleboxes, NAT, and load balancing techniques like ECMP and Anycast. The company actively works to mitigate issues arising from dropped ICMP 'Packet too big' messages and misconfigured routers, aiming to enable smart MTU black hole detection and broadcast ICMP MTU messages to all servers to ensure they hit the relevant server handling a flow, regardless of ECMP forwarding. They have also reduced the MTU on IPv6 to a safe value of 1,280 and are enabling RFC4821 path MTU discovery for IPv4.

2017

Broken packets: IP fragmentation is flawed

8/18/2017

This post details the fundamental problems with IP fragmentation, including the unreliability of fragment delivery, wasted router resources, memory exhaustion risks, and firewall filtering limitations. It highlights the importance of the 'Don't Fragment' (DF) flag and Path MTU Discovery (PMTUD) for TCP connections. The post also elaborates on real-world internet complexities, such as NAT, carrier-grade NAT, ISP middleboxes, ECMP, and Anycast routing, and how these factors often lead to PMTUD black holes and fragmentation issues. It provides specific scenarios illustrating client-to-server and server-to-client communication challenges with both DF set and cleared.

2015

Path MTU discovery in practice

2/4/2015

This post details an incident where a change in internal BGP load balancing using ECMP broke Path MTU Discovery for a small number of users tunneling IPv6 over IPv4. It explains the concepts of MTU, IP fragmentation, the 'Don't Fragment' bit, and ICMP 'Destination Unreachable' messages (type 3 code 4). It highlights how ECMP, by hashing only source and destination IPs for ICMP, can send these messages to the wrong server, leading to ICMP black holes. Temporary fixes included reducing the MTU for IPv6 to 1280 and enabling RFC4821 PMTUD for IPv4. The comprehensive solution developed is an open-source 'Path MTU Daemon' that broadcasts ICMP MTU messages to all servers.