TechPrufer
Get started
Cloudflare
Cloudflare logoCloudflare · Feature trail

IP Fragmentation Handling

2 posts · Feb 2015 – Mar 2026

What this trail covers

Cloudflare's platform has evolved to address the complexities and pitfalls of IP fragmentation. This includes understanding the limitations of IPv4 and IPv6 fragmentation, the critical role of Path MTU Discovery (PMTUD) and the 'Don't Fragment' (DF) flag, and the challenges posed by network middleboxes, NAT, and load balancing techniques like ECMP and Anycast. The company actively works to mitigate issues arising from dropped ICMP 'Packet too big' messages and misconfigured routers, aiming to enable the Cloudflare One Client to dynamically adjust its virtual interface MTU on the fly by periodically validating the capacity of the path, ensuring seamless transitions between networks with different MTU limits and maintaining stable connections for applications.

All Cloudflare trails
  1. Ending the "silent drop": how Dynamic Path MTU Discovery makes the Cloudflare One Client more resilientThis post introduces the implementation of Dynamic Path MTU Discovery (PMTUD) in the Cloudflare One Client, leveraging RFC 8899 and the MASQUE protocol over Cloudflare's QUIC library. The client actively probes the network path to determine the optimal packet size, removing reliance on potentially dropped ICMP messages. This allows the client to dynamically resize its virtual interface MTU on the fly, ensuring stable connections across networks with varying MTU restrictions, from standard Ethernet to specialized links like LTE/5G and satellite.
  2. Broken packets: IP fragmentation is flawedThis post details the inherent flaws in IP fragmentation, explaining why relying on intermediate routers to fragment packets is problematic due to issues like fragment loss, inefficient router resource usage, memory exhaustion attacks, and firewalling difficulties. It highlights the importance of the 'Don't Fragment' (DF) flag and ICMP 'Packet too big' messages for Path MTU Discovery (PMTUD). The post further elaborates on real-world internet complexities, including NAT, carrier-grade NAT, ISP middleboxes, Anycast, and ECMP load balancing, and how these factors often lead to PMTU black holes by dropping fragments or ICMP messages. It presents four scenarios illustrating these problems and discusses solutions like userspace daemons for handling ICMP messages in ECMP environments.
  3. Path MTU discovery in practiceThis post details the problem of Path MTU Discovery (PMTUD) breaking due to ECMP routing, where ICMP messages are not reliably delivered to the correct server handling a TCP connection. It explains the concepts of MTU, fragmentation, the 'Don't Fragment' bit, and ICMP type 3 code 4 messages. Solutions discussed include reducing MTU on the client, reducing MTU on the server, and enabling RFC4821 PMTUD. Cloudflare's specific issue arose from increased internal BGP load balancing with ECMP. Temporary fixes involved reducing IPv6 MTU to 1280 and enabling RFC4821 for IPv4. A comprehensive solution involves broadcasting ICMP MTU messages to all servers and open-sourcing a 'Path MTU Daemon' (pmtud) to achieve this.