Cloudflare

Lost in transit: debugging dropped packets from negative header lengths

Jun 26, 2023Terin Stock
Lost in transit: debugging dropped packets from negative header lengths
SummaryAI, from the source post

This post details the debugging and resolution of packet drops encountered after switching from Foo-over-UDP encapsulation with virtual interfaces to IPVS native Generic UDP Encapsulation (GUE). The issue stemmed from negative header lengths during encapsulation, causing packets exceeding the Internet MTU to be dropped by the traffic directors. The root cause was identified through eBPF tracing with `pwru` and `perf-probe`, revealing packet drops within the `validate_xmit_skb` function called from `ip_vs_tunnel_xmit`. The fix involved addressing the underlying kernel behavior related to header length calculation during GUE encapsulation.

More from the Load Balancing UI and Feature Enhancements trail