Lost in transit: debugging dropped packets from negative header lengths
Jun 26, 2023Terin Stock

Summary
This post details the debugging and resolution of packet drops encountered after switching from Foo-over-UDP encapsulation with virtual interfaces to IPVS native Generic UDP Encapsulation (GUE). The issue stemmed from negative header lengths during encapsulation, causing packets exceeding the Internet MTU to be dropped by the traffic directors. The root cause was identified through eBPF tracing with `pwru` and `perf-probe`, revealing packet drops within the `validate_xmit_skb` function called from `ip_vs_tunnel_xmit`. The fix involved addressing the underlying kernel behavior related to header length calculation during GUE encapsulation.
More from the Load Balancing UI and Feature Enhancements trail
- Load Balancing Monitor Groups: Multi-Service Health Checks for Resilient Applications
- Your IPs, your rules- enabling more efficient address space usage
- Simplify cloud routing and object storage configurations with Cloud Connector
- Eliminating hardware with Load Balancing and Cloudflare One
- Extending Private Network Load Balancing load balancing to Layer 4 with Spectrum