Debugging network stalls on Kubernetes

This post details the debugging process for network stalls observed in Kubernetes clusters. It outlines the reproduction of the issue using Vegeta and hping3, progressively simplifying the network path to isolate the problem. The analysis rules out application-level issues, kube-proxy NAT, and IPIP tunnels, pointing towards a problem within the Kubernetes node's network stack. The post introduces the concept of NIC receive queue hashing for TCP vs. ICMP packets as a key observation, suggesting a stall in processing incoming packets on the receive side of certain kube-node servers.