It’s always DNS . . . except when it’s not: A deep dive through gRPC, Kubernetes, and AWS networking | Datadog

What this post added

This post details the investigation and resolution of an incident where DNS errors were initially suspected but ultimately traced to deeper networking issues within a Kubernetes environment on AWS. It covers the analysis of NodeLocal DNSCache behavior, including concurrency limits and health check failures, and the correlation of these issues with increased TCP retransmits. The post also explores the interaction between Cilium, Kubernetes, and AWS networking, and the impact of reverse path filtering on packet routing.