Not just another network latency issue: How we unraveled a series of hidden bottlenecks | Datadog

This post details the investigation and resolution of a series of network latency issues impacting the usage estimation service. The investigation uncovered bottlenecks related to Envoy proxy CPU utilization during deployments, leading to TCP retransmits. It also identified a Linux kernel bug affecting the Elastic Network Adapter (ENA) on AWS, which caused traffic to be mapped to a single transmit queue instead of distributing it across multiple queues. The solutions involved allocating more CPU to Envoy and applying a hotfix for the Linux kernel bug, ultimately improving latency during both normal operations and deployments.