Troubleshooting Unusual AWS ELB 5XX Error

This post details the investigation into ELB 5XX errors triggered by Datadog alerts. It outlines the system architecture, analyzes AWS ELB metrics (SurgeQueue, SpillOver, Latency, HTTPCode_ELB_5XX, HTTPCode_Backend_5XX, Healthy/Unhealthy Instances), and confirms that backend instances were healthy but requests were not reaching them. The investigation ruled out ELB misconfiguration and DNS issues, and focused on client-side (Gothena) behavior. Tests using `dig`, `netstat`, `tcpdump`, `curl`, and programs in Go, Python, and Ruby revealed that Go's default behavior of reusing connections led to a single DNS query per connection, while other languages made multiple queries per request. This difference in DNS query behavior was identified as the root cause of the load imbalance across ELB nodes, leading to ELB 5XX errors during peak hours.