Latency Based Load Balancing

Spotify's load balancing strategies have evolved to address the challenges of distributing traffic efficiently across backend services. This post introduces the Expected Latency Selector (ELS) as a probabilistic load balancer that considers success latency, failure latency, and outstanding requests to dynamically adjust traffic distribution. ELS aims to outperform traditional methods like Round-Robin and Join the Shortest Queue (JSQ) by providing a more nuanced approach to machine selection, especially under high load or when machines exhibit varying performance or failure rates. Subsequent posts will detail the ELS formula and further benchmarks.