ELS: a latency-based load balancer, part 2

This post, part 2 of the ELS load balancer series, details the metrics used for load balancing (success latency, success rate, outstanding requests, failure latency, parallelism) and how they are measured using forward exponential decay. It explains the calculation of expected latency from the user's perspective, considering retries and client round-trip time, and how this expected latency is converted into a weight for probabilistic load distribution. It also introduces a tweak to the formula inspired by C3 for Cassandra to improve tail latency by penalizing machines with too many outstanding requests.