Taming the tail utilization of ads inference at Meta scale

This post details Meta's efforts to tame the tail utilization of ads inference services. Key technical contributions include: leveraging the 'power of two choices' in randomized load balancing via ServiceRouter with polling for fresh load data, tuning Shard Manager placement load balancing configurations (load bands, thresholds, balancing frequency) to address spiky tail utilization, and incorporating memory bandwidth as a resource during replica placement in Shard Manager to mitigate CPU stalls caused by memory latency. The post also highlights the mismatch in expectations between ServiceRouter's host-level consolidated load counter and Shard Manager/ReplicaEstimator's assumption of equal replica load, leading to imbalance issues.