Waiter: An Open-Source Distributed Auto-Scaler

This post introduces Waiter, a distributed auto-scaler designed for request-level scaling in service-oriented architectures. It addresses the challenge of anticipating traffic patterns by automatically scaling service instances based on incoming requests, rather than machine-level metrics. Waiter also supports service versioning, allowing multiple versions of a service to run concurrently and enabling graceful evolution without breaking downstream consumers. The system has been deployed in production at Two Sigma for over two years, handling significant traffic volumes.