Autoscaling Pub/Sub Consumers

This post details the implementation of autoscaling for Pub/Sub consumers within Spotify's Event Delivery system. It describes the architecture using dedicated Pub/Sub topics per event type, consumed by independent ETL processes. The Pub/Sub Consumer microservice, built with Apollo, Docker, and deployed via Helios, is managed by regional instance groups with CPU-based autoscaling. Key challenges addressed include handling unhealthy machines (e.g., Docker zombie states) by implementing a custom HTTP healthchecker that directly destroys unresponsive instances, and fine-tuning CPU usage to prevent autoscaler misbehavior during downstream service issues or event backlogs by employing exponential backoffs. The post also discusses future plans to explore Kubernetes for improved bin packing and resource optimization.