Scaling Slack’s Job Queue

This post details the first major architectural change to Slack's job queue system to address scaling and reliability issues. It introduces Kafka in front of Redis to prevent memory exhaustion and job loss during enqueue bursts. A new stateless service, Kafkagate (written in Go), handles enqueuing jobs to Kafka via an HTTP POST interface, prioritizing availability over consistency and optimizing for low latency by routing to the same Availability Zone. A second new stateless service, JQRelay (also in Go), relays jobs from Kafka to Redis, addressing data encoding quirks between Go and PHP/Hacklang JSON encoders and implementing self-configuration via Consul locks.