How we scaled async workload processing at GitLab.com using Sidekiq

This post details the evolution of GitLab's Sidekiq background job processing architecture. It describes the challenges faced with a previous many-to-many relationship between jobs and queues, leading to availability issues and inefficient resource utilization. The post outlines the migration to a one-to-one relationship between jobs and Sidekiq roles, where all worker threads process all jobs, and introduces queue selectors based on workload classification (urgency, external dependencies, CPU/memory bound). It also discusses the impact of Redis BRPOP command contention and the fix implemented in Redis 6, leading to a reduced Sidekiq fleet size and cost.