How we scaled async workload processing at GitLab.com using Sidekiq

What this post added

This post details the evolution of GitLab's Sidekiq architecture to scale asynchronous workload processing. It describes the shift from a many-to-many relationship between Sidekiq jobs and Sidekiq roles to a one-to-one relationship, where each job executes on exactly one Sidekiq role. It also introduces the concept of queue selectors based on workload classification (urgency, external dependencies, CPU/memory bound) to ensure predictable job execution and meet SLAs. The post highlights performance improvements achieved by optimizing Redis BRPOP commands and reducing the Sidekiq fleet size.