What we learned about configuring Sidekiq from GitLab.com

This post details the configuration of Sidekiq on GitLab.com to address Redis CPU saturation. It explains the historical one-queue-per-worker approach and its limitations with a large number of queues. The solution implemented is a one-queue-per-shard model using Sidekiq routing rules, which significantly reduces the overhead on Redis by simplifying BRPOP operations. The post also touches on concurrency and the GIL in Ruby, and the experiments conducted to validate the chosen approach.