TechPrufer
Get started
PlanetScale
PlanetScale logoPlanetScale · Feature trail

Sidekiq Job Control

2 posts · Feb 2022 – Aug 2022

What this trail covers

PlanetScale has introduced a mechanism to control Sidekiq jobs in production without requiring code deploys. This is achieved by integrating Sidekiq client middleware with a feature flagging system (Flipper). This allows for immediate disabling of specific job classes by enabling a corresponding feature flag. This capability enhances operational flexibility and reduces the risk associated with urgent hotfixes. The system also incorporates strategies for self-healing background jobs by storing state in the database, using scheduler jobs to re-queue missed tasks, implementing bulk scheduling for efficiency, adding jitter to spread out job execution, and handling job uniqueness through database locks and Sidekiq's unique jobs feature.

All PlanetScale trails
  1. How to kill Sidekiq jobs in Ruby on RailsThis post introduces a new Sidekiq client middleware that integrates with Flipper to disable specific Sidekiq jobs in production. The middleware checks if a feature flag named 'disable_<job_class_name>' is enabled. If it is, the job is prevented from being pushed to Redis. The post provides the middleware code, instructions on how to configure it in Rails applications, and an example of how to disable an 'InvoiceJob' by enabling the 'disable_invoice_job' flag.
  2. How we made PlanetScale’s background jobs self-healingThis post details how PlanetScale made its background jobs self-healing and resilient. Key contributions include: 1. Implementing scheduler jobs that run on a cron to re-queue missed tasks if queues are lost. 2. Storing job state in the database (e.g., 'pending' state for database creation) to enable recovery. 3. Integrating Sidekiq client middleware with Flipper for feature flags to disable scheduled jobs during incidents. 4. Introducing bulk scheduling (`perform_bulk`) to improve efficiency for large numbers of jobs. 5. Adding `perform_with_jitter` to spread job execution over time and avoid overwhelming external APIs. 6. Strategies for handling job uniqueness, including exiting quickly if state is no longer valid, using database locks, and leveraging Sidekiq Enterprise's unique jobs feature.