Reliable Cron Script Execution Service

3 posts · Dec 2017 – Sep 2023

What this trail covers

This thread tracks the evolution of Slack's system for executing cron scripts reliably at scale. Initial posts detail the challenges of a single-node cron setup with growing script numbers and data volumes, leading to reliability issues and maintenance burdens. This post details the design and implementation of a new, more robust service. The new system comprises a Golang 'Scheduled Job Conductor' running on Kubernetes (Bedrock), leveraging Slack's existing Job Queue for asynchronous compute, an

All Slack trails

Sep 2023Executing Cron Scripts Reliably At ScaleThis post details the design and implementation of a new reliable and scalable cron execution service. Key contributions include: 1. The 'Scheduled Job Conductor' Golang service using a cron library and Kubernetes Leader Election for high availability, offloading compute to the Job Queue. 2. Integration with Slack's Job Queue for asynchronous, high-volume job execution. 3. Utilization of a Vitess table for job deduplication (replacing flock) and state tracking, enabling monitoring and preventing concurrent script runs. The post also discusses the rationale for a single scheduling leader and offloading compute.
Feb 2022Building Self-driving Kafka clusters using open source componentsThis post details the development and operationalization of self-driving Kafka clusters at Slack. It covers the migration to Kafka 2.0.1, standardizing clusters under a single version managed by one team. Key components include Chef and Terraform for build, release, provisioning, and deployment, with an internal fork of Kafka for reproducible builds. Zookeeper provisioning was automated using Consul KV and ASGs. To address hot spotting and instability, partition counts were made multiples of broker counts, consumer counts were smoothed, and replication bandwidth was limited. Cruise Control was integrated for automated cluster rebalancing. Chaos engineering was employed to test failure modes, leading to the adoption of jumbo frames and the use of separate Zookeeper clusters per Kafka cluster. Self-serve tools like Kafka Manager and a Kafka offset exporter were deployed to provide visibility and simplify operations for consumer teams. The upgrade process for Kafka clusters was redesigned to involve cutting over to a new cluster rather than in-place upgrades.
Dec 2017Scaling Slack’s Job QueueThis post details the first major architectural change to Slack's job queue system to address scaling and reliability issues. It introduces Kafka in front of Redis to prevent memory exhaustion and job loss during enqueue bursts. A new stateless service, Kafkagate (written in Go), handles enqueuing jobs to Kafka via an HTTP POST interface, prioritizing availability over consistency and optimizing for low latency by routing to the same Availability Zone. A second new stateless service, JQRelay (also in Go), relays jobs from Kafka to Redis, addressing data encoding quirks between Go and PHP/Hacklang JSON encoders and implementing self-configuration via Consul locks.
Advertisement