Blogs›Cloudflare›Dynamic Task Scheduling with Nomad
Cloudflare has adopted HashiCorp Nomad as a dynamic task scheduling system to improve the availability and resource utilization of management services across its global edge data centers. This system ensures a desired number of service instances are reliably running, regardless of the underlying physical machine, and integrates with Consul for service discovery and Prometheus for observability. Challenges in deployment, such as initramfs rootfs compatibility and resource containment, have been a focus. This post details the implementation of Apache Airflow for automating data center expansions, replacing manual SOP steps with API calls and custom operators. The system handles failure, logging, notifications, and Jinja templating, with sensors for preconditions and human intervention. Reusable DAGs and branching logic are employed for complex workflows and scaling across hundreds of data centers.
1/27/2021
This post details the implementation of Apache Airflow for automating data center expansions, replacing manual SOP steps with API calls and custom operators. The system handles failure, logging, notifications, and Jinja templating, with sensors for preconditions and human intervention. Reusable DAGs and branching logic are employed for complex workflows and scaling across hundreds of data centers.
6/5/2020
This post introduces the adoption of HashiCorp Nomad for managing management services in Cloudflare's edge data centers. It details the reliability model for customer-facing and management services, the need for dynamic task scheduling, the reasons for choosing Nomad over Kubernetes (lightweight, dependencies, modular task drivers, open source), the deployment architecture (Nomad Servers and Clients across failure domains), job file templating with rack metadata constraints, service discovery via Consul integration, and observability using Prometheus and Grafana. It also highlights challenges faced, including initramfs rootfs issues resolved with kernel patches and resource containment via dedicated mount points.