TechPrufer
Get started
Cloudflare
Cloudflare logoCloudflare · Feature trail

Dynamic Task Scheduling with Nomad

2 posts · Jun 2020 – Jan 2021

What this trail covers

Cloudflare has adopted HashiCorp Nomad as a dynamic task scheduling system to improve the availability and resource utilization of management services across its global edge data centers. This system ensures a desired number of service instances are reliably running, regardless of the underlying physical machine, and integrates with Consul for service discovery and Prometheus for observability. Challenges in deployment, such as initramfs rootfs compatibility and resource usage containment, have been addressed through upstream contributions and custom solutions. Nomad job files are templated and managed via configuration management tools, with rack metadata used to ensure distinct failure domains for service instances. Mutual TLS authentication is used to secure API access.

All Cloudflare trails
  1. Automating data center expansions with AirflowThis post details the implementation of Apache Airflow for automating data center expansions, replacing manual SOP steps with API calls and custom operators. The system handles failure, logging, notifications, and Jinja templating, with sensors for preconditions and human intervention. Reusable DAGs and branching logic are employed for complex workflows and scaling across hundreds of data centers.
  2. How we use HashiCorp NomadThis post details Cloudflare's adoption of HashiCorp Nomad for managing 'management services' (services not in the direct request path but critical for data center operation) across its edge data centers. It explains the need for dynamic task scheduling to ensure reliability and cost-effectiveness, contrasting it with suboptimal options. The choice of Nomad over Kubernetes is justified by its lightweight nature, Consul integration, modular task drivers, and Go implementation. The deployment architecture, including Nomad Server and Client instances, failure domain considerations, and job file templating with rack constraints, is described. Service discovery via Consul and observability with Prometheus and Grafana are covered. Challenges encountered, such as initramfs rootfs compatibility with the exec driver and resource usage containment, are discussed, along with the solutions implemented, including upstream contributions to Nomad and the Linux kernel, and dedicated mount points for Nomad data directories.