TechPrufer
Get started
Cloudflare
Cloudflare logoCloudflare · Feature trail

Fleet Management System

3 posts · Jul 2023 – Mar 2024

What this trail covers

Cloudflare's fleet management has evolved from manual YAML file management to a robust, in-house system called Zinc. Zinc models logical and physical infrastructure assets (servers, network devices, data centers) in a strongly-typed system, providing APIs and interfaces for efficient fleet management. It integrates with other systems for workload assignment, maintenance scheduling, repair management, and diagnostics reporting. Zinc offers a native web interface and CLI tooling for engineers, with features like a change request and approval system, node reservations, and comprehensive metadata collection. It complements Netbox for physical inventory management.

All Cloudflare trails
  1. Autonomous hardware diagnostics and recovery at scaleThis post introduces Phoenix, an autonomous diagnostics and recovery automation system for broken servers. Phoenix automatically discovers broken servers, performs diagnostics (out-of-band connectivity checks via IPMI and node acceptance tests using an INAT-image), recovers servers that pass diagnostics by re-provisioning, and re-enables them for production. It handles ticket updates and failure pinpointing without human intervention. The system incorporates an 'Error Budget' concept to limit excessive server failures and maintain empathy in automation. Visibility is provided through detailed logging and dashboards.
  2. Redefining fleet management at CloudflareThis post introduces Zinc, an in-house fleet management system built in Rust. It replaces manual YAML file management for workload and server provisioning with a strongly-typed data model for logical and physical infrastructure assets. Key features include workload management, least-privilege user accounts via Cloudflare Access, a change request and approval system, node reservations, and unified node metadata collection. Zinc integrates with Salt for configuration management, Netbox for DCIM, INAT for diagnostics, and a reboot management system. It provides a web interface and CLI tooling for engineers.
  3. How the Cloudflare global network optimizes for system reboots during low-traffic periodsThis post details the automation of determining maintenance windows for server reboots in Cloudflare's global network. It introduces the use of sinusoidal wave fitting on historical CPU traffic data to identify low-traffic periods, replacing a manual process. The implementation uses Python with scipy.optimize.curve_fit for the sine fitting, caches results in Consul KV, and includes fallback mechanisms and goodness-of-fit calculations for accuracy assessment.