TechPrufer
Sign inGet started

Blogs›Cloudflare›Fleet Management System

Fleet Management System

Fleet Management System

3
posts
2023–2024

Cloudflare's fleet management has evolved from manual YAML file management to a robust, in-house system called Zinc. Zinc models logical and physical infrastructure assets (servers, network devices, data centers) in a strongly-typed system, providing APIs and interfaces for efficient fleet management. It integrates with other systems for workload assignment, maintenance scheduling, repair management, and diagnostics. Zinc offers a native web interface and CLI tooling for engineers, with features for automated maintenance window determination using sinusoidal wave fitting on traffic patterns to minimize customer impact during reboots.

2024

Autonomous hardware diagnostics and recovery at scale

3/25/2024

Introduced Phoenix, an autonomous diagnostics and recovery automation system for managing broken servers at scale. Phoenix automates the discovery, diagnostics (including out-of-band connectivity checks and INAT), and recovery (re-provisioning and re-enabling) of faulty hardware without human intervention. It also implements an Error Budget mechanism to manage the risk of repeated server failures. Enhanced visibility through detailed logging and dashboarding of automation activities.

Redefining fleet management at Cloudflare

3/19/2024

Introduced Zinc, an in-house Rust-based infrastructure system for managing logical and physical fleet assets. Zinc models servers, network devices, and data centers, providing APIs for workload assignment, maintenance, repairs, and diagnostics. It features a change request system, node reservations, and integrates with Netbox for physical inventory. Automations for repairs, diagnostics (via INAT), maintenance windows, and reboots are now orchestrated through Zinc, improving engineer productivity via web and CLI interfaces.

2023

How the Cloudflare global network optimizes for system reboots during low-traffic periods

7/12/2023

This post details the automation of determining maintenance windows for server reboots in Cloudflare's global network. It introduces the use of sinusoidal wave fitting on historical CPU traffic data to identify low-traffic periods, replacing a manual process. The implementation uses Python with scipy.optimize.curve_fit for the sine fitting, caches results in Consul KV, and includes fallback mechanisms and goodness-of-fit calculations for accuracy assessment.