TechPrufer
Get started
Cloudflare
Cloudflare logoCloudflare · Feature trail

Baseboard Management Controller (BMC) Security & Firmware

1 post · Nov 2020 – Oct 2024

What this trail covers

Cloudflare is enhancing the security and manageability of its server fleet by adopting OpenBMC, an open-source firmware for Baseboard Management Controllers (BMCs). This move aims to provide greater transparency, faster patching, and more control over firmware, including the integration of TLS certificates and fine-grained credential management. Furthermore, Cloudflare is extending its secure boot capabilities to the BMC itself, leveraging hardware root-of-trust and security co-processors to validate firmware integrity during boot. This post details how Cloudflare customized and extended OpenBMC to improve server boot reliability and diagnostics by better monitoring boot-up processes and addressing race conditions between the BMC and UEFI during subsystem initialization and memory module detection.

All Cloudflare trails
  1. Is this thing on? Using OpenBMC and ACPI power states for reliable server bootThis post details Cloudflare's customization and extension of OpenBMC firmware to address server boot failures. It describes how race conditions between the BMC and UEFI during subsystem initialization (e.g., platform initialization phase) and memory module detection (SPD access over I2C) led to boot failures and missing memory modules. The post explains the role of the BMC in server management, the advantages of using OpenBMC for transparency and customization, and the specific issues encountered: hosts not booting despite chassis power being on (indicated by SEL and CPU_POWER sensor discrepancies) and memory modules being missed due to simultaneous access attempts by UEFI and BMC. It introduces ACPI power states (S5_G2, S0_G0) and the System Event Log (SEL) as diagnostic tools, and highlights the need for better communication and synchronization between BMC and UEFI firmware.
  2. How we used OpenBMC to support AI inference on GPUs around the worldThis post details the specific application of OpenBMC for managing GPU thermals and communication in AI inference servers. It describes the process of tuning PID controllers for fan speed to maintain GPU temperatures below 95°C, achieving a stable 65°C with an additional fan. The post also explains how to establish communication with GPUs via SMBus, read temperature sensor data using the TMP75 driver, and leverage OpenBMC's Entity-Manager to expose this data via IPMI and Redfish, including inventory information from EEPROMs.
  3. Introducing the Project Argus Datacenter-ready Secure Control Module design specificationThis post introduces Cloudflare's partnership with Lenovo to design a Datacenter-ready Secure Control Module (DC-SCM) for their next-generation servers, named Project Argus. It details the benefits of this modular design, including PCB cost reduction by decoupling the Baseboard Management Controller (BMC) from the Host Processor Module (HPM), and enabling flexibility for management and security upgrades. The post highlights the use of OpenBMC for unified firmware development and the adherence to the DC-SCM 2.0 specification. Project Argus specifically details implementation choices for form factor, locking mechanisms, BMC chip (ASPEED AST2600), Hardware Root of Trust (ASPEED AST1060), and CPLD (Lattice MachXO3D). The design specification and reference files have been contributed to the Open Compute Project.
  4. Deploying firmware at Cloudflare-scale: updating thousands of servers in more than 285 citiesThis post details Cloudflare's automated firmware deployment process for its global fleet of servers. It outlines the use of iPXE and UEFI shell for BIOS updates, and a minimal Linux environment with initial ramdisks for BMC and NIC firmware updates. The process leverages UEFI variables for firmware version tracking and systemd for executing flashing scripts. The goal is to achieve fully automated, non-interruptive firmware updates across thousands of servers in hundreds of cities.
  5. Cloudflare’s approach to handling BMC vulnerabilitiesThis post details Cloudflare's response to critical BMC vulnerabilities, including patching firmware, reducing exposure of BMC interfaces (both remote and local), enforcing strong password policies, and enhancing BMC logging. It also announces the strategic shift towards OpenBMC for greater transparency and control, and the extension of secure boot capabilities to the BMC itself using hardware root-of-trust.
  7. Anchoring Trust: A Hardware Secure Boot StoryThis post details Cloudflare's implementation of a hardware root of trust (HRoT) using AMD's Platform Secure Boot (PSB) for their servers. It explains the boot process from power-on through UEFI and Linux kernel loading, highlighting the limitations of firmware-based secure boot. The post then describes AMD's PSB, which anchors trust in immutable hardware via the Platform Security Processor (PSP). It details the build process, including the Public Key Infrastructure (PKI) for signing BIOS segments with AMD and ODM keys, and the structure of the Certificate Signing Request (CSR) with specific fields for platform identification and security fuse bits.