TechPrufer
Get started
Cloudflare
Cloudflare logoCloudflare · Feature trail

Hardware Infrastructure & Performance Optimization

66 posts · Jul 2013 – Jun 2026

What this trail covers

Cloudflare's hardware infrastructure and performance optimization efforts have evolved to include strategic adoption of new server generations (Gen 12) featuring increased CPU core counts (AMD EPYC 9684X Genoa-X), improved memory capacity and bandwidth (DDR5), faster NVMe storage, and enhanced networking capabilities (25 GbE NICs). This evolution focuses on maximizing compute density and performance per watt to handle growing traffic demands and reduce data center expansion costs. The Gen 12 servers offer a 145% performance improvement and 63% power efficiency improvement over Gen 11 servers. The design also incorporates hardware root of trust (HRoT) and modular baseboard management controllers (OCP DC-SCM) for enhanced security and vendor-agnosticism, supporting openBMC. A multi-vendor strategy is employed to mitigate supply chain risks.

All Cloudflare trails
  1. How we reduced core unit boot time from hours to minutesThis post details an issue with UEFI firmware updates causing extremely long boot times on core servers. The root cause was identified as an over-eager linear search through all available network boot interfaces (IPv4 HTTPS, IPv4 iPXE, IPv6 HTTPS), each taking approximately five minutes to time out. This resulted in a total boot time of nearly four hours instead of minutes. The solution involved restructuring the boot automation workflow to declare the correct network boot interface order upfront, significantly reducing the time spent probing. Obstacles included legacy UEFI versions, vendor-imposed immutable boot order settings, and inconsistent network interface string formats from NIC vendors. These were overcome by implementing state validation, collaborating with vendors for BIOS updates, and enhancing configuration tools to handle varied string formats. The result is a dynamic system where boot times are reduced to minutes, with no manual BIOS interactions required.
  2. A one-line Kubernetes fix that saved 600 hours a yearThis post details a specific performance bottleneck encountered with Kubernetes persistent volumes, specifically the `fsGroup` setting which caused slow pod restarts due to recursive ownership changes on large filesystems. The post describes the diagnostic process, from observing pod events to delving into kubelet logs and finally identifying the root cause in the PV mounting process. The solution, a one-line change to `fsGroupChangePolicy` to `OnRootMismatch`, dramatically reduced restart times and reclaimed significant engineering hours, highlighting how default Kubernetes settings can become performance bottlenecks as data scales.
  3. Inside Gen 13- how we built our most powerful server yetThis post details the engineering rationale behind the selection of components for Cloudflare's Gen 13 server hardware. It covers the evaluation and choice of the AMD EPYC™ 9965 Turin processor, highlighting the trade-off of L3 cache for increased core count due to the FL2 software stack's reduced cache dependency. It also details the memory configuration (12x 64GB DDR5-6400 ECC RDIMMs) for optimal bandwidth and capacity, the move to PCIe 5.0 NVMe drives for increased storage capacity and bandwidth, and the adoption of dual 100 GbE network interface cards. The post also discusses the rationale for selecting the 9965 processor based on performance per watt, TCO, operational simplicity, and forward compatibility.
  4. Launching Cloudflare’s Gen 13 servers- trading cache for cores for 2x edge compute performanceThis post details the transition to Cloudflare's Gen 13 server fleet, featuring AMD EPYC Turin processors. It highlights the challenge of reduced L3 cache per core compared to Gen 12 servers and the resulting latency issues when running the legacy FL1 request handling layer. The post describes how performance counter analysis identified cache miss rates and memory fetch latency as key problems. It also details incremental gains achieved through hardware tuning and PQOS, but emphasizes that the true unlock came from the ongoing FL2 rewrite in Rust. FL2's cleaner architecture and leaner memory access patterns removed the dependency on large L3 caches, allowing Gen 13's higher core counts to deliver up to 2x throughput gains and a 50% boost in performance per watt compared to Gen 12, all while maintaining latency SLAs.
  5. So long, and thanks for all the fish- how to escape the Linux networking stackThis post details Cloudflare's efforts to push beyond the limitations of the Linux networking stack for specific high-performance use cases, particularly in managing soft-unicast IP addresses. It introduces 'fish' (SLATFATF), a custom service for proxying IP packets using soft-unicast address space, and explores the challenges of coexisting packet forwarding with socket binding. The post details the investigation into Netfilter/conntrack interactions and the development of solutions using Netlink and, more effectively, kernel features like TCP_REPAIR and TCP Fast Open to achieve precise control over connection state and avoid collisions, thereby enabling more efficient and reliable network operations for services like WARP.
  7. A deep dive into BPF LPM trie performance and optimizationThis post dives deep into the performance of BPF LPM tries, a critical data structure used for IP and IP+Port matching in network packet routing and firewall rules. It explains the mechanics of tries, highlights the shortcomings of the current BPF LPM trie implementation (limited branching, lack of level compression), and presents benchmark data showing performance degradation with increasing entry counts. The post identifies specific bottlenecks like slow lookups, updates, deletes, and particularly map freeing, which can lead to production issues like soft lockups and packet loss, setting the stage for future optimization efforts.
  8. How we found a bug in Go's arm64 compilerThis post details Cloudflare's discovery and resolution of a race condition bug in Go's arm64 compiler. The bug, identified through extensive monitoring of their global network, caused sporadic panics and segmentation faults during stack unwinding. Cloudflare's investigation traced the issue to the interaction between Go's asynchronous preemption mechanism and the `github.com/vishvananda/netlink` library, leading to the identification of a specific bug in the Go compiler's generated code for arm64. Cloudflare's contribution involved deep debugging, correlation with existing Go issues, and ultimately driving towards a root cause analysis that benefits the broader Go ecosystem.
  9. Cloudflare just got faster and more secure, powered by RustThis post details a significant upgrade to Cloudflare's core network processing system, codenamed 'FL2', which replaces the legacy NGINX/PHP/LuaJIT-based FL1. The new system is built in Rust using the Oxy framework, leading to a 10ms reduction in median response time and a 25% performance boost. Key improvements include enhanced security, faster development cycles, and robust graceful restart capabilities via systemd socket activation and the 'shellflip' coordination mechanism. The modular architecture of FL2 enforces strict separation of concerns, preventing the issues of complexity and latency that plagued FL1.
  10. Cloudflare’s commitment to CISA Secure-By-Design pledge: delivering new kernels, fasterThis post details Cloudflare's commitment to the CISA Secure-by-Design pledge by focusing on the secure and timely patching of the Linux kernel. It outlines the rigorous process for kernel updates, from upstream release to production deployment, including internal testing, phased rollouts, and the use of employee traffic for validation. The post also highlights the development of automated systems like 'unimog' for traffic management during reboots and 'Reboau' for control plane kernel maintenance, ensuring consistent security across the entire infrastructure.
  11. Is this thing on? Using OpenBMC and ACPI power states for reliable server bootThis post details the customization and extension of OpenBMC firmware to improve server boot reliability and diagnostics. It addresses issues where servers would not boot despite the BMC indicating power on, and where memory modules were randomly missed. The solution involves better management of BMC and UEFI interactions during the boot process, specifically by leveraging ACPI power states (S5_G2, S0_G0) for more accurate power status reporting and by analyzing i2c bus contention between the BMC and UEFI when accessing memory module SPD data. This allows for more reliable server boot-up and better diagnostics.
  12. Analysis of the EPYC 145% performance gain in Cloudflare Gen 12 serversThis post details the sensitivity analysis of Cloudflare's primary workload on AMD EPYC 4th generation processors (Genoa, Bergamo, Genoa-X). It quantifies performance gains from core count (linear scaling), TDP (incremental improvements up to 400W), and frequency. Crucially, it highlights the significant impact of L3 cache size, with the Genoa-X variant (9684X) showing a 2.75x performance multiplier in lab simulations over the Milan 7713 (Gen 11), suggesting Cloudflare's software stack is highly sensitive to L3 cache. The analysis uses the internal cf_benchmark tool and a performance simulation tool to evaluate CPU candidates, providing concrete metrics and trade-offs.
  13. Leveraging Kubernetes virtual machines at Cloudflare with KubeVirtThis post introduces Cloudflare's use of KubeVirt to run virtualized Kubernetes clusters on their bare metal infrastructure. This allows teams to run workloads requiring deep kernel integration (like Docker daemons or simulating network environments) alongside standard containerized applications within a secure, multi-tenant Kubernetes environment. This enhances the flexibility and efficiency of their hardware infrastructure by enabling a wider range of workloads on shared resources.
  14. Thermal design supporting Gen 12 hardware: cool, efficient and reliableThis post details the thermal design considerations for Cloudflare's Gen 12 server hardware, focusing on standardization, system layout, and flexibility. Key technical contributions include: defining a thermal specification with requirements for ambient conditions (5-40°C), thermal margins, fan failure support (requiring operation with one fan failure), and power efficiency in cooling; optimizing system layout with larger axial fans (80mm) and Extended Volume Air Cooled (EVAC) heatsinks for improved airflow and heat transfer; and designing for hardware flexibility to support up to 2x single-wide GPU add-in cards. It also discusses rack-level thermal considerations, such as cold/hot aisle management and airflow resistance.
  15. Cloudflare’s 12th Generation servers — 145% more performant and 63% more efficientThis post details the introduction of Cloudflare's Gen 12 servers, powered by AMD EPYC 9684X (Genoa-X) processors. Key technical contributions include: a 145% increase in performance (requests per second) and a 63% improvement in power efficiency (RPS per watt) compared to Gen 11 servers. Specific hardware details are provided: 96-core Genoa-X CPU with 1152MB L3 cache, 384GB DDR5 RAM, 16TB NVMe storage, dual 25 GbE NICs, and 800W PSUs. The design choice to double server chassis height (2U1N) for improved cooling efficiency is highlighted, reducing system power consumption to 600W. The integration of hardware root of trust (HRoT) and modular OCP DC-SCM for baseboard management is also detailed, supporting openBMC.
  16. Major data center power failure (again): Cloudflare Code Orange testedThis post details how Cloudflare's 'Code Orange' preparedness and investments in Highly Available (HA) topologies significantly improved its resilience to a major data center power failure. It highlights the rapid, automated failover of control plane services (APIs, Dashboard) within minutes, a stark contrast to the hours/days of downtime experienced in a previous similar incident. It also discusses improvements in Logpush infrastructure, the shift of Zero Trust functionality to the global network, and the faster data center recovery time, all demonstrating advancements in hardware infrastructure resilience and operational efficiency.
  17. Autonomous hardware diagnostics and recovery at scaleThis post introduces 'Phoenix,' an autonomous diagnostics and recovery automation system for Cloudflare's global hardware infrastructure. Phoenix automates the detection, diagnosis, and re-provisioning of broken servers, significantly reducing manual effort and improving operational efficiency. It includes out-of-band connectivity checks, integrated node acceptance tests (INAT), and an 'Error Budget' concept to balance automation with empathy for potential recurring hardware failures.
  18. Redefining fleet management at CloudflareThis post introduces 'Zinc,' an in-house infrastructure system built in Rust for comprehensive fleet management. Zinc models logical and physical infrastructure assets (servers, network devices, data centers) and provides APIs and interfaces for efficient fleet management, including workload assignment, maintenance scheduling, and repair cycle management. It automates processes like repair ticket generation and maintenance window orchestration, integrating with systems like Netbox for physical inventory and INAT for diagnostics. Zinc also enhances engineer productivity through a web interface and CLI for complex queries and changes, ensuring scoped and authenticated access to production infrastructure. This represents a significant evolution in managing Cloudflare's growing hardware infrastructure at scale.
  19. Linux kernel security tunables everyone should consider adoptingThis post details Cloudflare's implementation of secure boot at the Linux kernel level, focusing on enforcing signed kernel modules. It demonstrates how this prevents unauthorized code injection by requiring modules to have valid signatures, thereby enhancing system integrity and security. The post contrasts this with less secure configurations found in standard distributions and highlights the importance of this measure for Cloudflare's production environments.
  20. How we used OpenBMC to support AI inference on GPUs around the worldThis post details the use of OpenBMC to manage and optimize thermal and power consumption for GPU-equipped servers, enabling AI inference workloads. It describes the process of fine-tuning fan PID controllers by adjusting proportional, integral, and derivative gains to maintain GPU temperatures below critical thresholds. It also covers establishing communication with GPU sensors via SMBus and sysfs, and exposing this data through IPMI and Redfish using OpenBMC's Entity-Manager.
  21. Cloudflare Gen 12 Server: bigger, better, cooler in a 2U1N form factorThis post details the design decisions behind Cloudflare's Gen 12 compute servers, focusing on the adoption of a 2U form factor to address the increasing Thermal Design Point (TDP) of next-generation x86 CPUs (AMD Genoa/Bergamo/Genoa-X and Intel Sapphire Rapids). It explains the thermal limitations of the previous 1U form factor, the performance benefits of higher core counts and TDPs, and how the 2U design with larger fans and heatsinks allows for better cooling, full utilization of rack power budgets, and the integration of PCIe accelerators/GPUs, ultimately improving rack density and TCO.
  22. Introducing the Project Argus Datacenter-ready Secure Control Module design specificationIntroduced Project Argus, a Datacenter-ready Secure Control Module (DC-SCM) design specification developed in partnership with Lenovo. This design decouples the Baseboard Management Controller (BMC) and its related components from the main server motherboard, enabling independent development, cost reduction through specialized PCBs, and unified firmware development with OpenBMC. Project Argus adheres to the DC-SCM 2.0 specification and includes contributions to the Open Compute Project (OCP) database.
  23. Virtual networking 101: bridging the gap to understanding TAPThis post details the intricacies of using Linux tap devices for high-performance networking in virtualized environments, specifically within the context of Firecracker MicroVMs. It provides a comprehensive guide to the tap device API, including setup, flags like IFF_VNET_HDR and IFF_MULTI_QUEUE, buffer size management, and offloads such as checksumming and segmentation. This contributes to the broader theme of hardware infrastructure optimization by focusing on maximizing network performance at the virtualized interface level, a critical component for efficient VM operation.
  24. How Cloudflare is staying ahead of the AMD vulnerability known as “Zenbleed”This post details Cloudflare's response to the 'Zenbleed' vulnerability (CVE-2023-20593) affecting AMD Zen 2 processors. It explains the vulnerability's mechanism, which involves speculative execution and improper clearing of AVX registers, allowing sensitive data leakage. Cloudflare is mitigating this by patching its entire fleet of impacted servers with AMD's microcode update, which modifies Model Specific Registers to prevent speculative execution of instructions like `vzeroupper`. The post also provides background on CPU architecture, ALUs, registers, and AVX instructions, and discusses the attack's mechanics and initial CVSS score.
  25. How the Cloudflare global network optimizes for system reboots during low-traffic periodsThis post details the automation of determining maintenance windows for system reboots across Cloudflare's global network. It introduces a novel approach using sinusoidal wave fitting on CPU traffic patterns to dynamically identify low-traffic periods, replacing a manual and toilsome process. The post explains the mathematical theory behind sine fitting, its implementation in Python, and how it's integrated into Cloudflare's reboot system with caching and pre-fetch mechanisms. It also discusses accuracy metrics like goodness of fit (chi-squared) to handle edge cases and ensure reliable automated maintenance scheduling.
  26. Unbounded memory usage by TCP for receive buffers, and how we fixed itThis post details a specific, critical issue discovered in the Linux kernel's TCP receive buffer memory management. It explains how certain conditions can lead to unbounded memory allocation, impacting overall server performance and causing packet loss. The post outlines the root cause, provides a reproducible lab setup, and discusses the implications for Cloudflare's production network, highlighting the need for kernel-level fixes to maintain optimal performance and stability.
  27. How Pingora keeps countThis post introduces `pingora-limits`, a new library within Cloudflare's in-house proxy, Pingora. It addresses the need for efficient counting of inflight events and estimation of event rates to protect infrastructure. The library utilizes the Count-Min Sketch algorithm, offering significant improvements in space efficiency (O(log n)) and performance (lock-free, O(1) per event) compared to traditional hash table approaches, especially at Cloudflare's scale of millions of requests per second across tens of thousands of machines.
  28. The Linux Crypto API for user applicationsThis post introduces the Linux Kernel Crypto API as a mechanism for user applications to leverage hardware-accelerated cryptographic operations. It details how applications can interact with the kernel's crypto modules, explores the performance implications of using the API compared to user-space libraries like OpenSSL, and highlights the benefits of offloading encryption/decryption to the kernel for enhanced security and performance, particularly when integrated with kernel key retention services.
  29. DDR4 memory organization and how it affects memory bandwidthThis post details the impact of DDR4 memory organization, specifically rank and organization (e.g., 1Rx8, 2Rx4), on memory bandwidth performance. It explains memory rank as sets of DRAM chips on a module, dual-rank and quad-rank modules doubling or quadrupling capacity and using rank interleaving for reduced response time. It also covers memory depth and width, how memory device width (x4, x8, x16) affects the number of chips per rank and module capacity, and the relationship between page sizes and timing parameters like tFAW, which is influenced by memory device width and DDR4 speed.
  30. Deploying firmware at Cloudflare-scale: updating thousands of servers in more than 285 citiesThis post details Cloudflare's automated firmware deployment system for its global server fleet. It leverages iPXE for BIOS/UEFI updates via UEFI shell applications and a minimal Linux environment with initial ramdisks for BMC/NIC firmware updates. The system coordinates updates during scheduled reboots, ensuring no production interruption. It also covers the use of UEFI variables for storing firmware versions and the integration with systemd for executing flashing scripts.
  31. ROFL with a LOL: rewriting an NGINX module in RustThis post details the rewrite of the cf-html NGINX module into a new Rust-based module called ROFL. This initiative addresses memory safety concerns stemming from the original C implementation (highlighted by the Cloudbleed incident) and improves performance and maintainability. It showcases the technical challenges and solutions involved in writing an NGINX module in Rust using Bindgen for FFI bindings and dynamic module loading, demonstrating a shift towards safer and more modern languages for core infrastructure components.
  32. CVE-2022-47929: traffic control noqueue no problem?This post details the discovery and analysis of a kernel bug (CVE-2022-47929) related to Linux Traffic Control's 'noqueue' discipline when used with classful qdiscs, exacerbated by USER namespaces. Cloudflare engineers identified how this vulnerability could lead to an unprivileged denial-of-service attack and contributed a fix by disallowing 'noqueue' for qdisc classes, enhancing the security and stability of the underlying infrastructure.
  33. Armed to Boot: an enhancement to Arm's Secure Boot chainThis post details Cloudflare's implementation of hardware secure boot for Arm servers, building upon previous work with AMD. It introduces the Arm Trusted Firmware Secure Boot (TBBR) process and highlights its limitations for server environments. The post then describes Cloudflare's partnership with Ampere to develop and implement 'Single Domain Secure Boot' (SDSB) for Ampere Altra Max CPUs. SDSB enhances the boot chain's integrity by incorporating a customer's public key hash into the SoC's eFuse, allowing for end-to-end verification of UEFI firmware.
  34. A debugging story: corrupt packets in AF_XDP; a kernel bug or user error?This post details a deep dive into debugging a critical issue with AF_XDP, a high-performance networking API. It highlights the process of identifying and resolving a bug that caused packet corruption due to descriptors being used twice across different rings. The contribution lies in demonstrating Cloudflare's commitment to optimizing and ensuring the reliability of its low-level network infrastructure, including the meticulous debugging of kernel interactions and user-space applications to maintain network performance and security.
  35. The Linux Kernel Key Retention Service and why you should use it in your next applicationThis post introduces the Linux Kernel Key Retention Service as a novel approach to protecting cryptographic keys. It explains how memory access violations can lead to key leaks and how the kernel's key retention service can mitigate these risks by storing keys in a separate, isolated kernel address space, effectively acting as a secure agent. This adds a new dimension to Cloudflare's security infrastructure by leveraging kernel-level features for enhanced key management and protection.
  36. Assembly within! BPF tail calls on x86 and ARMThis post delves into the low-level implementation of tail call elimination in BPF (Berkeley Packet Filter) for x86 and ARM architectures. It explains how compilers optimize recursive functions by transforming tail calls into jumps, saving stack space and improving performance. The post then details how Cloudflare leverages BPF tail calls for its XDP-based packet processing pipeline, enabling modularity and efficiency. It also highlights the nuances of BPF tail calls, particularly their interaction with BPF-to-BPF function calls and the challenges faced on arm64, contributing to the ongoing optimization of Cloudflare's network edge processing capabilities.
  37. Hertzbleed explainedThis post details the Hertzbleed side-channel attack, which exploits data-dependent CPU power consumption variations caused by dynamic voltage and frequency scaling (DVFS). It explains how DVFS adjusts CPU frequencies based on workload power consumption, which is influenced by factors like Hamming weight and Hamming distance. This data-dependent frequency scaling can lead to different execution times for the same program, creating a vulnerability for cryptographic software, particularly for SIKE decapsulation servers.
  38. Cloudflare’s approach to handling BMC vulnerabilitiesThis post details Cloudflare's proactive response to critical vulnerabilities found in Baseboard Management Controllers (BMCs). It outlines the background of BMCs, the specific vulnerabilities discovered (related to Pantsdown and USBAnywhere), and the immediate mitigations implemented, including firmware updates, network segmentation, reduced interface exposure, and strong password policies. Crucially, it highlights Cloudflare's future strategy of adopting OpenBMC for greater transparency and control, and extending secure boot to the BMC firmware to enhance hardware security.
  39. Debugging Hardware Performance on Gen X ServersThis post details the debugging process for performance discrepancies observed in Gen X AMD EPYC Zen 2 (Rome) servers. The investigation involved using DGEMM to assess CPU performance and TDP, iPerf for network bottlenecks, and the AMD HSMP tool. The root cause identified was a mismatch between the memory clock frequency (1467 MHz) and the Infinity Fabric clock frequency (1333 MHz). A vendor-provided BIOS update that set the Infinity Fabric clock to 1467 MHz resolved the performance delta, bringing SKU-B servers to parity with SKU-A.
  40. Everything you ever wanted to know about UDP sockets but were afraid to ask, part 1This post delves into the complexities of operating stateful UDP services at scale, a critical component for modern protocols like QUIC and services like WARP and Spectrum. It details the nuances of connected vs. unconnected UDP sockets, the challenges of sourcing packets from wildcard sockets, and introduces the 'established-over-unconnected' technique as a method for graceful server restarts in UDP. This contributes to the understanding and optimization of Cloudflare's underlying network infrastructure for handling diverse and high-performance UDP traffic.
  41. The tale of a single register valueThis post details a deep dive into debugging a Linux kernel crash (kernel oops) in the ipv4 stack, triggered by a single network packet. It meticulously traces the crash to the `skb_gso_transport_seglen()` function, analyzing the assembly code and register values to pinpoint the cause: an invalid memory access when calculating the inner TCP header length for encapsulated packets. The investigation highlights the challenges of low-level kernel debugging and the importance of understanding packet structures and memory management in network stack operations.
  42. Measuring Hyper-Threading and Turbo BoostThis post details an experiment measuring the performance impact of AMD's Simultaneous Multithreading (SMT) and Core Performance Boost (CPB) on AMD EPYC 7642 processors. It quantifies the improvements in requests per second and latency in both simulated and production environments, and analyzes power consumption. The findings indicate that SMT provides significant performance gains with minimal power increase, while CPB offers further throughput and latency improvements at a higher power cost. The experiment also highlighted tail latency issues with SMT alone and the importance of normalizing performance by CPU utilization for accurate comparisons.
  43. The EPYC journey continues to Milan in Cloudflare’s 11th generation Edge ServerThis post details the selection and validation process for Cloudflare's Generation 11 edge servers, focusing on the adoption of AMD EPYC Milan processors (specifically the 64-core EPYC 7713) over Intel's Ice Lake offerings due to superior performance per watt. It also describes the upgrade in memory configuration to 384GB DDR4-3200 and the transition to 2x Samsung PM9A3 x 1.92TB SSDs for storage. The post outlines the rigorous testing and validation process, including synthetic benchmarking and production environment testing, to ensure performance and reliability.
  44. Designing Edge Servers with Arm CPUs to Deliver 57% More Performance Per WattThis post details the deployment of Arm-based CPUs (Ampere) in Cloudflare's edge servers, demonstrating a 57% improvement in performance per watt compared to AMD Rome CPUs. It highlights the energy efficiency gains and the strategic shift towards diversifying hardware architecture to reduce the carbon impact of internet usage.
  45. Automatic Remediation of Kubernetes NodesThis post introduces Cloudflare's development of an automated Kubernetes node remediation process to handle issues like kernel-level software failures and resource exhaustion on their self-managed bare-metal infrastructure. It details the challenges of manual remediation, explores existing solutions like Node Problem Detector and Kured, and introduces their open-source replacement, Sciuro, which synchronizes Kubernetes node conditions with Prometheus alerts, enabling out-of-band remediation.
  46. Branch predictor: How many "if"s are too many? Including x86 and M1 benchmarks!This post delves into the performance implications of branch prediction, specifically analyzing the cost of numerous conditional and unconditional branches within performance-critical code. It provides theoretical background on branch predictors (BPUs) and the Branch Target Buffer (BTB), and presents experimental results on AMD EPYC and Apple M1 processors. The findings highlight that the number of branches and their density significantly impact performance, with a noticeable degradation beyond a certain threshold (around 4096 branches) due to BTB limitations and potential cache misses. This contributes to the understanding of low-level hardware performance optimization.
  47. ARMs Race: Ampere Altra takes on the AWS Graviton2This post details the evaluation of the Ampere Altra Q80-30 processor against the AWS Graviton2, both based on ARM's Neoverse N1 platform. It provides a detailed comparison of their system specifications, including core counts, operating frequencies, cache hierarchies, and ISA extensions. The post presents benchmark results using Cloudflare's cf_benchmark suite, showing the Ampere Altra outperforming the AWS Graviton2 by 16% in single-core performance and 31% in multi-core performance. It also breaks down performance by category, including OpenSSL, LuaJIT, compression, and Golang workloads, highlighting the Altra's consistent advantage in compute-bound scenarios.
  48. The effect of switching to TCMalloc on RocksDB memory useThis post details how Cloudflare identified and resolved a significant memory consumption issue in its Quicksilver configuration distribution system after migrating its storage engine to RocksDB. By analyzing unexpected memory growth and heap profiling, the team discovered that the default glibc malloc allocator was causing memory fragmentation and inefficient memory reuse in a highly concurrent environment. Switching to TCMalloc, a memory allocator designed for multithreading, resulted in a 2.5x reduction in memory usage, demonstrating a key optimization in Cloudflare's memory management strategies for performance and efficiency.
  49. An introduction to three-phase power and PDUsThis post introduces the fundamental concepts of three-phase power and Power Distribution Units (PDUs) as critical components in Cloudflare's data center infrastructure. It explains how PDUs distribute power to rack-mounted machines, the importance of power redundancy, and the technical underpinnings of AC power, including Faraday's and Ohm's Laws. The post highlights the advantages of three-phase PDUs over single-phase ones in terms of cost-effectiveness, capacity, footprint, and resilience, detailing the electrical principles behind three-phase power generation and distribution (Delta and Wye circuits) and their application in managing power for Cloudflare's extensive fleet.
  50. ASICs at the EdgeThis post introduces the use of Application Specific Integrated Circuits (ASICs) in Cloudflare's network equipment for packet processing. It explains what ASICs are, their benefits (efficiency, power reduction, smaller form factor) and downsides (lack of flexibility, long development cycles). It details the ASIC development process, from FPGA prototyping to foundry manufacturing, and highlights the use of TCAM for fast lookups of forwarding tables, ACLs, and CoS. The post also explains the roles of switching and routing in networking and how ASICs support FIB installation, packet forwarding lookups, stateless firewalls, and advanced features like GRE encapsulation.
  51. Getting to the Core: Benchmarking Cloudflare’s Latest Server HardwareThis post details the upgrade of Cloudflare's core server hardware, specifically the Core Compute 2020 and Core Storage 2020 variants. The Core Compute 2020 servers transitioned from dual-socket Intel Xeon to single-socket AMD EPYC 7642 processors, increasing base frequency and memory speed, and switching to NVMe SSDs. This resulted in significant performance improvements for workloads like Kafka and the CSAM Scanning Tool. The Core Storage 2020 servers also moved to a single-socket Intel Xeon Gold 6210U processor, maintaining core count but increasing frequency and memory speed, with helium-filled SATA drives. Benchmarking showed substantial improvements in storage throughput, IOPS, and latency for ClickHouse workloads.
  52. Anchoring Trust: A Hardware Secure Boot StoryThis post introduces Cloudflare's implementation of hardware secure boot using AMD's Platform Security Boot (PSB) on their servers. It details the boot process, the limitations of UEFI secure boot against firmware attacks, and how PSB, anchored in immutable hardware (AMD PSP), provides a more robust root of trust by authenticating the initial BIOS/UEFI code before execution. It also outlines the Public Key Infrastructure (PKI) and build process involved in establishing this hardware-rooted integrity.
  53. Conntrack tales - one thousand and one flowsThis post delves into the intricacies of Linux's conntrack subsystem, a critical component of the firewall that tracks network connection states. It details how Cloudflare initially avoided conntrack for performance and simplicity but later needed to understand its behavior for new products. The post provides practical testing methodologies using user namespaces and tools like 'unshare' and 'scapy' to demonstrate conntrack's behavior, particularly its tendency to silently drop packets when the connection tracking table fills up. This deep dive into low-level networking behavior contributes to Cloudflare's understanding of hardware and system-level performance optimization, especially in high-throughput environments.
  54. Introducing Quicksilver: Configuration Distribution at Internet ScaleThis post introduces Quicksilver, a new system designed to replace Kyoto Tycoon for configuration distribution at Cloudflare's internet scale. It details the limitations and performance issues encountered with Kyoto Tycoon, specifically focusing on write lock contention that significantly degraded read latency and the operational challenges of data corruption and manual repair. Quicksilver represents a significant engineering effort to build a more robust, performant, and reliable system for managing configuration across Cloudflare's global network.
  55. Speeding up Linux disk encryptionThis post details the investigation into Linux disk encryption performance, identifying a significant performance penalty. It explains the storage stack, block vs. file encryption, and hardware vs. software encryption choices. The core contribution is the benchmarking of `dm-crypt` on a RAM disk, revealing a substantial performance drop. The post sets the stage for further investigation into optimizing this performance.
  56. Keepalives considered harmfulThis post details an investigation into sporadic latency spikes at Cloudflare's edge, specifically within the FL and SSL layers. It highlights how traditional 'keepalive' mechanisms, while generally beneficial for performance by reusing TCP connections, can exacerbate latency issues in specific Linux environments due to uneven worker load distribution. The post explains how a busy worker can lead to a cascade of load, increasing latency. The core contribution is the unconventional solution of disabling keepalives for internal connections between SSL and FL, forcing new connections per request. This strategy, validated through a custom Go program, proved effective in slashing the 99.9th latency percentile by 4x by ensuring requests are always handled by idle workers.
  57. The problem with thread^W event loopsThis post details the challenges and solutions related to Cloudflare's use of event loop architectures, particularly NGINX, for handling its infrastructure. It explains how CPU-intensive tasks, such as WAF processing, can block the event loop and degrade performance. The post discusses the trade-offs of various mitigation strategies and highlights the adoption of NGINX's built-in thread pool functionality to offload these tasks. It also touches upon the importance of kernel patches like EPOLLROUNDROBIN for load balancing and presents performance metrics demonstrating the improvements in TTFB and accept latency after implementing thread pools for WAF processing.
  58. Why Cloudflare Chose AMD EPYC for Gen X ServersThis post details Cloudflare's adoption of AMD EPYC processors for its Gen X server platform, marking a departure from Intel components. It highlights performance improvements such as up to 36% more requests handled, a ~50% decrease in L3 cache miss rate, and up to 50% decrease in NGINX p99 latency, all while using a CPU with a 25% lower TDP per core. Benchmarks for cryptography, compression, regular expressions, and LuaJIT were conducted to select the AMD 2nd Gen EPYC 7642. The post also discusses performance tuning with AMD and the utilization of AMD's Secure Memory Encryption (SME) feature for RAM encryption without significant performance degradation.
  59. Securing Memory at EPYC ScaleThis post details the adoption and performance testing of AMD EPYC's Secure Memory Encryption (SME) feature, specifically Transparent SME (TSME), on Cloudflare's Gen X servers. It explains the components of SME (AES-128 encryption engine and AMD Secure Processor) and how TSME works by encrypting all memory. Performance testing using STREAM, Cryptsetup, and Benchmarky showed a nominal performance decrease (average drag of 0.699%) when TSME was enabled, validating its use for protecting data in RAM against physical attacks.
  60. Gen X Performance TuningThis post details specific performance tuning efforts on AMD EPYC 7642 processors, achieving an additional 6% throughput. Tuning involved adjusting determinism modes (Performance vs. Power) and reconfiguring the Thermal Design Power (cTDP) from 225W to 240W. Performance determinism was found to stabilize frequency but introduced jitter, while power determinism maximized power and frequency. NPS configurations (4, 2, and 1 node per socket) showed no significant performance deltas. The tuning resulted in an additional 2% requests per second with power determinism and another 4% by reconfiguring TDP to 240W in production.
  61. Impact of Cache LocalityThis post details Cloudflare's evaluation of AMD's 2nd Gen EPYC processors (Rome) against Intel's Skylake-based Xeon processors. It highlights the significant performance gains observed due to AMD's larger L3 cache (256 MiB on the EPYC 7642) and improved core count, leading to a 50% increase in requests per second in lab testing and a 36% improvement in production compared to previous generations. The analysis emphasizes the impact of cache locality and low L3 cache miss rates on overall system throughput, informing hardware procurement decisions for future server generations.
  62. An EPYC trip to Rome: AMD is Cloudflare's 10th-generation Edge server CPUThis post details Cloudflare's evaluation and adoption of AMD EPYC 7642 processors as their 10th-generation edge server CPU, replacing Intel Xeon Platinum 6162. The evaluation focused on performance metrics across cryptography, compression, and Go benchmarks (crypto, compression, regexp, strings) and LuaJIT. The AMD EPYC 7642 demonstrated improved performance in most benchmarks, particularly in compression and general Go operations, while maintaining a lower TDP per core and enabling a single-socket configuration. This decision was driven by the key metric of Requests per Watt and the need for increased compute power within existing power envelopes.
  63. Cloudflare’s Gen X: Servers for an Accelerated FutureThis post details the introduction of Cloudflare's tenth generation of servers, 'Gen X', which utilize AMD EPYC 7642 processors instead of Intel. This shift is driven by improved performance per watt, higher core counts, and cost efficiency for Cloudflare's compute-heavy workload. The Gen X servers feature single-socket AMD EPYC 7642 CPUs with 48 cores, 256MB L3 cache, and 256GB of 2933MHz RAM. Storage has been upgraded to 3x1TB NVME SSDs, and the network interface remains a Mellanox 2x25G NIC. The server chassis has reverted to a 1U form factor for operational simplicity. The company also maintains ARM software stack compatibility and expresses interest in future high-core count ARM offerings.
  64. A Tour Inside Cloudflare's G9 ServersThis post details the technical specifications and rationale behind Cloudflare's G9 server generation, highlighting upgrades in CPU (2x 24-core Intel custom off-roadmap CPUs), Disk (Intel S4500 SATA SSDs for lower latency), Network (dual-port 25G Mellanox ConnectX-4 Lx OCP adapters), and Memory (256G DDR4 2666Mhz RDIMM). It also covers physical stress testing for thermal and power consumption, and presents initial production performance comparisons against the G8 generation, showing over 2x request handling capacity with lower CPU usage and improved long-tail latencies.
  65. Porting Our Software to ARM64This post details Cloudflare's process of porting its software stack to ARM64. It covers the challenges and solutions for cross-compiling and emulating code written in C, C++, Go, Lua, Python, and Rust. Key technical details include using `GOARCH` and `CGO_ENABLED=1` for Go, `--target` triples for Rust, and leveraging QEMU with a patched `qemu-user` for user-space emulation within Docker containers. It also addresses specific technical gotchas like `LD_LIBRARY_PATH` issues, intermittent Go segfaults under emulation, and shared library loading order differences between x86_64 and ARM64, along with the solutions implemented (e.g., limiting Go CPUs to one, using `--rpath`). The post also discusses the trade-offs between emulation and native builds for CI and developer access.
  66. NEON is the new black: fast JPEG optimization on ARM serverThis post details the optimization of the `jpegtran` library for ARM processors using NEON SIMD instructions. The author identified performance regressions on ARM compared to Intel Xeon processors and implemented NEON intrinsics to parallelize operations within `encode_mcu_AC_refine` and `encode_mcu_AC_first` functions. This resulted in a 2x speedup for `jpegtran` on ARM, bringing its performance closer to the Xeon implementation.
  67. On the dangers of Intel's frequency scalingThis post details the performance degradation observed when using AVX-512 instructions in cryptography libraries like OpenSSL, due to Intel's dynamic frequency scaling. It highlights how this throttling impacts overall server throughput, even when AVX-512 is used for a small percentage of requests. The author recommends disabling AVX-512 for general-purpose workloads if not specifically required, and notes that Cloudflare's use of BoringSSL avoids this issue.
  68. ARM Takes Wing: Qualcomm vs. Intel CPU comparisonThis post details an early evaluation of Qualcomm's Centriq ARMv8 server platform against Intel's Broadwell and Skylake processors. It benchmarks public-key and symmetric-key cryptography, as well as gzip and brotli compression, highlighting performance differences in single-core and multi-core scenarios. The post also discusses ecosystem readiness for ARMv8, including compiler and library support for NGINX, LuaJIT, and Go.
  69. A Tour Inside CloudFlare's Latest Generation ServersThis post details the design and deployment of Cloudflare's fourth generation (G4) of servers, a significant hardware refresh focused on addressing bottlenecks identified in the previous generation (G3). Key improvements include: a move to 10Gbps network interfaces from Solarflare for higher packet processing rates and larger buffers, the adoption of low-power Intel Xeon 2630L CPUs for increased core count and power efficiency, and a substantial increase in RAM to 128GB to improve in-memory caching. The post also discusses the exclusive use of SSDs, the decision to forgo hardware RAID in favor of software-based consistent hashing for storage, and the selection of SFP+ connectors for network flexibility.