Hardware Infrastructure & Performance Optimization

6/1/2026

This post details the debugging and resolution of a significant issue where core unit server boot times increased from minutes to hours after a firmware update. The root cause was identified as an over-eager linear search through available network boot interfaces (PXE, UEFI HTTPS) by the UEFI firmware, leading to lengthy timeouts. The solution involved restructuring the boot automation workflow to declare the network boot interface order early in the pre-boot PXE stage, implementing state validation to handle configuration resets after firmware upgrades, working with vendors to enable programmatic boot order control by addressing lazy loading of EFI_IFR_REF3 data structures and immutable vendor settings, and developing a tool to handle differing network interface card vendor strings for boot configuration. This resulted in reducing firmware upgrade automation time from nearly 4 hours to 3 minutes and subsequent single boot times from about 20 minutes to less than a minute.

3/26/2026

This post details a specific performance bottleneck encountered with Kubernetes persistent volumes, specifically the `fsGroup` setting which caused slow pod restarts due to recursive ownership changes on large filesystems. The post describes the diagnostic process, from observing pod events to delving into kubelet logs and finally identifying the root cause in the PV mounting process. The solution, a one-line change to `fsGroupChangePolicy` to `OnRootMismatch`, dramatically reduced restart times and reclaimed significant engineering hours, highlighting how default Kubernetes settings can become performance bottlenecks as data scales.

3/23/2026

This post details the engineering rationale behind the selection of components for Cloudflare's Gen 13 server hardware. It covers the evaluation and selection of the AMD EPYC Turin 9965 processor, highlighting the trade-offs in L3 cache versus core count and the benefits of increased core count for the FL2 software stack. It also details the memory configuration, emphasizing the use of 12 channels of DDR5-6400 ECC RDIMMs for maximized bandwidth and the decision to maintain a 4 GB per core ratio for sufficient capacity and future growth. The storage subsystem upgrade to PCIe 5.0 NVMe drives and the addition of front drive bays for U.2 drives are also discussed.

3/23/2026

This post details the transition to Cloudflare's Gen 13 server fleet, featuring AMD EPYC Turin processors with a higher core count but reduced L3 cache per core. It describes the performance challenges encountered with the legacy NGINX/LuaJIT-based request handling layer (FL1) due to increased L3 cache miss rates and memory latency. The post highlights the successful adoption of the new Rust-based request handling layer (FL2) which, by reducing cache dependency and optimizing memory access patterns, unlocked the full 2x throughput potential of the Gen 13 hardware while maintaining low latency and improving performance per watt.

10/21/2025

This post dives deep into the performance of BPF LPM tries, a critical data structure used for IP and IP+Port matching in network packet routing and firewall rules. It explains the mechanics of tries, highlights the shortcomings of the current BPF LPM trie implementation (limited branching, lack of level compression), and presents benchmark data showing performance degradation with increasing entry counts. The post identifies specific bottlenecks like slow lookups, updates, deletes, and particularly map freeing, which can lead to production issues like soft lockups and packet loss, setting the stage for future optimization efforts.

10/8/2025

This post details Cloudflare's discovery and resolution of a race condition bug in Go's arm64 compiler. The bug, identified through extensive monitoring of their global network, caused sporadic panics and segmentation faults during stack unwinding. Cloudflare's investigation traced the issue to the interaction between Go's asynchronous preemption mechanism and the `github.com/vishvananda/netlink` library, leading to the identification of a specific bug in the Go compiler's generated code for arm64. Cloudflare's contribution involved deep debugging, correlation with existing Go issues, and ultimately driving towards a root cause analysis that benefits the broader Go ecosystem.

9/26/2025

This post details a significant upgrade to Cloudflare's core network processing system, codenamed 'FL2', which replaces the legacy NGINX/PHP/LuaJIT-based FL1. The new system is built in Rust using the Oxy framework, leading to a 10ms reduction in median response time and a 25% performance boost. Key improvements include enhanced security, faster development cycles, and robust graceful restart capabilities via systemd socket activation and the 'shellflip' coordination mechanism. The modular architecture of FL2 enforces strict separation of concerns, preventing the issues of complexity and latency that plagued FL1.

4/4/2025

This post details Cloudflare's commitment to the CISA Secure-by-Design pledge by focusing on the secure and timely patching of the Linux kernel. It outlines the rigorous process for kernel updates, from upstream release to production deployment, including internal testing, phased rollouts, and the use of employee traffic for validation. The post also highlights the development of automated systems like 'unimog' for traffic management during reboots and 'Reboau' for control plane kernel maintenance, ensuring consistent security across the entire infrastructure.

10/22/2024

This post details the customization and extension of OpenBMC firmware to improve server boot reliability and diagnostics. It addresses issues where servers would not boot despite the BMC indicating power on, and where memory modules were randomly missed. The solution involves better management of BMC and UEFI interactions during the boot process, specifically by leveraging ACPI power states (S5_G2, S0_G0) for more accurate power status reporting and by analyzing i2c bus contention between the BMC and UEFI when accessing memory module SPD data. This allows for more reliable server boot-up and better diagnostics.

10/15/2024

This post details the performance analysis of Cloudflare's Gen 12 servers utilizing AMD EPYC 9684X (Genoa-X) processors. It compares these to other 4th generation EPYC variants (Genoa and Bergamo) and the previous generation (Milan 7713). The analysis includes results from cf_benchmark and an internal performance simulation tool, highlighting a 145% performance gain and 63% efficiency improvement with Genoa-X. Sensitivity analysis demonstrates linear scaling with core count and TDP, and significant performance gains with increased L3 cache size, indicating its critical role for Cloudflare's workload.

10/8/2024

This post introduces Cloudflare's use of KubeVirt to run virtualized Kubernetes clusters on their bare metal infrastructure. This allows teams to run workloads requiring deep kernel integration (like Docker daemons or simulating network environments) alongside standard containerized applications within a secure, multi-tenant Kubernetes environment. This enhances the flexibility and efficiency of their hardware infrastructure by enabling a wider range of workloads on shared resources.

10/7/2024

This post details the thermal design of Cloudflare's Gen 12 hardware, focusing on ensuring reliability, efficiency, and optimal performance. It covers the importance of Thermal Design Power (TDP), the standardization of thermal specifications including ambient conditions and fan failure support, and the strategic system layout for improved airflow. Specific optimizations include the Extended Volume Air Cooled (EVAC) heatsink and the use of larger, more efficient fans. The post also discusses hardware flexibility for incorporating accelerators like GPUs and the importance of rack-integrated thermal testing.

9/25/2024

This post details the deployment of Cloudflare's 12th generation servers, powered by AMD EPYC 9684X processors. It highlights a 145% increase in performance and a 63% improvement in power efficiency compared to the previous generation. Key technical details include the use of 3D V-cache technology for enhanced L3 cache, a 2U1N form factor for improved cooling, support for more powerful GPUs for AI/ML workloads, and the integration of hardware root of trust (HRoT) capabilities. The post also provides a detailed comparison of server specifications and CPU performance metrics.

4/8/2024

This post details how Cloudflare's 'Code Orange' preparedness and investments in Highly Available (HA) topologies significantly improved its resilience to a major data center power failure. It highlights the rapid, automated failover of control plane services (APIs, Dashboard) within minutes, a stark contrast to the hours/days of downtime experienced in a previous similar incident. It also discusses improvements in Logpush infrastructure, the shift of Zero Trust functionality to the global network, and the faster data center recovery time, all demonstrating advancements in hardware infrastructure resilience and operational efficiency.

3/25/2024

This post introduces 'Phoenix,' an autonomous diagnostics and recovery automation system for Cloudflare's global hardware infrastructure. Phoenix automates the detection, diagnosis, and re-provisioning of broken servers, significantly reducing manual effort and improving operational efficiency. It includes out-of-band connectivity checks, integrated node acceptance tests (INAT), and an 'Error Budget' concept to balance automation with empathy for potential recurring hardware failures.

3/19/2024

This post introduces 'Zinc,' an in-house infrastructure system built in Rust for comprehensive fleet management. Zinc models logical and physical infrastructure assets (servers, network devices, data centers) and provides APIs and interfaces for efficient fleet management, including workload assignment, maintenance scheduling, and repair cycle management. It automates processes like repair ticket generation and maintenance window orchestration, integrating with systems like Netbox for physical inventory and INAT for diagnostics. Zinc also enhances engineer productivity through a web interface and CLI for complex queries and changes, ensuring scoped and authenticated access to production infrastructure. This represents a significant evolution in managing Cloudflare's growing hardware infrastructure at scale.

3/6/2024

This post details Cloudflare's implementation of secure boot at the Linux kernel level, focusing on enforcing signed kernel modules. It demonstrates how this prevents unauthorized code injection by requiring modules to have valid signatures, thereby enhancing system integrity and security. The post contrasts this with less secure configurations found in standard distributions and highlights the importance of this measure for Cloudflare's production environments.

12/6/2023

This post details the use of OpenBMC to manage and optimize thermal and power consumption for GPU-equipped servers, enabling AI inference workloads. It describes the process of fine-tuning fan PID controllers by adjusting proportional, integral, and derivative gains to maintain GPU temperatures below critical thresholds. It also covers establishing communication with GPU sensors via SMBus and sysfs, and exposing this data through IPMI and Redfish using OpenBMC's Entity-Manager.

12/1/2023

This post details the design decisions and technical considerations for Cloudflare's Gen 12 compute servers, focusing on the transition to a 2U form factor to accommodate higher TDP CPUs (up to 500W) and GPUs. It analyzes the thermal challenges posed by increasing CPU power draw, the limitations of the 1U form factor, and the performance and TCO benefits of a 2U design with larger fans and heatsinks. The post also discusses the evaluation of new AMD Genoa and Intel Sapphire Rapids chipsets and the impact of CPU TDP on rack-level TCO.

10/16/2023

Introduced Project Argus, a Datacenter-ready Secure Control Module (DC-SCM) design specification developed in partnership with Lenovo. This design decouples the Baseboard Management Controller (BMC) and its related components from the main server motherboard, enabling independent development, cost reduction through specialized PCBs, and unified firmware development with OpenBMC. Project Argus adheres to the DC-SCM 2.0 specification and includes contributions to the Open Compute Project (OCP) database.

10/6/2023

This post details the intricacies of using Linux tap devices for high-performance networking in virtualized environments, specifically within the context of Firecracker MicroVMs. It provides a comprehensive guide to the tap device API, including setup, flags like IFF_VNET_HDR and IFF_MULTI_QUEUE, buffer size management, and offloads such as checksumming and segmentation. This contributes to the broader theme of hardware infrastructure optimization by focusing on maximizing network performance at the virtualized interface level, a critical component for efficient VM operation.

7/25/2023

This post details the mitigation of the 'Zenbleed' vulnerability (CVE-2023-20593) affecting AMD Zen 2 CPUs. It explains the vulnerability, which exploits speculative execution to leak data from AVX registers, and describes Cloudflare's response: patching the entire fleet of potentially impacted servers with AMD's microcode update. The post also provides background on CPU architecture, ALUs, registers, and Advanced Vector Extensions (AVX), as well as the technical details of the attack mechanism involving speculative execution and the `vzeroupper` instruction.

7/12/2023

This post details the automation of determining maintenance windows for system reboots across Cloudflare's global network. It introduces a novel approach using sinusoidal wave fitting on CPU traffic patterns to dynamically identify low-traffic periods, replacing a manual and toilsome process. The post explains the mathematical theory behind sine fitting, its implementation in Python, and how it's integrated into Cloudflare's reboot system with caching and pre-fetch mechanisms. It also discusses accuracy metrics like goodness of fit (chi-squared) to handle edge cases and ensure reliable automated maintenance scheduling.

5/25/2023

This post details a specific, critical issue discovered in the Linux kernel's TCP receive buffer memory management. It explains how certain conditions can lead to unbounded memory allocation, impacting overall server performance and causing packet loss. The post outlines the root cause, provides a reproducible lab setup, and discusses the implications for Cloudflare's production network, highlighting the need for kernel-level fixes to maintain optimal performance and stability.

5/12/2023

This post introduces `pingora-limits`, a new library within Cloudflare's in-house proxy, Pingora. It addresses the need for efficient counting of inflight events and estimation of event rates to protect infrastructure. The library utilizes the Count-Min Sketch algorithm, offering significant improvements in space efficiency (O(log n)) and performance (lock-free, O(1) per event) compared to traditional hash table approaches, especially at Cloudflare's scale of millions of requests per second across tens of thousands of machines.

5/11/2023

This post introduces the Linux Kernel Crypto API as a mechanism for user applications to leverage hardware-accelerated cryptographic operations. It details how applications can interact with the kernel's crypto modules, explores the performance implications of using the API compared to user-space libraries like OpenSSL, and highlights the benefits of offloading encryption/decryption to the kernel for enhanced security and performance, particularly when integrated with kernel key retention services.

4/19/2023

This post details the concepts of memory rank and organization in DDR4 memory modules, explaining how they affect memory bandwidth. It defines memory rank as sets of DRAM chips on a module (single, dual, quad) and discusses rank interleaving. It also explains memory depth and width, illustrating how x4, x8, and x16 device widths contribute to chip capacity. Finally, it explores the relationship between page sizes and timing parameters like tFAW, demonstrating how these hardware specifics influence performance.

3/10/2023

This post details Cloudflare's automated firmware deployment system for its global server fleet. It leverages iPXE for BIOS/UEFI updates via UEFI shell applications and a minimal Linux environment with initial ramdisks for BMC/NIC firmware updates. The system coordinates updates during scheduled reboots, ensuring no production interruption. It also covers the use of UEFI variables for storing firmware versions and the integration with systemd for executing flashing scripts.

2/24/2023

This post details the rewrite of the cf-html NGINX module into a new Rust-based module called ROFL. This initiative addresses memory safety concerns stemming from the original C implementation (highlighted by the Cloudbleed incident) and improves performance and maintainability. It showcases the technical challenges and solutions involved in writing an NGINX module in Rust using Bindgen for FFI bindings and dynamic module loading, demonstrating a shift towards safer and more modern languages for core infrastructure components.

1/31/2023

This post details the discovery and analysis of a kernel bug (CVE-2022-47929) related to Linux Traffic Control's 'noqueue' discipline when used with classful qdiscs, exacerbated by USER namespaces. Cloudflare engineers identified how this vulnerability could lead to an unprivileged denial-of-service attack and contributed a fix by disallowing 'noqueue' for qdisc classes, enhancing the security and stability of the underlying infrastructure.

1/25/2023

This post details Cloudflare's implementation of hardware secure boot for Arm servers, building upon previous work with AMD. It introduces the Arm Trusted Firmware Secure Boot (TBBR) process and highlights its limitations for server environments. The post then describes Cloudflare's partnership with Ampere to develop and implement 'Single Domain Secure Boot' (SDSB) for Ampere Altra Max CPUs. SDSB enhances the boot chain's integrity by incorporating a customer's public key hash into the SoC's eFuse, allowing for end-to-end verification of UEFI firmware.

1/16/2023

This post details a deep dive into debugging a critical issue with AF_XDP, a high-performance networking API. It highlights the process of identifying and resolving a bug that caused packet corruption due to descriptors being used twice across different rings. The contribution lies in demonstrating Cloudflare's commitment to optimizing and ensuring the reliability of its low-level network infrastructure, including the meticulous debugging of kernel interactions and user-space applications to maintain network performance and security.

11/28/2022

This post introduces the Linux Kernel Key Retention Service as a novel approach to protecting cryptographic keys. It explains how memory access violations can lead to key leaks and how the kernel's key retention service can mitigate these risks by storing keys in a separate, isolated kernel address space, effectively acting as a secure agent. This adds a new dimension to Cloudflare's security infrastructure by leveraging kernel-level features for enhanced key management and protection.

10/10/2022

This post delves into the low-level implementation of tail call elimination in BPF (Berkeley Packet Filter) for x86 and ARM architectures. It explains how compilers optimize recursive functions by transforming tail calls into jumps, saving stack space and improving performance. The post then details how Cloudflare leverages BPF tail calls for its XDP-based packet processing pipeline, enabling modularity and efficiency. It also highlights the nuances of BPF tail calls, particularly their interaction with BPF-to-BPF function calls and the challenges faced on arm64, contributing to the ongoing optimization of Cloudflare's network edge processing capabilities.

5/26/2022

This post details Cloudflare's proactive response to critical vulnerabilities found in Baseboard Management Controllers (BMCs). It outlines the background of BMCs, the specific vulnerabilities discovered (related to Pantsdown and USBAnywhere), and the immediate mitigations implemented, including firmware updates, network segmentation, reduced interface exposure, and strong password policies. Crucially, it highlights Cloudflare's future strategy of adopting OpenBMC for greater transparency and control, and extending secure boot to the BMC firmware to enhance hardware security.

5/17/2022

This post details the debugging process for performance discrepancies observed in Cloudflare's Gen X AMD EPYC Zen 2 (Rome) servers. The investigation involved using DGEMM to assess compute performance and TDP, comparing BIOS settings, running iPerf for network bottlenecks, and utilizing the AMD HSMP tool. The root cause identified was a mismatch between the memory clock frequency (1467 MHz) and the Infinity Fabric clock frequency (1333 MHz) on the underperforming SKU-B servers. A vendor-provided BIOS update that synchronized these frequencies to 1467 MHz resolved the performance gap.

11/25/2021

This post delves into the complexities of operating stateful UDP services at scale, a critical component for modern protocols like QUIC and services like WARP and Spectrum. It details the nuances of connected vs. unconnected UDP sockets, the challenges of sourcing packets from wildcard sockets, and introduces the 'established-over-unconnected' technique as a method for graceful server restarts in UDP. This contributes to the understanding and optimization of Cloudflare's underlying network infrastructure for handling diverse and high-performance UDP traffic.

11/3/2021

This post details a deep dive into debugging a Linux kernel crash (kernel oops) in the ipv4 stack, triggered by a single network packet. It meticulously traces the crash to the `skb_gso_transport_seglen()` function, analyzing the assembly code and register values to pinpoint the cause: an invalid memory access when calculating the inner TCP header length for encapsulated packets. The investigation highlights the challenges of low-level kernel debugging and the importance of understanding packet structures and memory management in network stack operations.

10/5/2021

This post details an experiment to quantify the performance impact of AMD's Simultaneous Multi-Threading (SMT) and Core Performance Boost (CPB) on Cloudflare's AMD EPYC 7642 processors. The experiment involved measuring requests per second, latency, and power consumption under simulated and production workloads. Key findings include significant performance improvements with both SMT and CPB enabled, with SMT showing particular power efficiency. The post also highlights challenges in establishing a baseline due to skewed CPU utilization in production and the trade-offs between performance gains and tail latency.

8/31/2021

This post details the selection and integration of AMD EPYC 7713 (Milan) processors and 384GB DDR4-3200 memory for Cloudflare's 11th generation edge servers. It highlights the performance gains (29% boost from 64-core EPYC 7713 over previous generation) and power efficiency improvements compared to Intel's Ice Lake processors. The memory configuration was optimized to balance performance and cost, and disk performance was validated using fio and an automation framework. The post also outlines the rigorous testing and validation process, including Design Validation Test (DVT) and Product Validation Test (PVT) phases.

7/27/2021

This post details the deployment of Arm-based CPUs (Ampere Altra) in Cloudflare's edge servers, achieving a 57% improvement in performance per watt compared to previous AMD Rome CPUs, and a 39% improvement over AMD Milan CPUs. It highlights the company's long-term testing of Arm CPUs since 2017 and the recent July 2021 deployment. The post also emphasizes the environmental benefits of this shift and encourages x86 manufacturers to improve energy efficiency.

7/15/2021

This post introduces Cloudflare's development of an automated Kubernetes node remediation process to handle issues like kernel-level software failures and resource exhaustion on their self-managed bare-metal infrastructure. It details the challenges of manual remediation, explores existing solutions like Node Problem Detector and Kured, and introduces their open-source replacement, Sciuro, which synchronizes Kubernetes node conditions with Prometheus alerts, enabling out-of-band remediation.

5/6/2021

This post delves into the performance implications of branch prediction, specifically analyzing the cost of numerous conditional and unconditional branches within performance-critical code. It provides theoretical background on branch predictors (BPUs) and the Branch Target Buffer (BTB), and presents experimental results on AMD EPYC and Apple M1 processors. The findings highlight that the number of branches and their density significantly impact performance, with a noticeable degradation beyond a certain threshold (around 4096 branches) due to BTB limitations and potential cache misses. This contributes to the understanding of low-level hardware performance optimization.

3/11/2021

This post details the evaluation of Ampere Altra processors against AWS Graviton2, both based on ARM's Neoverse N1 platform. It provides a technical comparison of their specifications, including core counts, clock speeds, cache hierarchies, and ISA extensions (ARMv8.2+ for Altra). The methodology for benchmarking using `cf_benchmark` is described, along with the results showing Ampere Altra outperforming AWS Graviton2 by 16% in single-core and 31% in multi-core performance. Specific results for OpenSSL, compression, and Golang workloads are also presented, highlighting the performance advantages of the Ampere Altra due to its higher core count and operating frequency.

2/3/2021

This post details how Cloudflare identified and resolved a significant memory consumption issue in its Quicksilver configuration distribution system after migrating its storage engine to RocksDB. By analyzing unexpected memory growth and heap profiling, the team discovered that the default glibc malloc allocator was causing memory fragmentation and inefficient memory reuse in a highly concurrent environment. Switching to TCMalloc, a memory allocator designed for multithreading, resulted in a 2.5x reduction in memory usage, demonstrating a key optimization in Cloudflare's memory management strategies for performance and efficiency.

12/4/2020

This post introduces the fundamental concepts of three-phase power and Power Distribution Units (PDUs) as critical components in Cloudflare's data center infrastructure. It explains how PDUs distribute power to rack-mounted machines, the importance of power redundancy, and the technical underpinnings of AC power, including Faraday's and Ohm's Laws. The post highlights the advantages of three-phase PDUs over single-phase ones in terms of cost-effectiveness, capacity, footprint, and resilience, detailing the electrical principles behind three-phase power generation and distribution (Delta and Wye circuits) and their application in managing power for Cloudflare's extensive fleet.

11/27/2020

This post introduces the use of ASICs in Cloudflare's network infrastructure, explaining their function, benefits, development process, and specific applications like TCAM for packet forwarding and stateless firewalling. It details how ASICs handle routing, switching, and initial security checks at line rate.

11/20/2020

This post details the design and benchmarking of Cloudflare's new 'Core Compute 2020' (AMD-based) and 'Core Storage 2020' (Intel-based) server hardware. It covers configuration changes from previous generations, including the move to single-socket CPUs, NVMe SSDs, and improved memory speeds. Benchmarking results for both synthetic and production workloads (Kubernetes, Kafka, CSAM Scanning Tool, ClickHouse) demonstrate significant performance improvements in areas like storage throughput, IOPS, latency, and CPU frequency sustainment, leading to potential cost savings and space efficiency.

11/17/2020

This post introduces Cloudflare's implementation of hardware secure boot using AMD's Platform Security Boot (PSB) on their servers. It details the boot process, the limitations of UEFI secure boot against firmware attacks, and how PSB, anchored in immutable hardware (AMD PSP), provides a more robust root of trust by authenticating the initial BIOS/UEFI code before execution. It also outlines the Public Key Infrastructure (PKI) and build process involved in establishing this hardware-rooted integrity.

4/6/2020

This post delves into the intricacies of Linux's conntrack subsystem, a critical component of the firewall that tracks network connection states. It details how Cloudflare initially avoided conntrack for performance and simplicity but later needed to understand its behavior for new products. The post provides practical testing methodologies using user namespaces and tools like 'unshare' and 'scapy' to demonstrate conntrack's behavior, particularly its tendency to silently drop packets when the connection tracking table fills up. This deep dive into low-level networking behavior contributes to Cloudflare's understanding of hardware and system-level performance optimization, especially in high-throughput environments.

3/30/2020

This post introduces Quicksilver, a new system designed to replace Kyoto Tycoon for configuration distribution at Cloudflare's internet scale. It details the limitations and performance issues encountered with Kyoto Tycoon, specifically focusing on write lock contention that significantly degraded read latency and the operational challenges of data corruption and manual repair. Quicksilver represents a significant engineering effort to build a more robust, performant, and reliable system for managing configuration across Cloudflare's global network.

3/25/2020

This post details the investigation into Linux disk encryption performance, identifying a significant performance penalty. It explains the storage stack, block vs. file encryption, and hardware vs. software encryption choices. The core contribution is the benchmarking of `dm-crypt` on a RAM disk, revealing a substantial performance drop. The post sets the stage for further investigation into optimizing this performance.

3/19/2020

This post details an investigation into sporadic latency spikes at Cloudflare's edge, specifically within the FL and SSL layers. It highlights how traditional 'keepalive' mechanisms, while generally beneficial for performance by reusing TCP connections, can exacerbate latency issues in specific Linux environments due to uneven worker load distribution. The post explains how a busy worker can lead to a cascade of load, increasing latency. The core contribution is the unconventional solution of disabling keepalives for internal connections between SSL and FL, forcing new connections per request. This strategy, validated through a custom Go program, proved effective in slashing the 99.9th latency percentile by 4x by ensuring requests are always handled by idle workers.

3/18/2020

This post details the challenges and solutions related to Cloudflare's use of event loop architectures, particularly NGINX, for handling its infrastructure. It explains how CPU-intensive tasks, such as WAF processing, can block the event loop and degrade performance. The post discusses the trade-offs of various mitigation strategies and highlights the adoption of NGINX's built-in thread pool functionality to offload these tasks. It also touches upon the importance of kernel patches like EPOLLROUNDROBIN for load balancing and presents performance metrics demonstrating the improvements in TTFB and accept latency after implementing thread pools for WAF processing.

3/1/2020

This post details Cloudflare's adoption of AMD EPYC processors for its Gen X server platform, marking a departure from Intel components. It highlights performance benchmarks, including a 36% increase in requests processed and a 50% decrease in L3 cache miss rate compared to Gen 9 servers. The post also discusses the impact of cache locality, performance tuning with AMD, and the integration of AMD's Secure Memory Encryption (SME) feature for enhanced security.

2/28/2020

This post details the implementation and performance testing of AMD's Secure Memory Encryption (SME) and Transparent SME (TSME) features on AMD EPYC processors. It explains the components of SME (AES-128 engine and AMD Secure Processor), how it works with page table entries and transparent encryption, and presents performance test results using STREAM, Cryptsetup, and Benchmarky. The findings indicate a nominal performance impact (average of 0.699% drag across tests) for enabling full memory encryption, making it a viable security enhancement for protecting data in RAM against physical server theft.

2/27/2020

This post details Cloudflare's performance tuning efforts for AMD 2nd Gen EPYC 7642 processors in their Gen X servers. It explores the impact of Thermal Design Power (TDP) and dynamic power on processor performance, explaining that TDP relates to heat dissipation rather than maximum power draw. The post delves into determinism modes (Performance vs. Power) and Configurable TDP (cTDP), demonstrating how adjusting these settings can yield performance improvements. Experiments showed that reconfiguring TDP to 240W and using power determinism resulted in an additional 6% throughput. The exploration of Nodes Per Socket (NPS) did not yield significant performance deltas.

2/26/2020

This post details the evaluation of AMD 2nd Gen EPYC processors (Rome) against Intel Skylake-based Xeon processors, focusing on the impact of L3 cache size and architecture on performance. It presents lab and production environment testing methodologies, CPU specifications, and performance metrics (RPS, operating frequency, instructions retired, L3 cache miss rate). The findings indicate that the AMD EPYC 7642, with its larger L3 cache and MCM design, significantly outperformed Intel processors in simulated and production workloads, leading to a 50% increase in RPS in lab tests and a 36% increase in production.

2/25/2020

This post details the evaluation and adoption of AMD EPYC 7642 processors for Cloudflare's Gen X edge servers, replacing dual-socket Intel Xeon Platinum 6162 CPUs. The evaluation focused on performance metrics like Requests per Watt, comparing cryptographic, compression, and Go-based benchmarks. The AMD EPYC 7642 demonstrated superior performance in most benchmarks, including public key cryptography, zlib and brotli compression, and various Go benchmarks, while maintaining a lower TDP per core and enabling a single-socket configuration for improved efficiency and reduced physical footprint.

2/24/2020

This post details the introduction of Cloudflare's tenth generation of servers, 'Gen X', which utilize AMD EPYC 7642 processors instead of Intel. Key improvements include a 36% increase in request processing capacity, a 50% decrease in L3 cache miss rate, and up to 50% decrease in NGINX p99 latency, all while using a CPU with a 25% lower TDP per core. The server design prioritizes core count and power efficiency for Cloudflare's compute-heavy workloads. The Gen X servers feature 256GB of 2933MHz RAM, 3x1TB NVME SSDs for storage with increased IOPS and endurance, and Mellanox 2x25G NICs. The chassis has been redesigned to a simpler 1U form factor for easier operational work. The company also notes its continued interest in ARM processors and its ongoing efforts to diversify manufacturing partners.

10/10/2018

This post details the introduction of Cloudflare's G9 servers, a significant hardware iteration. Key upgrades include a move to 2x 24-core Intel Skylake CPUs (off-roadmap custom), Intel S4500 SATA SSDs for improved latency and endurance, dual-port 25G Mellanox ConnectX-4 Lx OCP network adapters, and an increase in memory to 256GB DDR4 2666Mhz RDIMM. The post also covers physical stress testing for thermal and power consumption, and initial live production performance comparisons showing over 2x requests per second and reduced CPU usage compared to G8 servers.

9/2/2018

This post details the technical process of porting Cloudflare's software stack to ARM64 architecture. It covers the challenges and solutions for cross-compiling and emulating code for Go and Rust, and the use of QEMU for user-space emulation to build and test other languages. Specific technical details include the use of `GOARCH`, `CGO_ENABLED=1`, `--target` triples for Rust, Docker integration with `qemu-user` and `execve` patching, and workarounds for `LD_LIBRARY_PATH` issues, intermittent Go segfaults due to threading, and shared library ordering. It also discusses the trade-offs between emulation and native builds for performance-critical packages.

4/13/2018

This post details the optimization of the `jpegtran` library for ARM processors using NEON SIMD instructions. The author identified performance regressions on ARM compared to Intel Xeon processors and implemented NEON intrinsics to parallelize operations within `encode_mcu_AC_refine` and `encode_mcu_AC_first` functions. This resulted in a 2x speedup for `jpegtran` on ARM, bringing its performance closer to the Xeon implementation.

11/10/2017

This post details the negative impact of Intel's dynamic frequency scaling with AVX-512 instructions on CPU performance, particularly for cryptographic workloads like ChaCha20-Poly1305, leading to significant throughput degradation and highlighting the need for careful consideration of CPU instruction set usage and its interaction with power management features.

11/8/2017

This post details the evaluation of Qualcomm's Centriq ARM-based server platform against Intel's Broadwell and Skylake architectures. It includes a comparison of hardware specifications (cores, frequency, cache, TDP, instruction sets), an assessment of ecosystem readiness for ARM servers (compiler support, library compatibility), and performance benchmarks for public key cryptography (RSA, ECDSA) and symmetric key cryptography (AES-GCM, ChaCha20-Poly1305) using OpenSSL. It also presents compression benchmarks for gzip and brotli, adapted for ARMv8 architecture, and discusses the implications of these findings for Cloudflare's edge infrastructure.

7/22/2013

This post details the design and implementation of Cloudflare's fourth generation (G4) servers. Key technical contributions include the selection of Intel Xeon 2630L CPUs for power efficiency, the adoption of 10Gbps Solarflare SFC9020 network interfaces with testing of OpenOnload kernel bypass technology, an increase in RAM to 128GB for improved in-memory caching, and the exclusive use of SSDs with direct disk access and consistent hashing, foregoing RAID cards. The post also discusses the rationale behind these choices, focusing on performance bottlenecks, power consumption, and reliability.