TechPrufer
Get started
Cloudflare
Cloudflare logoCloudflare · Feature trail

Linux Kernel Upgrade Debugging

1 post · May 2018 – Jun 2026

What this trail covers

Cloudflare's Linux kernel upgrade process has evolved to be more consistent and automated across its network and control plane infrastructure. This includes a standardized monthly Edge Reboot Release cycle for the network and the development of Reboau for automated, rack-aware reboot logic in the Control Plane. The company is also leveraging live patching for faster critical security updates and has implemented robust testing and phased rollout strategies to ensure security and reliability.

All Cloudflare trails
  1. How we reduced core unit boot time from hours to minutesThis post details how Cloudflare reduced core unit boot time from hours to minutes by identifying and fixing a firmware quirk that caused an over-eager linear search through network boot interfaces. The solution involved restructuring the boot automation workflow to declare the correct network boot interface order early in the pre-boot PXE stage, addressing vendor-specific boot order limitations and string format inconsistencies, and implementing a boolean flag in iPXE to reduce boot time. This resulted in a 3-minute boot process compared to nearly 4 hours for firmware upgrades and about 20 minutes for subsequent single boots.
  2. A one-line Kubernetes fix that saved 600 hours a yearThis post details a specific instance where a default Kubernetes setting (`fsGroup: 1` with `fsGroupChangePolicy: Always`) caused significant delays (30 minutes per restart) during Atlantis pod restarts due to recursive `chgrp -R` operations on a large persistent volume with millions of files. The fix involved changing `fsGroupChangePolicy` to `OnRootMismatch`, reducing restart time to 30 seconds and reclaiming approximately 600 hours of engineering time annually. It highlights the potential for default Kubernetes settings to become bottlenecks at scale and the importance of auditing `securityContext` settings, particularly `fsGroup` and `fsGroupChangePolicy`.
  3. Cloudflare’s commitment to CISA Secure-By-Design pledge: delivering new kernels, fasterThis post details Cloudflare's implementation of CISA's 'Secure by Design' pledge, specifically focusing on the automated deployment of Linux kernel security patches. It describes the lifecycle of a kernel update from upstream release to production deployment, including internal testing, phased rollouts, and the Edge Reboot Release cycle. For the Control Plane, it introduces Reboau, an internal tool for managing rack-aware, automated reboots of clustered workloads to ensure consistent kernel patching.
  4. Linux kernel security tunables everyone should consider adoptingThis post details the implementation of secure boot for Linux kernel modules at Cloudflare. It explains how requiring signed kernel modules prevents unauthorized code injection, using a practical example of attempting to load a malicious module on a standard Debian kernel versus a Cloudflare production kernel. The post highlights the `CONFIG_MODULE_SIG_FORCE=y` kernel configuration and the `Key was rejected by service` error encountered when loading unsigned modules, demonstrating a key security hardening measure.
  5. Why is there a "V" in SIGSEGV Segmentation Fault?This post traces the historical evolution of the SIGSEGV signal name, explaining its origin from 'Segmentation Violation' and its roots in early memory segmentation hardware and later paging systems. It details the transition from SIGSEG in early UNIX documentation and kernel internals to SIGSEGV in user-space headers and eventually in kernel code, primarily in BSD-4. The post also includes a code example demonstrating how to catch and handle SIGSEGV in Linux to bypass a null pointer dereference.
  7. Tracing System CPU on Debian StretchThis post details the debugging process for a 5x CPU usage increase and RCU stalls encountered after upgrading Kafka nodes from Debian Jessie to Debian Stretch. The investigation involved using `perf top` and analyzing `dmesg` logs, which revealed memory allocation issues and missed kernel messages. Through bisection, the root cause was identified as a regression in the kernel upgrade from Jessie to Stretch, leading to page allocation stalls. The team rolled back to an older kernel and distribution as a temporary measure while continuing to investigate the specific kernel issue.