TechPrufer
Get started
Cloudflare
Cloudflare logoCloudflare · Feature trail

Linux Kernel Hung Task Debugging

6 posts · Jul 2019 – Apr 2025

What this trail covers

Cloudflare's Linux kernel upgrade process has evolved to be more consistent and automated across its network and control plane infrastructure. This includes a standardized monthly Edge Reboot Release cycle for the network and the development of Reboau for automated, rack-aware reboot logic in the Control Plane. The company is also leveraging live patching for faster critical security updates and has implemented robust testing and phased rollout strategies to ensure security and reliability. This post details the investigation and debugging of hung tasks, a critical aspect of kernel stability and reliability, by analyzing kernel messages, stack traces, and system configurations to identify root causes such as filesystem pressure, coredump generation, and mutex contention.

All Cloudflare trails
  1. Cloudflare’s commitment to CISA Secure-By-Design pledge: delivering new kernels, fasterThis post details Cloudflare's process for deploying Linux kernel updates, emphasizing the balance between security patching and customer experience. It describes the server maintenance and traffic draining procedures before reboots, the kernel lifecycle from upstream release to production deployment, and the automation of kernel updates in the Control Plane using an internal tool called Reboau. The post also highlights the use of live patching for urgent security fixes and the company's progress towards CISA's Secure-by-Design pledge.
  2. A steam locomotive from 1993 broke my yarn testThis post details the debugging of a specific 27-second timeout crash in Jest, a JavaScript testing framework, on Linux systems. The investigation used `strace` to identify the crash occurring after an `epoll_wait` call returns `EINTR` following a `SIGCHLD` signal. This suggests a potential race condition or issue with how Jest handles child processes or signals on Linux, distinct from the previously documented Linux kernel hung task debugging.
  3. Searching for the cause of hung tasks in the Linux kernelThis post details the investigation and debugging of hung tasks in the Linux kernel. It explains the `TASK_UNINTERRUPTIBLE` state, the role of `khungtaskd`, and configurable parameters like `kernel.hung_task_timeout_secs`. It provides three detailed examples: a hung task related to XFS filesystem pressure, a hung task caused by coredump generation during abnormal process termination, and a tricky case involving `rtnl_mutex` contention. The post also introduces the `PF_POSTCOREDUMP` flag and the 'Blocked by coredump.' kernel message.
  4. Linux kernel security tunables everyone should consider adoptingThis post details the implementation of signed kernel modules and secure boot to prevent unauthorized kernel module loading, enhancing system integrity. It explains how Cloudflare's production kernel requires signed modules, preventing the loading of unsigned modules with a 'Key was rejected by service' error, unlike the Debian stock kernel which allows unsigned modules with a warning. The post also covers the key management aspect for kernel module signing.
  5. Why is there a "V" in SIGSEGV Segmentation Fault?This post traces the historical evolution of the SIGSEGV signal name, from its origins as SIGSEG in early UNIX versions to its current form SIGSEGV in both userspace and kernel. It details the transition through different UNIX versions (V6, V7, BSD-4) and the underlying processor faults (memory segmentation limits, invalid page faults) that led to its naming. It also provides a practical example of how to catch and handle SIGSEGV in Linux to bypass offending instructions.
  7. A gentle introduction to Linux Kernel fuzzingThis post introduces a method for fuzzing the Linux kernel's netlink machinery using AFL and KCOV. It details how to compile a kernel with KCOV and KASAN enabled, set up KCOV within a userspace process, and feed KCOV coverage data into AFL's shared memory. It also provides a simplified example of crafting netlink messages using AFL's input data.