Linux Kernel Hung Task Debugging

4/4/2025

This post details Cloudflare's process for deploying Linux kernel updates, emphasizing the balance between security patching and customer experience. It describes the server maintenance and traffic draining procedures before reboots, the kernel lifecycle from upstream release to production deployment, and the automation of kernel updates in the Control Plane using an internal tool called Reboau. The post also highlights the use of live patching for urgent security fixes and the company's progress towards CISA's Secure-by-Design pledge.

4/2/2025

This post details the debugging of a specific 27-second timeout crash in Jest, a JavaScript testing framework, on Linux systems. The investigation used `strace` to identify the crash occurring after an `epoll_wait` call returns `EINTR` following a `SIGCHLD` signal. This suggests a potential race condition or issue with how Jest handles child processes or signals on Linux, distinct from the previously documented Linux kernel hung task debugging.

2/14/2025

This post introduces the concept of 'hung tasks' in the Linux kernel, explaining the `TASK_UNINTERRUPTIBLE` state and the `khungtaskd` daemon. It details how Cloudflare tunes `kernel.hung_task_timeout_secs` and `kernel.hung_task_warnings` for proactive monitoring. Three detailed examples are provided: 1) XFS file system slowdown identified via `kworker` and `xfs_log_worker`, 2) coredump process causing hung tasks due to memory snapshotting, and 3) a brief mention of `rtnl_mutex` as a tricky debugging scenario.

3/6/2024

This post details the implementation of signed kernel modules and secure boot to prevent unauthorized kernel module loading, enhancing system integrity. It explains how Cloudflare's production kernel requires signed modules, preventing the loading of unsigned modules with a 'Key was rejected by service' error, unlike the Debian stock kernel which allows unsigned modules with a warning. The post also covers the key management aspect for kernel module signing.

6/18/2020

This post traces the historical evolution of the SIGSEGV signal name, from its origins as SIGSEG in early UNIX versions to its current form SIGSEGV in both userspace and kernel. It details the transition through different UNIX versions (V6, V7, BSD-4) and the underlying processor faults (memory segmentation limits, invalid page faults) that led to its naming. It also provides a practical example of how to catch and handle SIGSEGV in Linux to bypass offending instructions.

7/10/2019

This post introduces a method for fuzzing the Linux kernel's netlink machinery using AFL and KCOV. It details how to compile a kernel with KCOV and KASAN enabled, set up KCOV within a userspace process, and feed KCOV coverage data into AFL's shared memory. It also provides a simplified example of crafting netlink messages using AFL's input data.