Linux Kernel Bug Hunting

4 posts · May 2017 – Oct 2026

What this trail covers

This post extends the Linux kernel bug hunting capability by detailing the discovery and resolution of six bugs found during recurring network maintenance. These bugs affected glibc, the OCaml networking stack, Async, the Kafka client, and a job runner, highlighting issues with DNS resolution, socket handling, request retries, and file descriptor limits. The post details the investigation process, the root causes of each bug (including an eleven-year-old glibc bug and a fourteen-year-old Async regression), and the fixes implemented. It also discusses the interplay between different system layers and the importance of robust error handling and resource management in distributed systems.

All Jane Street trails

Oct 2026How recurring network maintenance exposed 6 bugsThis post details the discovery and resolution of six bugs encountered during recurring network maintenance. It specifically addresses: 1. An eleven-year-old glibc bug in the DNS resolver leading to segfaults under specific error conditions. 2. A UDP socket leak in the OCaml socket library when `bind` fails under ephemeral-port pressure. 3. A fourteen-year-old regression in Async where `Tcp.connect_sock`'s timeout no longer covers DNS resolution. 4. A Kafka client request retry loop that times out without aborting the underlying attempt. 5. A bug in the Kafka client's bootstrapping logic that can duplicate concurrent lookups for broker discovery. 6. A job runner migration that silently reduced file-descriptor limits. The post outlines the investigation process, the root causes, and the fixes for each bug, contributing to the ongoing effort of identifying and mitigating complex system failures.
Oct 2026A resilient kernel log reporterIntroduces a resilient kernel log reporter that can gather kernel logs even when disk or network access is unavailable. It achieves this by reading directly from /dev/kmsg, using mlockall to keep the binary in memory, and implementing a DNS caching mechanism for service discovery. For scenarios without network access, it leverages the pstore subsystem to capture logs during kernel panics, with a custom parser for EFI backend logs and monitoring to ensure pstore is cleared.
Jul 2025A Higgs-bugson in the Linux KernelThis post introduces a novel "Higgs-bugson" phenomenon in the Linux kernel, detailing the technical approach to discovering and analyzing these elusive issues. This contributes to kernel stability and reliability.
May 2017When Bash Scripts BiteThis post details the challenges of debugging complex shell scripts, particularly those interacting with the Linux kernel, and introduces techniques for identifying and mitigating common pitfalls such as unexpected variable expansion, improper quoting, and race conditions. It highlights how seemingly simple scripts can lead to subtle and hard-to-diagnose issues, akin to the "Higgs-bugson" phenomenon in kernel bug hunting.
Advertisement