TechPrufer
Sign inGet started

Blogs›Cloudflare›Linux Kernel Upgrade Debugging

Linux Kernel Upgrade Debugging

Linux Kernel Upgrade Debugging

1
posts
2018

This thread tracks the debugging and resolution of issues encountered during Linux kernel upgrades, specifically focusing on performance regressions and memory allocation problems that manifest in production environments. It includes analysis of tools like `perf`, identification of kernel subsystems like RCU and memory management, and the process of bisecting to pinpoint problematic kernel versions or distribution upgrades.

2018

Tracing System CPU on Debian Stretch

5/13/2018

This post details the debugging process for a 5x CPU usage increase and RCU stalls encountered after upgrading Kafka nodes from Debian Jessie to Debian Stretch. The investigation involved using `perf top` and analyzing `dmesg` logs, which revealed memory allocation issues and missed kernel messages. Through bisection, the root cause was identified as a regression in the kernel upgrade from Jessie to Stretch, leading to page allocation stalls. The team rolled back to an older kernel and distribution as a temporary measure while continuing to investigate the specific kernel issue.