Our billing pipeline was suddenly slow. The culprit was a hidden bottleneck in ClickHouse

What this post added

This post details a critical performance bottleneck encountered in Cloudflare's petabyte-scale ClickHouse analytics platform, specifically impacting the billing pipeline. The issue stemmed from a new partitioning scheme ('namespace', 'day') which, while enabling per-namespace retention, led to a massive increase in data parts and subsequent query planning overhead and lock contention. The post outlines the investigation using flame graphs and the development of three patches: using shared locks, deferring vector copying, and optimizing part filtering, which significantly improved performance and were contributed upstream to ClickHouse.