Scaling Apache Druid for Real-Time Cloud Analytics at Confluent

This post details the scaling of Apache Druid for real-time cloud analytics at Confluent. It covers the rationale for choosing Druid over other solutions, the high-level architecture of the telemetry pipeline, and specific techniques for scaling Druid deployments. These techniques include splitting data sources to handle high-cardinality dimensions, implementing data tiering for hot and cold data to optimize query performance, and introducing query laning for prioritization of different query types. The post also addresses bottlenecks in MiddleManager and Historical nodes, detailing the migration to more optimized instance types and local storage. Finally, it discusses the use of compaction to reduce segment reads and improve query efficiency.