Scaling Apache Druid for Real-Time Cloud Analytics at Confluent

What this post added

This post details Confluent's strategies for scaling Apache Druid for real-time cloud analytics. It introduces splitting data sources by cardinality, implementing data tiering with hot/default tiers and Kubernetes StatefulSets, and query laning with a 'slow-lane' configuration. The post also covers optimizing MiddleManager and Historical node instance types (e.g., c5.9xlarge, i3.2xlarge) for better resource utilization and cost-effectiveness, and discusses compaction strategies to reduce segment counts for improved query performance. It highlights the use of Druid for customer-facing monitoring, billing, and internal operations.