Observability for Self-Managed Kafka on Kubernetes

6 posts · Dec 2020 – Nov 2022

What this trail covers

This feature thread tracks the evolution of observability for self-managed Kafka deployments on Kubernetes. It covers integrating monitoring tools like Datadog with Confluent for Kubernetes (CFK) to collect and visualize Kafka telemetry data, including metrics, logs, and traces. The goal is to provide comprehensive visibility into Kafka cluster health and performance, enabling proactive response and troubleshooting in complex, distributed environments.

All Confluent trails

Nov 2022Monitoring and Observability for Kafka Clusters on KubernetesThis post details the integration of Confluent Platform deployed via Confluent for Kubernetes (CFK) on AWS EKS with Datadog for monitoring. It outlines the steps for deploying CFK, creating Datadog API keys, installing Datadog agents on Kubernetes nodes, configuring Datadog site name, and integrating Confluent Platform components (Kafka, ZooKeeper, Connect, Schema Registry) using Datadog annotations. It also describes how to validate the Datadog agent installation and leverage the Confluent Platform dashboard in Datadog for comprehensive visibility into key metrics, logs, and traces.
Mar 2022SIEM Optimization for Better Cyber SecurityThis post details how Confluent augments SIEM strategies by providing a real-time data pipeline for SIEM data, enabling faster iteration and response. It highlights stream processing for agility and incident detection by enriching event data in flight with additional context. It also enables threat detection in live streams of data that are too cost prohibitive to store and index in the SIEM, reducing data indexing, analysis, and storage costs. The post discusses how Confluent helps organizations improve the return on investment in their SIEM tools.
Apr 2021Monitoring Event Streams: Visualize Kafka Clients in Confluent CloudThis post introduces the concept of monitoring Kafka client applications in Confluent Cloud, focusing on visualizing client metrics and identifying failure scenarios. It details the use of the kafka-lag-exporter for consumer lag and the ccloud-exporter for Confluent Metrics API data, both integrated with Prometheus and Grafana. The post also outlines common client failure scenarios like partition limits and request rate limits, explaining how these are reflected in monitoring dashboards and providing examples of metric thresholds and throttling times.
Mar 2021Monitor Apache Kafka Clusters with Prometheus, Grafana, and ConfluentThis post details the integration of Confluent Platform with Prometheus and Grafana for monitoring Kafka clusters. It explains how to export JMX data from Confluent clusters to a third-party metrics aggregation platform, enabling service correlations, consolidated dashboards, root cause analysis, and fine-grained alerts. The post provides a two-part series, with this article focusing on the initial setup and integration of Prometheus and Grafana with Confluent Platform, and a subsequent article covering client monitoring.
Mar 2021Integrate Kafka Clients with Jaeger for Distributed Tracing and MonitoringThis post introduces the integration of Kafka clients with Jaeger for distributed tracing using OpenTelemetry. It details intrusive (decorated consumer/producer, interceptors) and non-intrusive (in-process Java agent, out-of-process sidecar/eBPF) tracing methods for Kafka. It then focuses on OpenTelemetry's instrumentation libraries, collector, exporters, and propagators, demonstrating a scenario involving GitHub source connector, ksqlDB, and Jaeger UI to visualize event flow and latency. It also discusses challenges with stateful stream processing and header propagation.
Advertisement
Dec 2020Apache Kafka Lag Monitoring and Metrics at AppsFlyerThis post details AppsFlyer's implementation of a Kafka lag monitoring system using Burrow. It explains the challenges of monitoring Kafka lag in a large-scale event-driven architecture, the basic concepts of Kafka offsets and lag, and the decision to use Burrow due to its modular design, flexibility in metric calculation, and ability to support both __consumer_offsets and ZooKeeper consumers. The post also touches upon the importance of visibility in distributed systems and the need for actionable metrics to quickly diagnose and resolve issues.