Observability for Self-Managed Kafka on Kubernetes

11/29/2022

This post details the technical steps and configurations required to integrate Confluent Platform deployed on Kubernetes with Datadog for monitoring. It covers creating Datadog API keys, installing Datadog agents on Kubernetes nodes, configuring Datadog site names, applying specific annotations to Confluent Platform component CRDs for JMX and log collection, installing the Confluent Platform integration tile in Datadog, and validating the agent installation and dashboard. The technical depth includes specific YAML configurations and kubectl commands.

3/11/2022

This post details how Confluent augments SIEM strategies by providing a real-time data pipeline for SIEM data, enabling faster iteration and response. It highlights stream processing for agility and incident detection by enriching event data in flight with additional context. It also enables threat detection in live streams of data that are too cost prohibitive to store and index in the SIEM, reducing data indexing, analysis, and storage costs. The post discusses how Confluent helps organizations improve the return on investment in their SIEM tools.

4/22/2021

This post introduces the concept of monitoring Kafka client applications in Confluent Cloud, focusing on visualizing client metrics and identifying failure scenarios. It details the use of the kafka-lag-exporter for consumer lag and the ccloud-exporter for Confluent Metrics API data, both integrated with Prometheus and Grafana. The post also outlines common client failure scenarios like partition limits and request rate limits, explaining how these are reflected in monitoring dashboards and providing examples of metric thresholds and throttling times.

3/29/2021

This post details the integration of Confluent Platform with Prometheus and Grafana for monitoring Kafka clusters. It explains how to export JMX data from Confluent clusters to a third-party metrics aggregation platform, enabling service correlations, consolidated dashboards, root cause analysis, and fine-grained alerts. The post provides a two-part series, with this article focusing on the initial setup and integration of Prometheus and Grafana with Confluent Platform, and a subsequent article covering client monitoring.

3/11/2021

This post introduces the integration of Kafka clients with Jaeger for distributed tracing using OpenTelemetry. It details intrusive (decorated consumer/producer, interceptors) and non-intrusive (in-process Java agent, out-of-process sidecar/eBPF) tracing methods for Kafka. It then focuses on OpenTelemetry's instrumentation libraries, collector, exporters, and propagators, demonstrating a scenario involving GitHub source connector, ksqlDB, and Jaeger UI to visualize event flow and latency. It also discusses challenges with stateful stream processing and header propagation.

12/10/2020

This post details AppsFlyer's implementation of a Kafka lag monitoring system using Burrow. It explains the challenges of monitoring Kafka lag in a large-scale event-driven architecture, the basic concepts of Kafka offsets and lag, and the decision to use Burrow due to its modular design, flexibility in metric calculation, and ability to support both __consumer_offsets and ZooKeeper consumers. The post also touches upon the importance of visibility in distributed systems and the need for actionable metrics to quickly diagnose and resolve issues.