Observability Integrations

12/22/2025

This post details the hidden costs associated with DIY Kafka monitoring (Prometheus/Grafana) and fragmented observability stacks, emphasizing engineering time, license fees, and cognitive overhead. It highlights common pitfalls like over-monitoring irrelevant metrics, under-monitoring key metrics (consumer lag, partition skew), lack of centralized observability, and reactive monitoring. The post contrasts these with Confluent Cloud's integrated, zero-config monitoring, proactive alerting, and end-to-end visibility, positioning it as a cost-efficient and time-saving solution.

10/9/2020

Introduces the New Relic connector for Kafka Connect, detailing its configuration for ingesting Kafka events and metrics into New Relic. It covers connector setup, data formats for events (eventType, timestamp) and metrics (name, type, value, timestamp, interval.ms, attributes), and error handling strategies like dead letter queues, retries for network errors, and payload splitting for large messages. The post also mentions upcoming support for logs and traces.

6/18/2020

Details the creation and implementation of a custom Kafka Connect source connector for Confluent Cloud metrics and a sink connector for AppDynamics. It covers connector configuration, the `poll` method logic for fetching metrics from the Confluent Cloud Metrics API, schema definition using `MetricsAPISchemas`, and the use of Single Message Transformations (SMTs) for data manipulation in the sink connector.

5/7/2020

This post introduces a new agent-based integration with Datadog for monitoring Confluent Platform components including Kafka Connect, ksqlDB, Schema Registry, and REST Proxy. It details how to install and configure the Datadog Agent to collect JMX metrics from these services and host-level system metrics. It also highlights the complementary value of using Confluent Control Center for deeper dives into specific issues identified in Datadog dashboards.

2/11/2020

Introduces a method for integrating osquery event logs into Confluent Platform for SIEM. Details the use of a custom osquery Python extension leveraging librdkafka for publishing logs to Kafka, supporting both Confluent Platform and Confluent Cloud. Demonstrates creating ksqlDB streams from osquery 'processes' and 'open_files' topics and performing a join to identify processes opening files, with a practical example of creating a large file and observing the join results. Discusses the limitations of the built-in osquery Kafka producer plugin and highlights the benefits of the custom extension for cloud readiness and Avro/Schema Registry integration.

7/24/2019

Introduces a fault-tolerant approach for distributed tracing with Jaeger by leveraging Apache Kafka. Applications send tracing spans to a dedicated Kafka topic instead of directly to Jaeger. A separate consumer application then reads from this topic and forwards the spans to Jaeger. This design ensures data durability during Jaeger outages and provides continuous observability. The post details custom KafkaSender and TracingConsumer implementations in Java.

3/26/2019

This post details the implementation of distributed tracing for Apache Kafka-based applications using Zipkin. It explains how Kafka interceptors can be used to instrument Kafka clients, Kafka Streams applications, Kafka Connect pipelines, and KSQL to capture and correlate traces. The article covers context propagation, Zipkin's architecture, and provides examples of tracing Kafka Connectors, Kafka Streams, and KSQL, highlighting the benefits for debugging and latency analysis.

12/14/2017

This post introduces the challenges of monitoring Apache Kafka deployments, highlighting the pitfalls of generic monitoring tools that create a 'metrics swamp' and fail to provide actionable insights. It emphasizes the need for domain-specific monitoring solutions and introduces Confluent Control Center as a solution. Key features discussed include end-to-end stream monitoring using Confluent Monitoring Interceptors to track message delivery and consumption, system health monitoring for cluster, broker, and topic perspectives, and alerting capabilities. The post also addresses common business-level questions that Kafka operators need to answer, such as data reception, application latency, scalability, data loss, and service interruptions, and explains how Control Center helps answer these questions with Kafka-specific context.

12/12/2017

This post introduces Confluent Control Center as a Kafka-specific monitoring solution, detailing its features for system health, end-to-end stream monitoring, and alerting. It presents a demo scenario using Kafka Connect, KSQL, and Elasticsearch to illustrate monitoring a streaming ETL pipeline and provides playbooks for various operational events. It also mentions the quickstart for local deployment.

6/28/2016

This post introduces Confluent Control Center, focusing on two key features: a Kafka Connect GUI for building data pipelines and Stream Monitoring for verifying end-to-end message delivery. The Kafka Connect GUI automatically renders forms for connectors based on the Kafka Connect API. Stream Monitoring provides end-to-end metrics, allowing users to drill down to individual clients, topics, and partitions to identify delivery issues. It also incorporates error detection for lost or duplicate messages and works with custom applications, Kafka Connect, and Kafka Streams via interceptors.