TechPrufer
Get started
Confluent
Confluent logoConfluent · Feature trail

Observability Integrations

11 posts · Jun 2016 – May 2026

What this trail covers

Confluent Cloud Console enhances observability for streaming workloads with new metrics and a refreshed cluster monitoring experience, including client limit breaches, rebalance duration, connection attempts, and partition counts. It also introduces a dedicated Kafka Streams view for consolidated monitoring and debugging of Kafka Streams applications. The console now offers improved client overview with details on client IDs, software versions, and request counts, aiding in the identification of performance bottlenecks. The next generation of Control Center for Confluent Platform integrates with Prometheus, supporting up to 400K partitions, accelerating start-up time to 1 minute, achieving metrics freshness within 2-3 minutes, and eliminating the need for a separate Kafka cluster for metrics storage.

All Confluent trails
  1. New Observability Updates in Confluent Cloud KafkaThis post introduces new metrics for Confluent Cloud Kafka observability: `io.confluent.kafka.server/client_limit_milliseconds` to identify and diagnose client limit breaches by principal and reason, `io.confluent.kafka.server/max_pending_rebalance_time_milliseconds` to evaluate consumer group rebalance duration and frequency, `io.confluent.kafka.server/connection_accept_count` to correlate connection attempts with elastic cluster scaling, and an updated `io.confluent.kafka.server/partition_count` metric with a `cleanup.policy` dimension for compacted partitions. It also highlights improvements to the Confluent Cloud Console's cluster monitoring experience, including an eCKU usage chart for elastic clusters.
  2. Apache Kafka® Monitoring Is Costing You More Than You ThinkThis post argues that traditional DIY Kafka monitoring (e.g., Prometheus/Grafana) becomes prohibitively expensive and complex at scale due to engineering time, license fees, cognitive overhead, tool sprawl, manual dashboard maintenance, alert tuning, and storage costs. It highlights common pitfalls like over-monitoring irrelevant metrics, under-monitoring key metrics (consumer lag, partition skew), lack of centralized observability, and reactive monitoring. The post advocates for integrated, Kafka-native observability solutions like Confluent Cloud, which offer zero-config monitoring, pre-built dashboards, proactive alerting, and end-to-end visibility to reduce TCO and MTTR.
  3. Transforming Confluent Operations With GenAIThis post details the integration of NeuBird's Hawkeye, a GenAI-powered SRE assistant, with Confluent's observability stack. It describes the technical architecture for deploying this enhanced solution on Kubernetes (Amazon EKS), integrating Prometheus Alertmanager for incident triggering, and forwarding metrics and logs to Amazon CloudWatch. The post provides a concrete example of an 'authorization revoked' scenario, contrasting the traditional manual troubleshooting workflow with the automated Hawkeye workflow, highlighting reduced MTTR and improved operational efficiency. It emphasizes the read-only, secure access of Hawkeye to telemetry data and its role in democratizing Kafka operations knowledge.
  4. Check out the latest in the Confluent Cloud ConsoleThis post details significant enhancements to the Confluent Cloud Console's observability features. Key contributions include: a completely overhauled Topic Message Browser with support for up to 1 million messages, ISO timestamp display, and a histogram for message distribution; improved 'produce with schema' functionality for generating schema-compliant message bodies; a remade Clients Overview page providing visibility into client IDs, software versions, and request counts, with specific mention of KIP-896 and deprecated client requests; a simplified Consumer Group monitoring interface with time charts for production, consumption, and lag, including current offset positions, client IDs, and summary metrics, along with the ability to reset consumer group offsets directly from the console; and a new, dedicated Kafka Streams view for consolidated monitoring of applications, including name, version, thread count, and production/consumption rates. Other updates include downloading topic lists to CSV, viewing consumer group protocol types, and viewing schema IDs in the message browser.
  5. Next Generation of Control Center for Confluent PlatformThis post details the re-architecture of Confluent Control Center to integrate with Prometheus, enabling support for up to 400K partitions, reducing start-up time to 1 minute, and achieving metrics freshness within 2-3 minutes. It also eliminates the need for a separate Kafka cluster for metrics storage.
  7. What’s New in Health+: 7 New Dashboards, 70% Monitoring Infra Savings | ConfluentThis post introduces seven new dashboard categories for Health+ (Kafka Health, System, Producers, Consumers, Throughput, Thread utilization, Topics, ksqlDB, Connect) to provide more granular insights into Kafka and Confluent Platform health. It highlights the automatic availability of these dashboards upon activating Confluent's Telemetry Reporter and details how Health+ works by sending telemetry data to Confluent Cloud for analysis. The post also addresses security concerns and emphasizes that message content is not collected, only telemetry data about service health.
  8. SIEM Optimization for Better Cyber SecurityThis post discusses how Confluent can augment SIEM strategies by acting as a real-time data pipeline for security data. It highlights the fragmentation of existing SIEM tools, their batch-oriented nature, lack of incident response agility, and high costs. Confluent's platform is presented as a solution to ingest, aggregate, transform, and enrich data streams in real-time, enabling faster threat detection and response. It also mentions stream processing for real-time threat detection and cost reduction by processing data in-flight. The post outlines how Confluent enables a move from batch to real-time at scale, improves operational agility through stream processing, reduces costs by processing data that is too expensive to index, and enhances ROI in existing SIEM tools.
  9. Bring Your Own Monitoring (BYOM) with Confluent CloudThis post details how to integrate external APM tools (Prometheus, Grafana, Datadog, Splunk) with Confluent Cloud's Metrics API. It provides step-by-step instructions for setting up Prometheus and Grafana using Docker Compose, including configuration files for Prometheus and Grafana dashboards. It also outlines the integration process for Datadog and Splunk, emphasizing the use of Confluent Cloud API keys with the MetricsViewer role and resource IDs for monitoring Kafka clusters, connectors, ksqlDB, and Schema Registry.
  10. Scaling Apache Druid for Real-Time Cloud Analytics at ConfluentThis post details the scaling of Apache Druid for real-time cloud analytics at Confluent. It covers the rationale for choosing Druid over other solutions, the high-level architecture of the telemetry pipeline, and specific techniques for scaling Druid deployments. These techniques include splitting data sources to handle high-cardinality dimensions, implementing data tiering for hot and cold data to optimize query performance, and introducing query laning for prioritization of different query types. The post also addresses bottlenecks in MiddleManager and Historical nodes, detailing the migration to more optimized instance types and local storage. Finally, it discusses the use of compaction to reduce segment reads and improve query efficiency.
  11. Monitoring Event Streams: Visualize Kafka Clients in Confluent CloudThis post details how to monitor Kafka client applications in Confluent Cloud, focusing on client resource usage and performance. It introduces the use of kafka-lag-exporter for consumer lag metrics and ccloud-exporter for Confluent Metrics API data. The post highlights specific client scenarios (general, producer, consumer) and failure modes, such as hitting partition limits and request rate limits, and how these manifest in metrics and dashboards. It also references a tutorial for practical implementation using Prometheus and Grafana.
  12. Monitor Apache Kafka Clusters with Prometheus, Grafana, and ConfluentThis post details the integration of Confluent Platform with Prometheus and Grafana for enhanced observability. It explains how to configure JMX exporters on Kafka brokers and other Confluent components to expose JVM metrics in a Prometheus-compatible format. The post provides guidance on setting up Prometheus for scraping these metrics and configuring Grafana for creating custom dashboards to visualize cluster health, broker performance, and topic-level statistics. It also references a GitHub repository with code examples and demo setups.
  13. Integrate Kafka Clients with Jaeger for Distributed Tracing and MonitoringThis post introduces the integration of OpenTelemetry for distributed tracing with Kafka clients, ksqlDB, Kafka Connect, and Confluent Schema Registry. It describes intrusive (decorated consumer/producer, interceptors) and non-intrusive (Java Instrumentation API, sidecar containers, eBPF) tracing methods for Kafka. The solution leverages OpenTelemetry's instrumentation libraries, collector, exporters, and propagators to visualize the flow of events and associated latency in Jaeger. It also discusses challenges related to stateful stream processing and header propagation in Kafka Streams.
  14. Connect Kafka to New Relic for Real-Time Monitoring and ObservabilityIntroduces the New Relic connector for Kafka Connect, detailing its configuration for ingesting Kafka events and metrics into New Relic. It covers connector setup, data formats for events (eventType, timestamp) and metrics (name, type, value, timestamp, interval.ms, attributes), and error handling strategies like dead letter queues, retries for network errors, and payload splitting for large messages. The post also mentions upcoming support for logs and traces.
  15. Monitoring & Metrics for Confluent Cloud ksqlDB - Stream Processing ServiceThis post details the strategy and implementation of monitoring and alerting for Confluent Cloud ksqlDB, focusing on three key goals: prompt response to failures, proactive mitigation of potential issues, and confidence in system stability. It describes the use of health check instances, fine-grained metrics for provisioning and resource utilization, and redundant monitoring pipelines to ensure reliability. Lessons learned include the importance of fine-grained metrics, propagating failure information bidirectionally along the dependency chain, and investing in avoiding alert fatigue.
  16. Build Observability Pipelines Using Confluent and AppDynamicsThis post details the technical implementation of an observability pipeline using Kafka Connect to ingest Confluent Cloud metrics and push them to AppDynamics. It covers the architecture of Kafka Connect, the anatomy of a custom Java source connector (MetricsAPISourceConnector), its configuration (MetricsAPISourceConnectorConfig, MetricsAPISourceTask), and the use of Single Message Transformations (SMT) for the AppDynamics sink connector. It also provides instructions for installing and configuring the connectors and demonstrates how to set up monitoring dashboards in AppDynamics.
  17. Confluent & Datadog Bring Enterprise Monitoring & Metrics for KafkaThis post announces a new agent-based integration with Datadog for monitoring Confluent Platform. It details how the Datadog Agent can be configured to collect JMX metrics from Kafka Connect, ksqlDB, Confluent Schema Registry, and Confluent REST Proxy, as well as Java-based Kafka clients. It highlights the inclusion of a default Confluent Platform dashboard in Datadog and the optional configuration for log collection. The post also explains how this integration complements Confluent Control Center by providing a broader IT infrastructure monitoring view, enabling users to correlate Confluent Platform metrics with other system metrics for better debugging and operational insights.
  18. SIEM with Osquery Event Log Aggregation and Confluent PlatformThis post details how to integrate osquery event logs with Confluent Platform for SIEM use cases. It demonstrates using an osquery Python extension with librdkafka to publish logs to Kafka, and then using ksqlDB to create streams and join process and open_files events for real-time analysis. The implementation covers setting up a local Confluent Platform cluster with Docker and viewing the data in Confluent Control Center.
  19. Fault Tolerance in Distributed Systems: Tracing with Apache Kafka and Jaeger | ConfluentThis post introduces a pattern for making distributed tracing solutions (specifically Jaeger) more fault-tolerant by using Apache Kafka as an intermediary for trace data. Instead of applications sending trace spans directly to Jaeger, they send them to a Kafka topic. A separate application then consumes from this topic and forwards the spans to Jaeger. This ensures that trace data is not lost if Jaeger is unavailable and provides continuous tracing insight during downtime. The implementation involves custom Jaeger sender and consumer configurations.
  20. The Importance of Distributed Tracing for Apache-Kafka-Based Applications | ConfluentThis post introduces the concept and implementation of distributed tracing for Apache Kafka-based applications using Zipkin. It details how to instrument Kafka clients, Kafka Streams applications, Kafka Connect pipelines, and KSQL to gain visibility into dataflows, analyze latency issues, and debug runtime problems. The article covers the architecture of Zipkin, the use of Kafka interceptors, and provides examples of tracing Kafka Connectors and Kafka Streams applications, demonstrating how to correlate spans across different components to achieve end-to-end visibility.
  21. The Apache Kafka Monitoring Blog Post to End Most Posts | ConfluentThis post introduces the concept of monitoring Apache Kafka deployments, highlighting the limitations of generic monitoring tools and the 'metrics swamp' problem. It emphasizes the need for domain-specific solutions and introduces Confluent Control Center as a solution. Key monitoring areas discussed include end-to-end stream monitoring to ensure message delivery, and system health monitoring for cluster well-being. It details how Control Center uses Confluent Monitoring Interceptors and the Confluent Metrics Reporter to provide application-level visibility and answer business-critical questions about data delivery, latency, and capacity.
  22. Monitoring Apache Kafka with Confluent Control Center | ConfluentThis post introduces Confluent Control Center as a Kafka-specific monitoring solution, highlighting its features for system health, end-to-end stream monitoring, and alerting. It details how Control Center addresses key operational questions like broker status, application data reception, and performance bottlenecks. The post also outlines a demo scenario involving a streaming ETL pipeline using Kafka Connect, KSQL, and Elasticsearch, demonstrating Control Center's capabilities in monitoring consumer rebalances, slow consumers, and failed brokers.
  23. Build and monitor Kafka pipelines with Confluent Control Center | ConfluentThis post introduces Confluent Control Center, focusing on two key features: a Kafka Connect GUI for building data pipelines and Stream Monitoring for verifying end-to-end message delivery. The Kafka Connect GUI automatically renders forms for connectors based on the Kafka Connect API. Stream Monitoring provides end-to-end metrics, allowing users to drill down to individual clients, topics, and partitions to identify delivery issues. It also incorporates error detection for lost or duplicate messages and works with custom applications, Kafka Connect, and Kafka Streams via interceptors.