Kafka Client Connectivity Troubleshooting

2 posts · Jan 2019 – Nov 2022

What this trail covers

This feature thread tracks the evolution of diagnosing and debugging Apache Kafka issues, focusing on common symptoms like increased connections, reduced message throughput, increased request rates, and longer consumer rebalance times. It aims to help engineers understand the root causes of these problems by examining broker and client metrics, connection types, and common misconfigurations. The series provides a systematic approach to troubleshooting Kafka deployments. This post specifically details how to use Apache Kafka command-line tools with Confluent Cloud, including configuration, producer/consumer usage, and connectivity troubleshooting.

All Confluent trails

Nov 2022Diagnose and Debug Apache Kafka Issues: Understanding Increased ConnectionsThis post specifically addresses the symptom of increased connections to Kafka brokers. It details the types of connections (producer, consumer, broker-to-broker, AdminClient) and their behavior. It introduces key metrics like `connection-count`, `connection-creation-rate`, and `AcceptorBlockedPercent` for monitoring and diagnosis. It also provides specific debugging questions related to increased memory consumption (consumer per thread), increased consumer group size (undersized workloads), and increased request rates (multiple KafkaProducer instances).
Nov 2022Diagnose and Debug Apache Kafka Issues: Understanding Increased Consumer Rebalance TimeThis post details the causes and mitigation strategies for increased consumer rebalance times in Apache Kafka. It explains the 'Stop-the-World' and 'Incremental Cooperative' rebalancing types, and introduces 'Static Group Assignment' as a method to reduce rebalance impact. Key consumer metrics like 'join-rate' and 'records-lag-max' are highlighted for monitoring. The post also discusses potential root causes related to consumer-per-thread patterns, undersized KafkaConsumer workloads, and overloaded group coordinators.
Nov 2022Diagnose and Debug Apache Kafka Issues: Understanding Increased Request Rate, Response Time, and/or Broker LoadThis post details the client request lifecycle in Kafka, from producer.send() to broker processing, replication, and response. It introduces key JMX metrics for monitoring broker health, including RequestMetrics (RequestsPerSec, TotalTimeMs, RequestQueueTimeMs, LocalTimeMs, RemoteTimeMs, ResponseQueueTimeMs, ResponseSendTimeMs), SocketServer (NetworkProcessorAvgIdlePercent), KafkaRequestHandlerPool (RequestHandlerAvgIdlePercent), and RequestChannel (RequestQueueSize). It also provides diagnostic questions to help identify root causes for increased request rates, such as inefficient batching, improperly sized cloud workloads, client implementation issues, and multiple KafkaProducer instances within a single service.
Nov 2022Reduced Message Throughput | Diagnose and Debug Kafka Issues SeriesThis post, the first in a series on diagnosing and debugging Kafka issues, focuses on reduced message throughput. It details key producer metrics (batch-size-avg, records-per-request-avg, record-size-avg, record-queue-time-avg, request-latency-avg, requests-in-flight) and explains how to interpret them. It also covers essential producer configuration parameters for batching (batch.size, linger.ms, buffer.memory) and discusses potential root causes beyond batching, such as topic-partition count, workload sizing, and multiple KafkaProducer instances.
Jan 20225 Common Pitfalls When Using Apache KafkaThis post identifies and explains five common pitfalls when using Apache Kafka: setting request.timeout.ms too low, misunderstanding producer retries and retriable exceptions, leaving key broker metrics unmonitored, going overboard with partitions, and setting segment.ms too low. It provides detailed explanations and recommendations for each pitfall, focusing on client and broker configurations and design considerations.
Advertisement
Dec 2020Apache Kafka Lag Monitoring and Metrics at AppsFlyerThis post details AppsFlyer's implementation of Burrow for monitoring Apache Kafka consumer lag. It explains the importance of lag monitoring, the limitations of previous solutions (Kafka CLI wrappers), and the selection criteria for a new monitoring system. Burrow was chosen for its modular design, scalability, and ability to provide actionable metrics. The post outlines Burrow's architecture and how AppsFlyer uses it to monitor its large-scale Kafka infrastructure, addressing a critical visibility gap.
Sep 2020How to Connect to MongoDB Atlas in a Secure EnvironmentThis post details how to securely connect to MongoDB Atlas from Confluent Cloud by setting up VPC peering between Confluent Cloud and an AWS VPC, and then establishing a PrivateLink between the AWS VPC and MongoDB Atlas. This setup avoids the need to whitelist broad IP ranges (0.0.0.0/0) in MongoDB Atlas. It provides step-by-step instructions for creating the VPC peering connection and configuring the PrivateLink, along with instructions for setting up the MongoDB Atlas source connector.
Jun 2020Why Can’t I Connect to Kafka? | Troubleshoot ConnectivityThis post details how to troubleshoot Kafka client connectivity issues by focusing on the `advertised.listeners` configuration. It explains the two-step connection process (initial bootstrap and subsequent broker connections) and highlights how misconfigurations in `advertised.listeners` can lead to connection failures. The article provides practical examples and solutions for common scenarios: client and Kafka on the same local machine, client and Kafka on different machines, and client and Kafka running in Docker containers. It emphasizes that `advertised.listeners` must return an address resolvable and accessible from the client machine.
Jun 2019Kafka Listeners – Explained | ConfluentThis post explains the critical difference between Kafka listeners and advertised listeners, emphasizing that advertised listeners must be set to the external address clients can reach. It details how clients obtain broker metadata and then attempt to connect using the advertised addresses. The post illustrates common misconfigurations with Docker and AWS examples, showing how incorrect advertised listeners lead to 'Host resolution failure' errors. It also provides a clear explanation of the configuration parameters (KAFKA_LISTENERS, KAFKA_ADVERTISED_LISTENERS, etc.) and demonstrates how to verify listener configurations using kafkacat.
Jan 2019Using Apache Kafka Command Line Tools with Confluent Cloud | ConfluentThis post details how to use standard Apache Kafka command-line tools (kafka-console-producer, kafka-console-consumer, kafka-consumer-groups) with Confluent Cloud. It explains how to create a configuration file with bootstrap servers, API keys, and secrets, and how to use this file with the command-line tools. It also covers troubleshooting connectivity issues using netcat and discusses common network topologies (public internet, VPC peering).