Confluent Feature Trails

This release details bug fixes and general improvements for Apache Kafka 0.9.0.1, focusing on correctness, bug fixes, and operability. Key areas addressed include the new Java consumer (fixing rebalancing and offset issues), compatibility fixes, security enhancements (ticket renewal, ACLs, SASL packet handling), performance improvements (fetch wait, CPU usage, log cleaner, sensor leaks), and topic deletion fixes. This patch release addresses 68 issues, with 45 bug fixes and 20 general improvements. This post also discusses KIPs for incremental batch processing in Kafka Streams, improved client RPC compatibility, and the proposal for exactly once delivery and transactional messaging. It also mentions community events and other streaming system insights.

This release enhances Confluent Intelligence with generally available Real-Time Context Engine featuring low-latency querying capabilities (filters, ranges, compound queries, projections, ordering) directly on live tables, eliminating the need for external databases. It also introduces GA for Streaming Agents and the Agent Management Console, providing production-ready, event-driven AI agents with enterprise-grade operations and a centralized UI for management. New ML functions include Multiva. This release expands Confluent Intelligence with Agent2Agent (A2A) Integration for Streaming Agents, Multivariate Anomaly Detection for Built-in ML Functions, Vector Search for Azure Cosmos DB and Amazon S3 Vectors, AWS and Azure Private Link for Model Inference, External Tables, and Search, and Confluent Support for the open source MCP Server for Confluent Cloud.

This release introduces Confluent AI Developer Tools, including an open-source local Model Context Protocol (MCP) server, a managed MCP server hosted by Confluent, and Agent Skills. These tools enable AI coding assistants to interact with and understand Confluent environments, facilitating tasks like discovering topics, diagnosing issues, building pipelines, and managing schemas. The MCP server provides a standardized interface for AI assistants to connect to external systems, while Agent Skills enable agents to communicate with each other and external resources via Confluent's event streaming platform, acting as a proxy for MCP and A2A protocols. This post details the integration of Confluent with Google Cloud to provide a real-time backbone for agentic AI systems, leveraging MCP and A2A protocols for inter-agent communication and tool integration.

This release enhances Confluent Cloud's secure networking capabilities by detailing options for both public and private integrations. For public endpoints, it covers Public Egress IPs for firewall rules and AWS Gateway/Azure VNet Service Endpoints for traffic within cloud provider networks. For private endpoints, it introduces DNS Forwarding for VPC/VNet Peering or Transit Gateway setups and Egress Access Points for AWS PrivateLink, enabling secure connections to data sources and sinks without t

This release details how Confluent Cloud addresses digital sovereignty for real-time streaming through architectural guarantees rather than policy assurances. Key concepts include Bring Your Own Cloud (BYOC) and Bring Your Own Key (BYOK) for data plane control and encryption, treating schemas as sovereignty boundaries with integrated data contract enforcement, sensitivity/jurisdiction tagging, encryption directives (CSFLE/CSPE), and quality/policy rules. It also emphasizes protocol-level portability with open formats like Apache Kafka and Apache Flink, contrasting contractual exit plans with executable workflows. The post outlines the operational costs of architectural sovereignty, such as managing the data plane, upgrades, patching, capacity planning, key lifecycle management, and cross-cluster governance, and provides a checklist for evaluating vendor architectures and stream-level governance.

This release enhances Confluent Cloud's availability by making it accessible through major cloud provider marketplaces (AWS, Azure, Google Cloud). This allows for unified billing, utilization of pre-committed cloud spend, and a consistent experience across different cloud environments, simplifying procurement and adoption for enterprises. It also emphasizes the portability offered by Confluent Cloud's presence across multiple clouds. This post details the validation of Confluent Platform on AWS, Azure, and GCP, and demonstrates building a persistent bridge from on-premises or other clouds to Microsoft Azure using Confluent Replicator, integrating with Azure services like ADLS and Power BI.

This release details the GA for Kafka release 0.11.0.0, bringing Exactly Once Semantics (EOS) with more stringent idempotent producer semantics, exactly once, in-order delivery per partition, and stronger transactional guarantees with atomic writes across multiple partitions. It also highlights the growing adoption of Kafka Connect API with new community-contributed connectors for AWS S3 sink, Redis source, RabbitMQ source, GitHub source, Amazon Kinesis source, and MongoDB sink. This post details the successful migration of thousands of Kafka clusters from AWS to GCP, involving petabytes of data, using Confluent Replicator for mirroring traffic and Confluent Platform for operational stability. It highlights the shift from batch processing to event streaming, reducing latency from two days to 15 minutes, and improved reliability leading to Kafka not being a suspect for issues. The post also mentions plans for feeding machine learning models with real-time data and using Confluent Cloud for real-time fraud detection, processing over 500,000 daily transactions. It highlights the integration of Confluent Cloud for real-time transaction processing with Apache Spark for batch processing and ML model training. The system analyzes transaction data, customer information, and account details streamed through Confluent Cloud to train ML models and trigger real-time alerts. This has led to a significant reduction in fraud losses and improvements in p99 latency for critical trading pipelines, achieving sub-5ms latency and sustaining 1.6 million messages/sec with messages smaller than 5KB through meticulous tuning of brokers, producers, and consumers, architectural discipline, and in-depth monitoring.

This release details the engineering efforts behind making Confluent Cloud a 10x better Apache Kafka service. It highlights the transformation from a simple managed Kafka offering to a true cloud-native, elastic utility. Key areas of focus include building a massively multi-tenant architecture, optimizing for heavily tuned cloud environments, achieving elastic scaling, implementing self-protective operational limits, and designing for highly automated data-driven operations. The post emphasizes the use of Kafka's built-in multi-tenancy features (security, namespace isolation, performance quotas) and extends them to a cloud-native context with logical clusters representing tenants, enabling cost-efficiency, elasticity, and stringent SLOs.

This release details the development and release of a Kafka Connect sink connector for Azure Cosmos DB, leveraging its capabilities for improved efficiency and performance. It introduces integration with Confluent Platform for real-time data streaming into Cosmos DB. The connector supports JSON and Avro data formats and enables data ingestion from Kafka topics into Cosmos DB containers. This builds upon the existing portfolio of connectors, adding a specific integration for a popular NoSQL database.

This post details the integration of osquery event log aggregation with Confluent Platform for SIEM use cases. It demonstrates using a custom osquery Python extension with librdkafka to publish logs to Kafka, enabling real-time processing and analysis with ksqlDB. This builds upon existing observability capabilities by providing a concrete method for ingesting and analyzing system-level audit logs.

This release enhances Confluent Cloud by integrating Apache Flink as a fully managed service, building upon existing stream processing capabilities with Kafka Streams and ksqlDB. The integration aims to provide a seamless experience for developing stream processing applications, offering multi-language support (SQL, Java, Python), advanced state management, and fault tolerance. This move positions Flink as a core component of Confluent's go-forward stream processing strategy, complementing Kafka Streams and ksqlDB. This post discusses the evaluation of stream processing engines, highlighting Apache Flink and Kafka Streams, and the importance of choosing the right data streaming platform and stream processing engine for business needs. It also touches upon the complexities of building robust streaming frameworks and the role of managed services in simplifying stream processing.

This release introduces Private Link for Schema Registry and Apache Flink, enabling secure, private access to these critical components from within customer virtual private clouds. It also adds mTLS authentication for Dedicated clusters, Egress Private Link for connectors on Enterprise clusters, and private networking on Azure for Confluent Cloud for Apache Flink. User-defined functions (UDFs) are now generally available in Java on AWS for Flink SQL, with table functions also introduced. WarpStream gains WarpStream Orbit for simplified migration and replication, and BYOC Schema Registry for managing schemas within customer cloud accounts. Follower fetching is now generally available for AWS VPC peering to reduce costs. Five new Connect with Confluent integrations are added, including SAP Datasphere hydration.

This post details Unity's successful migration of thousands of Kafka clusters from AWS to GCP, involving petabytes of data, using Confluent Replicator for mirroring traffic and Confluent Platform for operational stability. It highlights the shift from batch processing to event streaming, reducing latency from two days to 15 minutes, and improved reliability leading to Kafka not being a suspect for issues. The post also mentions plans for feeding machine learning models with real-time data and using Ansible Playbooks for upgrades.

This post details how the Levyx Kafka-Connect-Xenon connector integrates with Kafka Connect to stream data into the Xenon analytics engine. It showcases high-performance ingestion of 470GB of data at 3.5 million messages per second on commodity hardware with NVMe SSDs. The connector supports Avro and JSON formats, and Xenon offers multi-tenancy for concurrent analytics via Apache Spark. Confluent Hub is launched as a central repository for Kafka Connectors, Transformations, and Converters, simpl

This release details how Confluent Control Center integrates with Confluent Schema Registry to manage and evolve schemas. It allows users to view, edit, and check the compatibility of schemas directly through the UI. The post explains schema evolution concepts, compatibility policies (backward, forward, full, none), and demonstrates how Control Center enforces these policies, preventing incompatible schema changes. It also highlights the ability to compare schema versions. This enhances data governance and provides a central location for managing schemas across an organization, ensuring data consistency and enabling resilient data pipelines.

This release enhances Confluent Cloud's capabilities for real-time order notifications and logistics management. It details how ACERTUS migrated from siloed, batch-processed data systems to a modern microservices architecture powered by Confluent's event streaming platform. Key technical aspects include integrating disparate databases (Postgres, MySQL, SQL Server) using CDC connectors, leveraging ksqlDB for real-time data processing and microservice development, and utilizing Stream Governance for data governance. This post details building a real-time inventory system for retail, consuming data from various sources, processing it with ksqlDB to maintain an up-to-the-minute view of stock on hand, and distributing this data to downstream systems like MongoDB for enhanced customer experiences and fulfillment decisions.

This release details building an end-to-end IoT motion detection and alerting system using Confluent Cloud and ksqlDB. It covers hardware choices (Raspberry Pi Zero W), software components (Motion, librdkafka, Docker), and the architecture for ingesting motion events into Kafka. The system utilizes ksqlDB for real-time stream processing and querying, enabling both real-time monitoring with push queries and historical event search with pull queries. The UI is built with React and Next.js, demonstrating real-time asset tracking with Apache Kafka and Kafka Streams, integrating MQTT data, enriching it with Kafka Streams, and visualizing it in Kibana via Elasticsearch.

This post details how EVO Banco uses Confluent Cloud for real-time fraud detection, processing over 500,000 daily transactions. It highlights the integration of Confluent Cloud for real-time transaction processing with Apache Spark for batch processing and ML model training. The system analyzes transaction data, customer information, and account details streamed through Confluent Cloud to train ML models and trigger real-time alerts. This has led to a significant reduction in fraud losses and improved operational efficiency.

This release details how the Kafka ecosystem can be leveraged to build scalable machine learning infrastructure, addressing the impedance mismatch between data scientists and production engineers. It highlights the use of Kafka for real-time data ingestion, processing, model deployment, and monitoring. The post explores using Jupyter Notebooks with KSQL and TensorFlow/Keras for model training, and discusses how the same Kafka pipelines can be reused for model inference. It emphasizes Kafka's role in enabling two primary approaches for model deployment: model servers with RPC communication and embedding models directly into Kafka client applications. The post also discusses considerations for embedding models, such as performance, size, and the need for model server features, and evaluates the tradeoffs between the two approaches. Finally, it touches upon cloud-native deployment strategies using Kubernetes to combine the benefits of both approaches.

This release enhances Confluent Cloud's capabilities for real-time order notifications and logistics management. It details how ACERTUS migrated from siloed, batch-processed data systems to a modern microservices architecture powered by Confluent's event streaming platform. Key technical aspects include integrating disparate databases (Postgres, MySQL, SQL Server) using CDC connectors, leveraging ksqlDB for real-time data processing and microservice development, and utilizing Stream Governance for data quality and lineage. This post details building an end-to-end IoT motion detection and alerting system using Confluent Cloud and ksqlDB. It covers hardware choices (Raspberry Pi Zero W), software components (Motion, librdkafka, Docker), and the architecture for ingesting motion events into Kafka. The system utilizes ksqlDB for real-time stream processing and querying, enabling both real-time monitoring with push queries and historical event search with pull queries. The UI is built with React and Next.js, demonstrating a practical application of stream processing for fraud detection.

This release details the integration of AWS Lambda with Confluent Cloud, offering two primary patterns: a Confluent Lambda Sink Connector and AWS Lambda's event source mapping for Kafka. The sink connector supports synchronous and asynchronous invocations, with asynchronous mode offering at-most-once delivery and dead-letter queue capabilities, suitable for fire-and-forget scenarios. Synchronous invocation with task parameters is recommended for ordered processing across partitions. AWS Lambda's

This post introduces the integration of policy-as-code with Confluent's Terraform provider, enabling self-service management of Confluent resources with automated enforcement of security, compliance, and auditing requirements. It details how policy-as-code is applied during the Terraform plan phase using HashiCorp Sentinel, with examples of policies for cloud provider and region restrictions, naming conventions, and RBAC role assignments. A Confluent Policy Library is provided to help users adopt.

This release details ongoing community discussions and upcoming features for Apache Kafka 0.10.0.0, including KIPs related to fetch response size limits, timestamp searching in the new consumer, adding record headers, supporting multiple SASL authenticated Java clients in a single JVM, and dynamic JAAS configuration for Kafka clients. It also mentions Google's use of Kafka in GCP, Strata announcements, O'Reilly's insights on streams architecture, a customer 360 architecture tutorial, testing Kaf. This post details how Confluent Cloud reduces the cost to run data streaming workloads by achieving efficient resource utilization (up to 3x compared to self-managing on your own), while reducing the complexity for customers to manage and scale their clusters. It details how multi-tenancy, serverless abstractions, and elasticity are leveraged to improve overall service utilization and lower the infrastructure costs of running Kafka. Confluent Cloud uses serverless abstractions across compute, storage, and networking to continually improve system utilization while saving customers time and complexity. Elasticity is achieved by decoupling compute and storage layers, enabling dynamic scaling and continuous rebalancing of partitions across clusters.

This post introduces Multi-Datacenter Replication, a new capability within Confluent Platform that automates data replication between Kafka clusters across multiple datacenters. It addresses common pain points with MirrorMaker by providing dynamic topic creation with matching partition counts and replication factors, automatic resizing and reconfiguration of topics, topic selection via whitelists/blacklists/regex, support for secure Kafka clusters, and scalability/fault tolerance via Kafka Connect API. It integrates with Confluent Control Center for configuration, deployment, and monitoring. The implementation leverages a Kafka source connector and Kafka Connect for robust and scalable replication, enabling use cases like data aggregation for analytics, disaster recovery backups, and datacenter migration.

This release enhances Confluent Cloud's capabilities for building secure shared services platforms. It introduces Cluster Role-Based Access Control (RBAC) for granular management access to resources like production environments and sensitive clusters, complementing existing ACLs for application access to Kafka resources. Audit Logs are now generally available, enabled by default, to track user/application resource access for anomaly detection and bad actor identification, with logs available in Confluent Cloud. This post details the security features of Kafka and Confluent Platform, covering authentication (SASL, mTLS), authorization (ACLs, RBAC), and encryption (data in transit, secrets management). It provides configuration examples for Kafka brokers, ZooKeeper, and HTTP services, and explains how RBAC integrates with Confluent's Metadata Service for centralized authorization.

This post details building streaming applications using Apache Kafka and the Python client library, leveraging the Task Parallel Library (TPL) for in-process parallelization. It details the use of TPL Dataflow blocks (BufferBlock and TransformBlock) to create a pipeline for consuming, processing (including simulated CPU-intensive tasks), and producing data, emphasizing throughput optimization and maintaining data order. This post also introduces using Rust with Apache Kafka, exploring the `kafka` client.

This post details AppsFlyer's implementation of a human-readable and actionable Apache Kafka lag monitoring system using Burrow. It covers the challenges of monitoring Kafka lag in a large-scale event-driven architecture, the basic concepts of Kafka offsets and lag, and the decision to use Burrow due to its modular design, flexibility in metric calculation, and ability to support both __consumer_offsets and ZooKeeper consumers. The post also touches upon the importance of visibility in distributed systems and the need for actionable metrics to quickly diagnose and resolve issues.

This post details the implementation of derivative event sourcing using Change Data Capture (CDC) and Kafka Streams to handle multiple, disparate order services. It addresses the challenge of integrating legacy systems by observing database changes (Oracle GoldenGate, Debezium for MySQL) and transforming them into a unified event stream. The approach simplifies event sourcing in complex environments by decoupling event emission from the original services, allowing for centralized transformation and emission to Kafka topics. This enables a more manageable and scalable event-driven architecture, especially when dealing with older or acquired applications.

Stream Lineage provides an interactive, graphical UI to visualize data flows in Apache Kafka and event-driven architectures. It automatically maps producers, consumers, Kafka topics, Connectors, Kafka Streams applications, and ksqlDB streams/tables. Users can trace data origins, transformations, and destinations, search for specific components, and adjust time windows up to 24 hours. Use cases include developing data pipelines, monitoring and troubleshooting data flows, and supporting compliance. This post introduces the Stream Governance suite, which includes Stream Lineage, Stream Catalog, and Stream Quality features. Stream Catalog enables classification and discovery of event streams, while Stream Quality focuses on data integrity and validation through features like schema linking and broker-side schema ID validation.

This release enhances ksqlDB with highly available state for queries during cluster rebalances by routing queries to standby servers. It also introduces new aggregate and table functions (count_distinct, cube), supports primitive data types (INT, BIGINT, DOUBLE) for record keys, and adds first-class constructors for native arrays, maps, and structs. Performance improvements and bug fixes are also included. This post details the engineering efforts to make pull queries remain highly available in ksqlDB, and demonstrates a practical application of ksqlDB, Kafka, and Quarkus for real-time sports tracking, including event sourcing, player score computation, and match result processing.

This post details building an end-to-end IoT motion detection and alerting system using Confluent Cloud and ksqlDB. It covers hardware choices (Raspberry Pi Zero W), software components (Motion, librdkafka, Docker), and the architecture for ingesting motion events into Kafka. The system utilizes ksqlDB for real-time stream processing and querying, enabling both real-time monitoring with push queries and historical event search with pull queries. The UI is built with React and Next.js, demonstrat

This release details how to implement message prioritization in Apache Kafka, a feature not natively supported due to its event streaming architecture. The post explains why Kafka's commit log and partition-based design preclude direct message prioritization. It then introduces the 'bucket priority pattern' as a viable workaround, where messages are grouped into different topics (buckets) based on priority during production. This allows consumers to process messages from higher-priority topics first. This post details the evolution of Tesco's Risk Engine from a JMS-based system to an event-driven architecture using Kafka and Kafka Streams to handle real-time risk management and fraud detection. It highlights the challenges faced with the initial JMS implementation, including lack of scalability, persistence, and concurrency issues. The rebuild focused on a new event-driven design with finer-grained topics and the use of Kafka Streams for stream processing. Initial attempts with a Session Aggregator faced issues with high memory demand, state management, and rebalancing. The improved design emphasizes multiple independent and parallel pipelines, focusing on specific goals like blocking malicious IPs, rather than a monolithic aggregate. The post details the technical challenges and solutions encountered during the migration and redesign, including data skew, large message sizes, and the limitations of early Kafka Streams features. It also touches upon the use of ksqlDB for specific tasks like calculating user account activity.

This release details how to use Apache Kafka command-line tools with Confluent Cloud, enabling users to leverage familiar tools for managing topics, consumers, and consumer groups. It covers creating configuration files with API keys and secrets, and provides examples for kafka-console-producer, kafka-console-consumer, and kafka-consumer-groups. The post also addresses connectivity troubleshooting and highlights the availability of Confluent Platform tools for hybrid and multi-cloud environments.

This release enhances Confluent Cloud's capabilities for building real-time, high-throughput UIs that visualize Kafka message streams. It details the evolution from traditional pagination and long polling to WebSockets and then Web Workers to achieve performant message consumption and rendering in a browser. The post highlights the challenges of UI responsiveness with high message throughput and the technical solutions implemented to overcome them, including optimizing state management (Redux to React components) and offloading processing to web workers.

This post introduces the concept of using GraphQL as an API layer for accessing data stored in Apache Kafka. It clarifies common misconceptions about both technologies and explores how they can be used together to bridge the gap between backend Kafka data and frontend applications. The post details GraphQL basics, including its query language, schema definition, introspection, and operation types (query, mutation, subscription). It also addresses misconceptions about GraphQL's database requirements, ownership, comparison to REST, and its applicability beyond frontend JavaScript development. For Kafka, it clarifies that it's more than just a message queue, highlighting its immutability, retention policies, and the extensive ecosystem (Kafka Streams, Kafka Connect) that enables advanced stream processing and data integration. The post suggests that GraphQL subscriptions could be a scalable fit for Kafka's immutable event streams, especially when combined with Kafka Connect for CDC and other data sources.

This release details the integration of Apache Kafka with Apache Mesos, focusing on running Kafka clusters on Mesos to simplify the management of stream data at scale. It introduces a Kafka Mesos framework with a scheduler and executor, providing operational automation, task failure handling, and scaling capabilities. The post contrasts this framework approach with using Marathon, highlighting Marathon's limitations for stateful services like Kafka. It also provides a quickstart guide for installation and administration via CLI and REST API, and discusses future improvements such as auto-scaling and rack-aware partition assignment.

This release enhances Confluent Cloud's capabilities for building secure shared services platforms. It introduces Cluster Role-Based Access Control (RBAC) for granular management access to resources like production environments and sensitive clusters, complementing existing ACLs for application access to Kafka resources. Audit Logs are now generally available, enabled by default, to track user/application resource access for anomaly detection and bad actor identification, with logs available in

This release details enhancements to Confluent Control Center for managing large-scale Kafka deployments. Key improvements include better visualization and dynamic modification of broker configurations without requiring restarts, addressing challenges in distributed systems. Additionally, Control Center's scalability has been significantly improved to manage a larger number of partition replicas, covering over 90% of existing customer workloads. This post complements the existing thread on managing multiple Kafka Connect and KSQL clusters by focusing on the core Kafka broker management aspects.

This post introduces a containerized setup for testing Kerberos authentication with SSH, enabling passwordless authentication using Kerberos tickets. It details the configuration of KDC, SSH server, and SSH client containers, along with methods for principal and keytab management. This capability is relevant for testing Confluent connectors that support Kerberos authentication in a local development environment.

This post details building an asyncio-enabled Kafka producer and integrating it with FastAPI web applications. It covers creating a wrapper class for the Confluent Python client to support asyncio, handling delivery notifications, and implementing out-of-band error handling. The integration with FastAPI demonstrates how to use the producer in startup events and endpoint handlers, with examples of awaitable produce methods and load testing with Siege.

This post details Zendesk's implementation of a self-hosted mTLS authentication system for their Kafka clusters to secure inter-pod communication and enable a global event bus. It covers the problem statement, key requirements like automated certificate generation and rotation, and the decision to use mTLS over SASL. The solution components include using HashiCorp Vault as a private CA, a PKI auth manager sidecar for Kubernetes clients, and a TLS monitor for brokers to handle certificate hot-reloading. It also discusses security considerations like certificate revocation challenges and a robust root CA rotation strategy.

This release details the foundational security features added in Apache Kafka 0.9, including client authentication (TLS/Kerberos), authorization (ACLs), network encryption (TLS), and broker-ZooKeeper authentication. It details the configuration steps for TLS and Kerberos, including certificate generation and JAAS configuration, laying the groundwork for secure Kafka deployments. Confluent Cloud enforces TLS for all traffic, encrypts data at rest with support for BYOK, and uses SASL PLAIN for Kafka authentication. It also supports SSO for centralized identity management and RBAC for management operations, complementing Kafka ACLs for data access. An audit log tracks changes to clusters.

This post introduces a modified Python producer that adds cryptographic hashes and digital signatures to message headers, allowing consumers to cryptographically prove the integrity of streamed data. This enhances security and trust in data pipelines by enabling verification against tampering.

This post details common Kafka client connectivity issues, focusing on the role of `advertised.listeners` and how to correctly configure it across various deployment scenarios (local, different machines, Docker). It provides practical examples and troubleshooting steps to ensure clients can successfully connect to Kafka brokers.

This post introduces Bincover, an open-source tool developed by Confluent to measure code coverage for Go binaries, particularly for end-to-end and system tests. It addresses limitations of the built-in `go test` tool by enabling coverage measurement for binaries that execute `main()` and handle panics or `os.Exit()` calls. Bincover instruments binaries to collect coverage profiles, supports passing command-line arguments and environment variables, and merges profiles from multiple test runs. This capability is crucial for ensuring the robustness and testability of Go-based Confluent products like CLIs.

This post introduces Mox, a tool developed by Confluent to address the 'blocked factor' in API-driven development. Mox acts as a hybrid proxy and mocking framework, allowing developers to proxy requests to a live backend while selectively mocking specific endpoints. This approach aims to balance the flexibility of API mocks with the maintainability of using a live backend, reducing developer downtime and improving iteration speed. The post explores the trade-offs between pure mocking and live backend approaches and details Mox's features for request interception, modification, and proxying to multiple backends.

This post introduces CP-Ansible, open-source Ansible playbooks for deploying Apache Kafka and Confluent Platform. It details the benefits of Ansible for automation, configuration management, and agentless deployment. The post outlines how to use CP-Ansible for setting up components, applying best practices for security and logging, and customizing installations. It provides a step-by-step guide for deploying a cluster with SASL PLAIN security and validating the installation via Confluent Control Center.

This post introduces the sticky partitioning strategy for Apache Kafka producers, a new feature in version 2.4. This strategy aims to improve producer latency by optimizing batching, particularly for records with null keys. It addresses the inefficiency of the previous round-robin approach for null keys, which led to smaller batches and increased latency. The sticky partitioner selects a single partition for non-keyed records until the batch is full or a timeout occurs, then randomly selects a new partition. This results in larger batches and reduced latency, especially as the number of partitions increases. Performance tests using the Castle framework and Trogdor benchmarks demonstrate significant reductions in p99 latency and decreased CPU utilization compared to the default partitioning strategy. The post also explores the performance of the sticky partitioner with keyed messages and different linger.ms settings, showing consistent or improved latency.

This release details advanced error handling strategies for Kafka Connect, including "fail fast" (errors.tolerance = none), silent ignoring (errors.tolerance = all), and routing problematic messages to a Dead Letter Queue (DLQ) topic. It explains how to configure DLQ topics, leverage message headers for detailed error context (errors.deadletterqueue.context.headers.enable = true), and inspect DLQ messages for debugging and reprocessing. The post also touches upon monitoring error rates via JMX metrics.

Introduces Incremental Cooperative Rebalancing for Kafka clients (Consumer, Connect, Streams) to improve load balancing efficiency and reduce disruption during rebalances. This new protocol allows for staged convergence to a balanced state and avoids 'stop-the-world' rebalancing, particularly beneficial for large-scale deployments, multi-tenancy, and dynamic environments like Kubernetes. Kafka Connect is the first client to implement this, treating tasks as lightweight threads for more flexible rebalancing. Also includes logging improvements using Mapped Diagnostic Context (MDC) for better context in worker logs, consolidated REST API endpoints for connector status and info, distinct client.id per task for easier diagnostics, and connector-level producer/consumer configuration overrides.

This release details the fundamental mechanics of Kafka consumers and consumer groups, including their role in scalability, availability, and consistency. It explains how consumers pull data, the concept of offsets, and how offsets are managed using the `__consumer_offsets` topic. It also covers the high watermark offset for consistency and the rebalancing process within consumer groups to ensure load balancing and fault tolerance. The post introduces Kafka's membership API, which underpins these functions. The new consumer client in 0.9.0 introduces a consolidated API, reduced dependencies (pure Java, no Zookeeper), and better security support, along with a new group coordination protocol built into Kafka itself, simplifying cross-language client development.

This post details the results of a survey on Apache Kafka client usage, highlighting language adoption trends and user priorities for client selection. Key findings include the dominance of Java, Scala, and Python, with a surprising interest in Clojure. Reliability and performance were identified as critical factors for client choice, followed by usable APIs and documentation. Security and compatibility were also important, though ranked slightly lower. The survey provides valuable insights for client development and selection.

This release details the GA for Kafka release 0.11.0.0, bringing Exactly Once Semantics (EOS) with more stringent idempotent producer semantics, exactly once, in-order delivery per partition, and stronger transactional guarantees with atomic writes across multiple partitions. It also highlights the growing adoption of Kafka Connect API with new community-contributed connectors for AWS S3 sink, Redis source, RabbitMQ source, GitHub source, Amazon Kinesis source, and MongoDB sink. This post details LinkedIn's deployment of Apache Kafka surpassing 1.1 trillion messages per day, showcasing its evolution from a user activity data pipeline to the company's central nervous system for data. It highlights Kafka's role in powering critical data pipelines, real-time analytics, and stream processing for numerous companies.

This release details the redesign of Kafka's request purgatory to improve performance and reduce memory usage. The old implementation using DelayQueue had deficiencies in handling large numbers of asynchronous requests, leading to potential OutOfMemoryErrors and performance degradation due to frequent purging. The new design leverages Hierarchical Timing Wheels and doubly linked lists for O(1) insert/delete operations, significantly improving throughput and reducing CPU load. Benchmarks show a dramatic improvement.