Kafka Broker and Streams Enhancements

5/31/2026

This post details the release of Apache Kafka 4.3, highlighting 25 KIPs and over 600 commits. Key technical contributions include: KIP-1023 (Follower fetch from tiered offset), KIP-1066 (Mechanism to cordon brokers and log directories), KIP-1196 (group.coordinator.append.max.buffer.size config), KIP-1208 (Prefix for RemoteLogMetadataManagerConfig), KIP-1211 (Align num.partitions and default.replication.factor), KIP-1219 (KRaft Fetch and FetchSnapshot Byte Size configs), KIP-1235 (Default min.insync.replicas for __remote_log_metadata topic), KIP-1240 (Additional group configurations for share groups), KIP-1251 (Assignment epochs for consumer groups), KIP-1257 (Partition Size Percentage Metrics), KIP-1258 (OAuth Client Assertion for client_credentials), KIP-1263 (Group Coordinator Assignment Batching and Offload), KIP-1274 (Deprecate Classic rebalance protocol), KIP-1035 (StateStore managed changelog offsets), KIP-1247 (Bytes utils class as public API), KIP-1250 (Metric for in-memory state stores), KIP-1259 (Wipe Kafka Streams local state on startup), KIP-1270 (Extend ProcessingExceptionHandler for GlobalThread), KIP-1271 (Store Headers in State Stores), KIP-1285 (DSL Opt-in for Headers-Aware State Stores), KIP-1239 (Batch offset translation in RemoteClusterUtils), KIP-1273 (Improve Connect configurable components discoverability), and KIP-1280 (Update MirrorMaker metrics). Deprecations for streams-scala, group.coordinator.rebalance.protocols, and MirrorMaker metrics are also noted.

5/26/2026

This post introduces the concept and practical implementation of adding Flink streaming transformations to Kafka for real-time data processing. It details the shift from passive to active data streams, contrasting batch processing with continuous streaming. The post outlines the benefits of Flink for real-time data pipelines, including reduced latency, simplified architectures, and centralized transformation logic. It provides a step-by-step guide for implementing a first Flink transformation, focusing on stateless operations like filtering and field projection, and introduces common first transformations such as event filtering, field projection/reshaping, real-time enrichment, and introductory event aggregation. The distinction between stateless and stateful transformations is also discussed.

5/25/2026

Introduces the concept and architecture of agentic event-driven systems, which are AI-native architectures where software agents continuously sense events, reason over shared state, take actions, and learn from outcomes in real time. This architecture extends traditional event-driven systems by embedding decision intelligence and control loops directly into the event flow, enabling autonomous operation. Key components include an event-driven backbone (e.g., Apache Kafka), stateful stream processing (e.g., Apache Flink), an agent execution layer for reasoning and decision-making, a shared state and context layer, and an orchestration and policy engine. The architecture emphasizes closed-loop control patterns where actions generate outcome events that feed back into the decision process for continuous adaptation and optimization.

5/25/2026

This post details the application of event streaming (Kafka) to build a secure, compliant Retrieval-Augmented Generation (RAG) system for regulated sectors. It outlines a streaming RAG ML pipeline that uses Change Data Capture (CDC) for real-time data ingestion from databases, a governance layer for PII masking, schema enforcement via a schema registry, and tokenization. It also describes secure vector index retrieval with Context Scoping, LLM integration with safety guardrails, and an observability layer for response provenance. The post contrasts batch RAG with real-time streaming RAG, highlighting the latter's advantages in data freshness, compliance timing, audit complexity, and policy enforcement for regulated environments.

5/14/2026

Introduces Tableflow, a new capability that directly converts Apache Kafka topics into analytics-ready tables in formats like Apache Iceberg and Delta Lake. This feature aims to reduce data ingestion costs by 30-50% compared to traditional ETL pipelines. It achieves this by reusing Kafka segments stored in Confluent Cloud's Kora engine, converting them to Parquet, automatically handling schema evolution and CDC, compacting small files, and publishing tables to external catalogs. The post details the cost drivers of traditional ingestion stacks and highlights how Tableflow addresses them, including Kafka services and storage, connect infrastructure, lakehouse compute, and engineering overhead. It also provides benchmark data comparing Tableflow to managed lakehouse solutions and discusses architectural benefits like openness, operational simplicity, near-real-time data, and shift-left processing.

5/14/2026

Introduces new observability metrics for Confluent Cloud Kafka: `io.confluent.kafka.server/client_limit_milliseconds` to identify users breaching limits and reasons, `io.confluent.kafka.server/max_pending_rebalance_time_milliseconds` to monitor consumer group rebalance duration and frequency, `io.confluent.kafka.server/connection_accept_count` to track connection attempts for elastic cluster scaling, and a new `cleanup.policy` dimension for `io.confluent.kafka.server/partition_count` to differentiate compacted partitions. Also details a refreshed cluster monitoring experience in Confluent Cloud Console.

4/23/2026

This post details the integration of Confluent Cloud Gateway with KCP to automate Kafka client migrations. It explains the challenges of client migration (coordination, authentication, offset management, rollback) and how Gateway addresses them by acting as a proxy and performing authentication translation. The post outlines a four-stage client migration strategy (Deploy, Plan, Onboard, Migrate) and provides the specific KCP commands for initializing the migration, including parameters for source/destination cluster details, Gateway CR YAMLs, and authentication methods. It highlights the benefits of Gateway's authentication swapping feature and the seamless cutover process with minimal downtime.

4/16/2026

Introduces native queue semantics for Kafka, enabling elastic consumer scaling and per-message acknowledgment for task-processing workloads. Flink SQL is now GA in Confluent Platform, offering a declarative stream processing experience with DDLs, changelogs, and shared compute pools. Confluent Private Cloud Gateway 1.2 adds intelligent fencing and unfencing for migration and failover, and protocol-level auth swapping. Native Dead Letter Queue (DLQ) support is added for handling unprocessable records.

3/16/2026

This post details how Confluent's data streaming platform, including Freight clusters, WarpStream, and Tableflow, enables the creation of 'Open Security Lakes' using Apache Iceberg. It explains the architectural shift from traditional SIEMs to a decoupled model where data streaming handles ingestion and routing, and Iceberg manages storage for forensic data. The post highlights the cost-effectiveness of Freight clusters for high-throughput logging and WarpStream's BYOC model. It also discusses the integration with Apache Flink for real-time threat detection and the use of Tableflow to convert Kafka topics into Iceberg tables with minimal friction.

3/12/2026

This post details the limitations of traditional batch ELT pipelines at enterprise scale, highlighting issues like cascading failures, schema drift, resource inefficiency, and orchestration chaos. It advocates for a streaming-first data integration architecture using Kafka as a durable, scalable, and reusable data movement layer. The post explains how Kafka, combined with connectors and Tableflow, Schema Registry, and open table formats like Iceberg/Delta Lake, can unify operational and analytical data, enabling write-once-read-anywhere data ingestion and reducing the complexity and cost associated with traditional ELT.

3/10/2026

Introduces the capability to store Schema Registry schema IDs in Kafka headers, enabling backward-compatible schema attachment to Kafka records. This feature allows for schematizing existing topics without breaking legacy consumers or changing payload formats, reducing data errors, and improving data usability for analytics and AI. It details producer and consumer behavior changes, the new 16-byte GUID format for schema IDs in headers, and broad ecosystem support. Migration paths for different starting points are also outlined.

3/5/2026

Introduces Queues for Kafka, a new capability built on KIP-932, enabling queue semantics within Kafka. This includes the share consumer API and share groups, allowing multiple consumers to process messages from the same topic elastically, independent of partition count. It implements per-message acknowledgment, release, rejection, and lock renewal mechanisms. The post details the infrastructure consolidation benefits, elastic scaling advantages, per-message processing controls, and Kafka's durability guarantees. It also provides code examples for both implicit and explicit acknowledgment modes of the KafkaShareConsumer and discusses its availability and management on Confluent Cloud and Confluent Platform.

2/26/2026

Introduces KCP, an open-source tool for automating Kafka migrations to Confluent Cloud. KCP orchestrates four key steps: discovery and planning (automating inventory of clusters, topics, ACLs, schemas, connectors, and clients), provisioning infrastructure (generating Terraform scripts for Confluent Cloud resources and networking like PrivateLink), data migration (using Cluster Linking for topic data, ACLs, schemas, and connectors), and client migration (planned for H1 2026). The post details KCP commands for discovery, reporting costs/metrics, and scanning various components. It also highlights the use of Cluster Linking over PrivateLink for direct private connections.

2/20/2026

Introduces production-ready Kafka Queues (Share Groups) with RENEW acknowledgement, adaptive batching, and lag metrics. Kafka Streams gains GA server-side rebalance protocol, dead letter queue support, anchored wall-clock punctuation, and explicit control over leave group requests. Enhancements include standardized CLI arguments, corrected metric naming, new idle ratio metrics, an allowlist connector client configuration override policy, and thread-safety improvements to RecordHeader.

2/10/2026

This post details a Proof of Concept (POC) for building Kafka client failover using Confluent Cloud Gateway to achieve disaster recovery within 60 seconds. It explores the challenges of RPO/RTO in Kafka outages, discusses server-side replication options like Cluster Linking, and highlights the complexities of client-side failover due to the static `bootstrap.servers` configuration. The POC demonstrates how Confluent Cloud Gateway can act as a traffic control layer to redirect client traffic to a passive cluster during an outage without client restarts. It also addresses the issue of read-only mirror topics and proposes a workflow involving connection dropping, reversing replication, and gateway switching. The architecture of the demo service is presented, emphasizing identity passthrough and OAuth2 support for failover.

1/29/2026

This post contrasts streaming data integration with ETL, arguing that Kafka's original use case of a centralized log for broadcasting and sharing data in real-time aligns with modern needs for interconnected data products. It details the limitations of ETL, such as data silos, lack of reusability, batch processing, and high latency, and highlights the advantages of streaming data integration with Kafka, including one-to-many and many-to-one source/destination support, reusable canonical streams, unification of real-time and historical data analysis, and the ability to build richly interconnected data products. It also suggests incorporating ETL best practices like intuitive workflows and reduced toil into streaming data integration.

1/16/2026

This post details the technical aspects of migrating from hosted Kafka to Confluent Cloud, focusing on the KRaft era and the role of Kafka Copy Paste (KCP) and Cluster Linking. It explains how KCP simplifies migration planning and provisioning using Terraform, and how Cluster Linking enables byte-for-byte mirroring of topics, preserving partitioning, ordering, and offsets. The post also touches on the underlying mechanisms of Cluster Linking for continuous data propagation, error handling, and its application in global and hybrid cloud scenarios. It highlights cost savings through features like Kora, Private Network Interface (PNI), and Freight clusters, and emphasizes the reliability benefits of Confluent Cloud's 99.99% uptime SLA.

12/23/2025

This post details the hidden costs of manual Apache Kafka cluster rebalancing, including resource drain, operational overhead, downtime risk, and opportunity cost, particularly at scale. It highlights how Confluent's automated and self-balancing clusters eliminate these costs through non-disruptive partition redistribution, continuous monitoring, and predictive scaling, leading to reduced TCO and improved operational efficiency.

11/6/2025

This post details Superwall's migration from a self-managed Kafka and ClickHouse OSS stack to WarpStream (a Kafka-compatible streaming platform) and ClickHouse Cloud. It highlights the use of ClickPipes for managed ingestion from WarpStream to ClickHouse Cloud, addressing challenges with disk space, cost, and operational overhead. The post emphasizes the benefits of a fully managed, elastic, and ops-light data foundation for their monetization platform, enabling them to handle high event volumes and large datasets with confidence.

10/29/2025

This post details the cost drivers and mitigation strategies for Apache Kafka migrations, highlighting the benefits of Confluent Cloud and Cluster Linking for reducing engineering hours, downtime risk, and operational overhead. It emphasizes phased migrations, scope reduction, and parallel testing environments.

10/27/2025

This post delves into the Total Cost of Ownership (TCO) of data streaming, specifically contrasting self-managed Kafka with Confluent Cloud. It quantifies infrastructure, operations, engineering, governance, and opportunity costs, providing a detailed breakdown of monthly estimates for each scenario. Key Confluent Cloud features like eCKUs (Elastic Compute Units for Streaming) and elastic storage are highlighted as drivers of cost efficiency and scalability. The post also analyzes the latency-cost tradeoffs between batch processing and real-time streaming, emphasizing how streaming reduces critical business risks and operational costs.

10/14/2025

This post introduces Confluent's new capability for secure cross-cloud data replication over private networks, powered by Cluster Linking. It details how this feature enables data movement between AWS, Azure, and Google Cloud without public internet exposure, addressing security and compliance concerns for multicloud strategies. The post contrasts this with the older Confluent Replicator method, highlighting the simplified, managed nature of Cluster Linking for private networking scenarios. It also provides a practical demonstration using Terraform and Confluent CLI for setting up and configuring private cross-cloud replication, focusing on use cases like data sharing and disaster recovery.

10/1/2025

This post details cross-data-center replication strategies for Apache Kafka using Kafka MirrorMaker. It outlines decision frameworks for replication patterns (active-passive, active-active), scope (within-region vs. across-regions), and recovery goals (RTO/RPO). It provides a step-by-step operational guide for setting up and configuring MirrorMaker 2, including best practices for addressing common issues like configuration errors, replication lag, memory constraints, and broker connectivity failures. Key features of MirrorMaker 2 for resilient multi-cluster architectures are also highlighted.

10/1/2025

This post details strategies for scaling Kafka Streams applications for high-volume data processing, focusing on parallelism through partitioning, scaling out vs. scaling up, fine-tuning configuration parameters like num.stream.threads and RocksDB settings, and considerations for stateful vs. stateless applications. It also highlights key monitoring metrics such as consumer lag and CPU utilization.

9/30/2025

This post details how to ensure data quality with real-time validation and monitoring using Apache Kafka and Confluent. It introduces a two-layer approach: validation (ensuring data conforms to structure and business rules in motion) and monitoring (tracking health metrics for trends and anomalies). Key technical components discussed are Schema Registry for schema enforcement at ingestion, and Apache Flink or ksqlDB for real-time business rule checks. It also covers integrating monitoring tools like Grafana or Datadog for visibility and actionable insights. The post emphasizes a 'shift left' strategy for data quality, moving validation closer to the source to prevent bad data from propagating.

9/30/2025

This post details how to build real-time alerts using Apache Kafka, covering threshold-based, anomaly detection, composite event, and automated action alerts. It emphasizes designing trustworthy alerts by addressing alert fatigue through filtering and prioritization, and discusses the importance of timely and actionable notifications for proactive system monitoring and response.

9/30/2025

This post provides a comprehensive guide to deploying, scaling, and securing Kafka Connect in production. It details the distributed mode architecture, Kubernetes deployment considerations, strategies for scaling workers and tasks, tuning for throughput (data formats, batching, SMTs, worker resources), error handling patterns (backpressure, Dead Letter Queue, retries, idempotency), and security best practices including transport encryption, authentication, authorization, and secret management using Secret Providers.

9/30/2025

This post explains how event-driven architectures, enabled by technologies like Kafka, solve the challenges of microservices communication that arise from synchronous API-based patterns. It details how asynchronous event streams decouple services, prevent cascading failures, reduce latency, and improve scalability. The post also discusses the business benefits and common industry use cases for event-driven microservices, and introduces the concept of data governance for managing event streams at scale.

9/17/2025

This post details how to build real-time Apache Kafka dashboards that drive action by leveraging Kafka as an integration hub and stream processing engines like Flink, Kafka Streams, or ksqlDB to create materialized views for immediate consumption by BI tools or custom applications. It emphasizes data modeling for real-time performance, including denormalized schemas and effective Kafka partition strategies, and discusses delivery patterns such as read-optimized views, search-optimized indexes, and app-embedded state using Kafka Streams interactive queries.

9/4/2025

Apache Kafka 4.1.0 introduces preview support for Queues for Kafka (KIP-932) and early access to a new Streams Rebalance Protocol (KIP-1071) based on KIP-848. Broker and client improvements include plugin metric registration (KIP-877), consistent transaction error handling (KIP-1050), explicit consumer group leaving on close (KIP-1092), rack topology change rebalance triggers (KIP-1101), unified consumer topic metrics (KIP-1109), deadlock protection for producer network threads (KIP-1118), OAuth JWT-bearer grant type support (KIP-1139), endpoint listener name cleanup (KIP-1143), and transactional ID pattern filtering for ListTransactions (KIP-1152). Kafka Streams enhancements include moving window configurations to SerDes (KIP-1020) and enforcing explicit naming for internal topics (KIP-1111). Kafka Connect now supports running multiple versions of connector plugins (KIP-891).

6/27/2025

Introduced a new topic message browser with enhanced capacity (1 million messages), human-readable ISO timestamps, and a histogram for message distribution. Added schema-compliant message production directly within the console. Overhauled the clients overview page to display client IDs, software types/versions, and connection metrics, specifically highlighting deprecated requests related to KIP-896. Enhanced consumer group monitoring with time charts for production, consumption, and lag, and enabled direct offset resetting from the console. Introduced a dedicated Kafka Streams view for application monitoring. Added functionality to export topic lists to CSV, view consumer group protocol types, and display schema IDs in the message browser.

6/24/2025

Confluent Platform 8.0 introduces KRaft as the default metadata management system, replacing ZooKeeper, which simplifies Kafka architecture, improves scalability (millions of partitions), and enables near-instant controller failover. The next-generation Confluent Control Center is now Prometheus-based, offering significant performance improvements: 2x faster metrics, support for up to 400K partitions, metrics freshness within 2-3 minutes, and 15x faster startup times (1 minute). It also integrates Confluent Manager for Apache Flink (CMF) for centralized management of Flink environments and applications. Client-Side Field Level Encryption (CSFLE) is now Generally Available, providing granular encryption of individual message fields for enhanced data protection and compliance. FlinkSQL is available in Open Preview via REST and CLI. Kafka Queues (Share Groups) are in Early Access, offering a native, cooperative consumption model for queue-style workloads. Confluent Community software will now align more closely with the Kafka release cycle. Passwordless authentication is now supported for Confluent Server and Schema Registry using OAuth client credentials grant type with pre-signed assertions. CFK 2.11.1 and Confluent Ansible 8.0.0 have been released to support these updates, with CFK allowing side-by-side deployment of new and legacy Control Center.

6/3/2025

Introduces KIP-848, a new, optional consumer rebalance protocol for Apache Kafka 4.0+. This protocol shifts coordination from clients to the broker-side group coordinator, enabling server-driven, incremental reconciliation. Key changes include a shift from stop-the-world rebalances to incremental partition assignment/revocation via heartbeats, server-side assignors, and dynamic server-side configuration of session and heartbeat intervals. This results in reduced consumer downtime, faster rebalances, improved stability, and simplified management. The post details the protocol's mechanics, benefits, and adoption steps, including support for live upgrades.

5/20/2025

Introduces a new Prometheus architecture for Confluent Control Center, enabling support for up to 400K partitions, reducing start-up time to 1 minute, and achieving metrics freshness within 2-3 minutes. Eliminates the need for a separate Kafka cluster for metrics storage.

5/1/2025

This post details the architecture and implementation of a multi-agent orchestrator using Apache Flink and Apache Kafka. It explains how Flink can be used for real-time message interpretation and routing, with Kafka serving as the messaging backbone and short-term shared memory. The post outlines a reference architecture for event-driven multi-agent systems, including message production, Flink processing and routing via LLM inference, and agent execution. A practical example of an AI-driven sales development representative (SDR) system is provided, showcasing how Flink SQL can be used to dynamically assign tasks to different agents based on message content and agent descriptions.

4/21/2025

This post introduces KIP-1094, which adds a new constructor method with nextOffsets to ConsumerRecords, enabling more accurate and reliable retrieval of offset and leader epoch information. It also elaborates on manual offset committing strategies (commitSync, commitAsync) and highlights the critical importance of committing the leader epoch alongside the offset to ensure consumer group consistency and prevent issues like OFFSET_OUT_OF_RANGE errors during leader changes and rebalances.

3/27/2025

This post introduces Confluent Cloud Freight clusters as a cost-effective solution for high-throughput, relaxed latency Kafka workloads, up to 90% cheaper than self-managed Kafka. It details how to integrate these serverless clusters with AWS Lambda functions using various patterns (managed/self-managed connectors, Lambda ESM, EventBridge Pipes) for serverless stream processing. The post provides a specific e-commerce use case for ingesting customer actions and sending survey links, highlighting the use of Lambda's Kafka Event Source Mapping (ESM) with Provisioned Mode for fine-tuned throughput control, VPC-connected Lambda functions for private resource access, and event filtering for cost optimization.

3/18/2025

Apache Kafka 4.0 defaults to KRaft mode, deprecates ZooKeeper, and introduces a new consumer rebalance protocol for faster rebalances. Queues for Kafka (early access) enable traditional queue semantics. Eligible Leader Replicas (ELR) and Pre-Vote mechanisms enhance KRaft stability. Client metrics collection is extended, and duration-based offset reset is supported. Group administration is improved with new command-line tools. Client resilience is enhanced with rebootstrap capabilities. Support for old client protocol API versions, message formats v0/v1, and Java 8 is dropped. Kafka Streams allows foreign key extraction from both key and value in KTable joins, custom processor wrapping, and adds a 'retry' return-option to ProductionExceptionHandler.

3/13/2025

This post traces the evolution of Confluent's offerings from Apache Kafka to a comprehensive Data Streaming Platform (DSP). It outlines three 'acts': Confluent Platform (focusing on self-management, security, monitoring, replication), Confluent Cloud (emphasizing SaaS-like offering, elasticity, pay-as-you-go), and the current DSP (defining it as a platform that connects, streams, processes, and governs data in motion). The DSP's core components are detailed: Stream (Kora engine), Connect (120+ connectors), Process (Apache Flink, Tableflow integration with Databricks), and Govern (Stream Quality, Stream Catalog, Stream Lineage, Data Portal). The post highlights the business value derived from these evolutions, citing Forrester TEI reports and emphasizing the shift from technical problem-solving to enterprise-wide data transformation.

2/19/2025

This post introduces four design patterns (orchestrator-worker, hierarchical agent, blackboard, market-based) for building event-driven, multi-agent systems. It details how to adapt these patterns using Apache Kafka for agent coordination, communication, and fault tolerance, moving away from traditional request/response models to a reactive, event-driven approach. The post emphasizes the operational simplifications and scalability benefits gained by leveraging Kafka's data streaming capabilities for agent interactions.

2/12/2025

Introduces Provisioned Mode for AWS Lambda's Kafka Event Source Mapping (ESM), enabling granular control over throughput via minimum and maximum event pollers. This mode simplifies networking by eliminating the need for PrivateLink or NAT Gateway, and provides responsive auto-scaling based on offset lag. Performance benchmarks show significant improvements in message processing time compared to the default On-Demand mode, with specific scenarios demonstrating up to 78.57% faster backlog draining.

2/7/2025

This post details the implementation of an event-driven architecture using Kafka and Flink to automate podcast promotion. It covers downloading podcast episodes, converting audio to text using OpenAI's Whisper model, splitting audio files due to size limits, and using an LLM (GPT-4) via Flink SQL for LinkedIn post generation. The architecture decouples the front-end application from the AI workflow, enabling independent scaling and model swapping. It highlights the use of Kafka topics for inter-service communication and Flink SQL for stream processing and AI model integration.

2/3/2025

Introduces Confluent Cloud Freight clusters, a new cluster type optimized for cost-effective, high-throughput, latency-insensitive workloads. Key technical advancements include autoscaling to optimize resource utilization, direct writes to object storage (e.g., S3) to bypass local storage and inter-AZ replication, and the use of Fetch From Follower to reduce cross-zone traffic costs. The post also highlights the integration with Private Networking Interface (PNI) for secure and cost-effective VPC interconnection. Freight clusters are built on the Kora engine and can scale from 2 to 152 eCKUs.

1/21/2025

This post details how Allium leverages Confluent's data streaming platform to build a foundational data platform for blockchain data. It covers challenges in accessing and processing blockchain data, and how Confluent's managed connectors, Stream Sharing, and cloud-agnostic nature enable Allium to index more blockchains, provide real-time data to customers, and build real-time applications. The future direction involves shifting processing and governance left using Confluent Cloud for Apache Flink and Tableflow to ensure data quality at the source.

12/16/2024

Introduces Confluent's JavaScript Client for Apache Kafka (CJSK), a new officially maintained client. Details its architecture based on librdkafka, API similarities to KafkaJS and node-rdkafka, and support for TypeScript. Provides code examples for producing and consuming messages with Schema Registry integration (Avro serialization/deserialization) and demonstrates CSFLE capabilities. Mentions OAuth integration.

11/7/2024

Introduces Dynamic KRaft Quorums (KIP-853) allowing dynamic addition/removal of controller nodes via tooling or AdminClient API, improving operational flexibility for KRaft mode. Enhances ZooKeeper migration tooling, making Kafka 3.9 the final and best iteration before ZooKeeper mode is removed in 4.0. Makes Tiered Storage (KIP-405) production-ready with new features like dynamic disablement (KIP-950), quotas (KIP-956), and exposed remote offsets (KIP-1005). Adds Kafka Streams config for log summary interval (KIP-1049) and an exception handler (KIP-1033). Kafka Connect improves null handling in transformations (KIP-1040), adds offset control in MirrorSourceConnector (KIP-1031), and introduces a health check endpoint (KIP-1017).

10/4/2024

This post delves into the challenges of handling 'bad data' within event streams, particularly in Apache Kafka, due to its immutable log nature. It contrasts batch processing techniques for data quality management (ETL/ELT, full refresh, incremental processing) with the complexities of streaming. Key takeaways include the reactive nature of data engineering, the reliance on deletion and reprocessing in batch, and the difficulty of applying these methods directly to event streams. The post sets the stage for discussing specific strategies for event streams in a subsequent part.

9/26/2024

This post details how Atomic Tessellator, a computational chemistry startup, uses Confluent and Apache Kafka to build a scalable and flexible platform for molecular research. It describes their transition from a batch-based ETL process using RabbitMQ, Celery, and a Django app to a data streaming architecture. The new architecture enables rapid ingestion of complex molecular data from databases and generative models into Confluent, followed by streaming to downstream "evaluation" applications. This facilitates quick and reliable replication of chemical reactions for catalyst discovery. The post highlights business benefits such as avoiding over-provisioning of compute resources, the ability to tweak pipelines mid-workload, and clear visibility into production pipeline status through JMX monitoring. It also mentions plans to explore Apache Flink for building a "pipeline templates" system.

9/24/2024

This post details the internal lifecycle of a Kafka producer request on the broker, covering the stages of socket receive buffer, network threads, request queue, I/O threads, page cache and disk writes (including segment structure and flush configurations), purgatory for replication (and its relation to acks and replica fetchers), response queue, network thread handoff, and socket send buffer. It also enumerates specific configuration parameters and monitoring metrics for each stage, such as socket.receive.buffer.bytes, socket.request.max.bytes, num.network.threads, NetworkProcessorAvgIdlePercent, queued.max.requests, queued.max.request.bytes, RequestQueueSize, RequestQueueTimeMs, num.io.threads, RequestHandlerAvgIdlePercent, log.flush.interval.ms, log.flush.interval.messages, log.segment.bytes, cleanup.policy, LogFlushRateAndTimeMs, LocalTimeMs, default.replication.factor, replication.factor, replica.fetch.wait.max.ms, num.replica.fetchers, RemoteTimeMs, ResponseQueueSize, ResponseQueueTimeMS, and ResponseSendTimeMs.

9/9/2024

Confluent has acquired WarpStream, a Kafka-compatible streaming solution with a unique architecture. The WarpStream product is designed for BYOC (Bring Your Own Cloud) architectures, offering a balance between ease of use and control. Key features mentioned include offset-preserving replication and direct-to-S3 writes. Confluent plans to invest in security and hardening for WarpStream and integrate it into their existing systems for signup, billing, and account management. The acquisition aims to expand Confluent's offerings to serve customers who want a cloud-native streaming solution within their own cloud accounts.

9/5/2024

This post dives into the technical details of Kafka producer behavior, explaining the serialization process (using serializers and Schema Registry), partitioning logic (default hash-based, sticky, and adaptive strategies), and batching configurations (`batch.size`). It provides insights into how producers prepare data before sending it to brokers, offering configuration parameters and their implications for performance and data distribution.

8/12/2024

Skai re-architected their Kafka data ingestion to drastically reduce topic count for scalability and cost-efficiency in Confluent Cloud. They moved from a per-table topic model to consolidated topics ('united-topic-single-pipeline', 'united-topic-multiple-pipelines', 'united-topic-heavy-load') based on traffic. This involved developing a dynamic lag handling API to promote tables to higher-tier topics and a caching library for external Kafka clients to manage dynamic topic mappings. They also implemented enhanced monitoring to pinpoint lag causes within consolidated topics.

7/29/2024

Apache Kafka 3.8.0 release introduces 17 KIPs, including support for specifying compression levels (KIP-390), preview of the next-generation consumer rebalance protocol (KIP-848), and a Docker image for GraalVM-based native Kafka brokers (KIP-974). It also includes Shareable State Stores for Kafka Streams (KIP-813) and enforcement of tasks.max in Kafka Connect (KIP-1004). ZooKeeper is marked for removal in Kafka 4.0.

7/11/2024

Introduces and demonstrates the use of the confluent-kafka-javascript client, a new library for interacting with Apache Kafka and Confluent Platform. The post details how to use the client for producing messages with manual offset commits and consuming messages, including handling the last clicked state and emitting updates via WebSockets. It showcases the client's two API styles: one based on node-rdkafka and a promisified API similar to KafkaJS. The example application built is a real-time voting app to settle the tabs vs. spaces debate.

6/27/2024

This post details how Apache Kafka is being used at the edge, addressing challenges related to disconnected, intermittent, and constrained networking, low-latency demands, and cost optimization. It explains Kafka's role as a buffer for local data collection and replication, and how stream processing can optimize data sent to the cloud. The post also highlights Confluent's solutions for deploying and operating Kafka at the edge, including Confluent for Kubernetes, Confluent Ansible, and Health+ for observability.

5/22/2024

Introduces Elastic Confluent Units (eCKUs) for Confluent Cloud's Enterprise clusters, enabling serverless, instantaneous autoscaling of Kafka resources. This addresses challenges with fluctuating workloads, unpredictable spikes, and rapid growth by automatically allocating and deallocating resources (throughput, partitions, client connections) as needed, eliminating manual capacity planning and over-provisioning.

5/14/2024

This post applies Confluent's data streaming platform, specifically Kafka, to the financial services domain for T+1 trade settlement. It details how Kafka can ingest, store, and distribute trade data in real-time, overcoming batch processing limitations. Key technical aspects include using Confluent connectors for data ingestion from various financial systems (e.g., Oracle CDC, AMPS), leveraging the Kora engine for elasticity and reliability in handling market volatility, and employing stream processing (e.g., windowed sorting, KTables, point-in-time joins) to enrich and deliver data to siloed processes. It also mentions Role-Based Access Control (RBAC) for security and data governance. The post illustrates how these components enable faster settlement, reduce risk, and meet regulatory requirements.

2/27/2024

Introduces early access support for JBOD disk failure handling in KRaft (KIP-858). Adds broker-side support for client-level metrics via a standardized telemetry interface (KIP-714) and a mechanism to configure these metrics (KIP-1000). Previews the next generation of the Consumer Rebalance Protocol for improved efficiency (KIP-848). Optimizes leader discovery for clients to reduce latency during leadership changes (KIP-951). Provides an official Apache Kafka Docker image (KIP-975). Implements exponential backoff for Kafka client retries to reduce overload after broker failures (KIP-580). Adds new metrics for Tiered Storage monitoring (KIP-963). Enables independent stopping of KRaft processes (KIP-979). Adds server-side transaction verification to prevent hanging transactions (KIP-890). Marks old client protocol API versions (pre-2.1) as deprecated for removal in Kafka 4.0 (KIP-896). Allows AdminClient to communicate directly with the KRaft Controller Quorum for operations like DESCRIBE_QUORUM and INCREMENTAL_ALTER_CONFIGS (KIP-919). Enables dynamic reloading of certificates with different DN/SANs (KIP-978). Adds a `CurrentControllerId` metric to identify the current controller (KIP-1001). Introduces 11 new metrics for measuring KRaft performance (KIP-938). Deprecates Java 11 support for Kafka broker and tools, planned for removal in Kafka 4.0 (KIP-1013). In Kafka Streams, introduces rack-aware task assignment strategies (`balanced_subtopology`) (KIP-925). Expands default DSL store configuration to custom types with support for RocksDB and in-memory stores (KIP-954). Relaxes the non-null key requirement for certain left-joins (KIP-962). Adds a new interface for handling standby task updates (KIP-988). Adds `VersionedKeyQuery` and `MultiVersionedKeyQuery` for querying versioned state stores (KIP-960, KIP-968). Adds `reverseRange` and `reverseAll` queries for key-value stores in IQv2 (KIP-985).

1/10/2024

This post details benchmarking results comparing Confluent Cloud's Kora engine performance against Apache Kafka. It highlights significant improvements in p99 end-to-end latency, with Confluent Cloud outperforming Apache Kafka by up to 16x under various throughput and partition scaling scenarios. The post also discusses the methodology for measuring latency, including steady-state and real-world conditions, and the benchmark setup, emphasizing the use of CKUs for consistent resource allocation in Confluent Cloud and equivalent hardware configurations for Apache Kafka. It also touches upon Kora's ability to maintain low latency even during cloud provider disruptions.

1/2/2024

This post details BT's implementation of Confluent Platform and Apache Kafka for network automation in their TV signal distribution infrastructure. It describes the Vena orchestration system, which leverages Kafka for reliable message distribution and event-driven architectures. The post outlines two key use cases: event-driven network onboarding using LLDP events and syslog connectors, and real-time network monitoring and fault detection. It highlights the technical reasons for choosing Kafka (de facto standard, ecosystem, stream processing) and Confluent (resilience, automated deployment, replication, monitoring, security, support). The post also discusses the challenges of a legacy network and the need for scalability and high availability in handling network events and potential alarm storms.

12/6/2023

This post provides guidance on managing enterprise Apache Kafka clusters, advocating for a platform approach to reduce complexity and cost. It details strategies for consolidating siloed Kafka deployments into a more centralized model, discussing the benefits of standardization, improved service quality, and reduced governance risks. The post also outlines the trade-offs of full centralization, such as meeting diverse requirements and organizational transformation challenges. It references data mesh, data products, and domain-driven design as conceptual frameworks for building effective data streaming platforms.

10/23/2023

This post details NASA's adoption and expansion of Apache Kafka, particularly leveraging Confluent Cloud for its GCN project and future initiatives like the Data and Reasoning Fabric. It highlights the use of data streaming for real-time astronomical alerts and the potential for drone delivery systems, emphasizing the need for immediacy and scalability. NASA's journey involved moving from self-managed Kafka to Amazon MSK and finally to Confluent Cloud, driven by the desire for reduced operational overhead and enhanced capabilities. The post also touches upon NASA's distributed IT structure and the challenges of lowering the barrier to cloud adoption for its mission teams.

9/18/2023

This post details Apache Kafka's message compression capabilities. It explains how compression works in Kafka producers by batching data to the same partition, the impact of compression on throughput, network traffic, and disk utilization. It lists available compression algorithms (none, gzip, snappy, lz4, zstd) and provides recommendations (lz4 for performance, zstd for gzip-like ratio with less overhead). The post elaborates on configuring `compression.type` at both the topic and producer levels, detailing precedence and common/uncommon combinations. It also covers broker-side decompression scenarios and how consumers handle compressed messages. Finally, it offers advice on avoiding compression with encrypted data and inspecting compression codecs on brokers.

8/31/2023

This post details the Kora Engine, Confluent's cloud-native event streaming platform, which powers Confluent Cloud. It highlights Kora's architecture designed for reliability, elasticity, and cost-efficiency, including its cellular architecture for multi-tenancy, serverless abstraction, self-balancing clusters, automatic failure mitigation for high availability, and seamless replication/backup using object storage. The post also mentions the award received at VLDB 2023 for the Kora paper.

8/31/2023

This post provides detailed guidance on tuning Kafka Connect source connectors to increase throughput. It covers what aspects of a connector can be tuned (connector configurations, producer configurations) and what cannot (converter, transformations). Specific producer configurations like batch.size, linger.ms, buffer.memory, and compression.type are explained with examples. It also lists essential JMX metrics for monitoring connector, broker, and producer performance, and provides a walk-through example using a JDBC Source connector to demonstrate the tuning process.

8/8/2023

This post provides a foundational explanation of Apache Kafka clusters, detailing the roles of brokers, partitions, and replication in achieving speed, durability, and scalability. It introduces KRaft as the successor to ZooKeeper for metadata management, explaining its function in leader election and replica failure recovery. The post also briefly discusses on-premise vs. cloud deployment options for Kafka.

7/20/2023

This post addresses the decision-making process for creating new Kafka clusters, detailing physical limitations (e.g., 200,000 partitions per cluster, 4,000 partitions per broker) and operational concerns like data privacy, noisy neighbors, and chargeback. It argues for the benefits of a single cluster for real-time data correlation and introduces Confluent Cluster Linking and Schema Linking as mechanisms for inter-cluster data exchange. It also provides a federated governance model for operating large Kafka deployments, distinguishing responsibilities between shared services and line-of-business teams, with a focus on security, capacity assurance, and self-service operations, and suggests Confluent Cloud as a SaaS solution for simplified management.

5/26/2023

This post details BigCommerce's migration from self-managed Kafka and a Hadoop MapReduce-based system to Confluent Cloud. It highlights the technical challenges of managing Kafka clusters, including provisioning, scaling for seasonal traffic, software upgrades, and the operational burden on their engineering team. The migration strategy involved a dual-write approach to ensure zero downtime and no data loss. The post emphasizes the benefits of Confluent Cloud for real-time data streaming, reduced operational overhead, and automatic scaling capabilities, enabling BigCommerce to provide real-time analytics to merchants.

5/16/2023

This post introduces Kora, Confluent's cloud-native engine for Apache Kafka, detailing its architecture and benefits. Kora is designed from the ground up for cloud constraints, focusing on multi-tenancy, disaggregation of compute, network, metadata, and storage layers, intelligent data locality management (memory, SSD, object storage), performance optimization for cloud environments, real-time usage capture for operations, and cost optimization. It enables elastic scaling (30x faster), higher availability (99.99% SLA), and lower latency compared to self-managed Kafka. The post also touches on continuous improvements through multi-phase fleet updates and advanced data balancing algorithms.

5/16/2023

Introduces Kora, a cloud-native Kafka engine for Confluent Cloud, offering enhanced elasticity, reliability, and performance. Enhances Stream Governance with Data Quality Rules (Domain Validation, Event-Condition-Action, Complex Schema Migration) for data integrity and contract enforcement. Introduces Custom Connectors for bringing user-defined Kafka Connect plugins to Confluent Cloud without infrastructure management. Launches Stream Sharing for secure, easy real-time data exchange across organizations. Mentions early access to Apache Flink.

5/12/2023

Loggi details their adoption of an event-driven architecture using Apache Kafka and Confluent Cloud to scale their logistics platform. They migrated from a monolithic system, focusing on improving developer productivity and system resilience. Key technical implementations include the transactional Outbox Pattern for producing events with strong transactional guarantees (ensuring database writes and Kafka event production are atomic) and an Event Broker microservice that consumes events from Kafka and delivers them to consumer services via gRPC. This design abstracts Kafka complexity for engineers, allowing them to focus on business logic. They also discuss lessons learned regarding potential bottlenecks in the Event Broker microservice, organizational trade-offs, and the importance of observability and performance tuning, including the use of dedicated Kafka consumers for high-volume events.

4/27/2023

This post analyzes the engineering costs of running Kafka, breaking them down into infrastructure, development, and operations. It quantifies these costs by estimating the number of engineers required for different scales of operation and their associated salaries and fully loaded employment costs. It also discusses capacity planning challenges, the need for platform development beyond core Kafka (e.g., connectors, security, monitoring), and the complexities of cluster scaling and ongoing maintenance. The post highlights the opportunity cost of dedicating engineering talent to infrastructure management versus strategic business initiatives and contrasts these self-managed costs with the benefits of managed services like Confluent Cloud.

4/25/2023

This post explains the concepts of at-most-once, at-least-once, and exactly-once (effectively once) delivery guarantees in messaging systems. It uses the Two Generals Problem as an analogy to illustrate the challenges of achieving reliable communication over unreliable networks. The post discusses the implications of each delivery guarantee, drawing parallels to science fiction teleportation scenarios. It concludes by relating these concepts to modern software development, especially in distributed systems and financial transactions, and mentions Apache Kafka's transactional guarantees as a solution.

4/20/2023

This post details the infrastructure costs of running Kafka, highlighting networking as the biggest cost driver (over 50% of infrastructure costs). It breaks down costs for compute, storage, and networking, providing example calculations for different ingress rates. It emphasizes the importance of cross-AZ traffic for producers, consumers, and partition replication as the primary driver of networking costs. The post also touches upon the impact of utilization and introduces a four-part series on understanding and optimizing Kafka costs, with subsequent posts focusing on operations, general cost reduction, and a savings challenge.

2/14/2023

This post details Michelin's successful migration from self-managed on-prem Kafka to Confluent Cloud, achieving a 35% cost reduction. It elaborates on the operational burden of self-managing Kafka, the scaling challenges encountered, and the benefits of a managed service for reliability and faster time-to-market. The migration allowed Michelin to reallocate dedicated Kafka teams to innovation and higher-value activities, improving inventory management and customer experience.

2/8/2023

Introduces and explains the four types of windowing in Kafka Streams and ksqlDB: Hopping, Tumbling, Session, and Sliding. Details configuration parameters for each (window size, advance size, inactivity gap, time difference) and provides examples. Discusses the handling of late events with grace periods and notes differences in grace period defaults between Kafka Streams and ksqlDB.

12/6/2022

This post elaborates on the fundamental concept of Kafka consumer group IDs, detailing their role in detecting new data via offsets, enabling work sharing among consumers for parallel processing, and ensuring fault tolerance through rebalancing mechanisms. It introduces the concept of static membership (using `group.instance.id`) as an optimization for applications with large local state to reduce rebalance-related data transfer and improve rolling upgrades.

12/1/2022

This post focuses on diagnosing and debugging increased connection issues in Apache Kafka. It details the types of connections across the cluster (producer, consumer, broker), explains relevant metrics like connection-count, connection-creation-rate, and AcceptorBlockedPercent, and explores common root causes such as undersized workloads, incorrect consumer-per-thread patterns, and multiple KafkaProducer instances within a single process.

11/15/2022

This post focuses on diagnosing and debugging increased consumer rebalance time in Apache Kafka. It details the mechanics of Stop-the-World and Incremental Cooperative rebalancing, introduces Static Group Assignment as a method to reduce rebalance impact, and highlights key consumer metrics (join-rate, records-lag-max) for monitoring rebalance health. It also identifies common root causes for increased rebalance time, including one-consumer-per-thread patterns, undersized KafkaConsumer instances, and overloaded group coordinators.

11/10/2022

This post details the client request lifecycle in Kafka (producer.send() and consumer.poll()) and identifies key JMX metrics for monitoring broker health, including RequestMetrics, SocketServer, KafkaRequestHandlerPool, and RequestChannel metrics. It also provides diagnostic questions to help identify root causes of increased request rates, response times, and broker load, such as inefficient batching, client workload sizing, and excessive metadata requests.

11/3/2022

This post provides a detailed guide to diagnosing reduced message throughput in Apache Kafka. It focuses on producer-side metrics such as batch-size-avg, records-per-request-avg, record-size-avg, record-queue-time-avg, request-latency-avg, and requests-in-flight. It also explains how to tune producer configurations like batch.size, linger.ms, and buffer.memory to optimize batching. Additionally, it suggests investigating topic partitions, request rates, and the number of KafkaProducer instances as potential root causes for throughput issues.

9/6/2022

Introduces seven new Health+ dashboard categories (Kafka Health, System, Producers, Consumers, Throughput, Thread Utilization, Topics, ksqlDB, Connect) to provide deeper insights into Kafka and Confluent Platform health. Explains the telemetry data collection and transmission process to Confluent Cloud, emphasizing that message content is not collected. Discusses security and privacy aspects of Health+ and provides guidance on getting started.

5/3/2022

Introduces the ability to safely remove multiple Kafka brokers with a single command, addressing previous limitations of removing only one broker at a time and the risk of under-replicated partitions. Details the use of Confluent's Self-Balancing Clusters for intelligent partition reassignment and bin-packing, operation persistence using a state machine and a Kafka topic for resilience, preliminary validations to prevent capacity issues, and a new Broker Replica Exclusion API to handle race conditions with new topic creations.

3/9/2022

This post focuses on optimizing Kafka client performance on Confluent Cloud for low latency and high throughput. It highlights five key resources that delve into cloud-native Kafka design principles, client configuration parameters (producer and consumer), benchmarking methodologies, and latency-influencing factors within Kafka. It also touches upon the 'time to market' aspect by emphasizing Confluent Cloud's managed nature reducing operational burden.

1/11/2022

This post details common pitfalls when using Apache Kafka, covering client-side issues like setting request.timeout.ms too low and misunderstanding producer retries and retriable exceptions. It also addresses broker-side issues such as leaving key broker metrics unmonitored, going overboard with partitions, and setting segment.ms too low. Specific configuration parameters and their implications are discussed, along with recommendations for optimal settings.

9/22/2021

This post details how Confluent enables modernization of legacy messaging architectures by integrating with existing middleware (RabbitMQ, ActiveMQ, IBM MQ, TIBCO EMS) using Kafka Connectors. It highlights the use of Schema Registry for metadata integration and governance, and ksqlDB for stream processing and analysis of data in motion. The post emphasizes the benefits of a fully managed cloud service for these capabilities, reducing complexity and cost compared to traditional setups. An example walkthrough demonstrates the end-to-end integration process.

6/18/2021

This post elaborates on consistency and completeness guarantees in stream processing, specifically within Apache Kafka and Kafka Streams, detailing how persistent logging, idempotent and transactional writes, and speculative processing with revision enable exactly-once semantics and ordered results even with out-of-order data. It highlights the decoupling of latency, throughput, and correctness trade-offs through Kafka's log-based approach.

5/21/2021

Introduces four error handling patterns for Kafka applications: stop-on-error, dead-letter queue, retry topic with potential out-of-order processing, and a more complex retry topic pattern that maintains original event order using in-memory stores and redirect topics for tombstone events. Discusses recovery mechanisms for the in-memory store.

3/10/2021

This post details performance tuning for RocksDB state stores within Kafka Streams. It explains how Kafka Streams uses RocksDB for stateful operations, introduces RocksDB's internal workings including compaction styles (level and universal), and provides guidance on configuring RocksDB via RocksDBConfigSetter. It also highlights the importance of RocksDB-specific metrics (introduced by KIP-471 and KIP-607) for diagnosing and resolving operational issues with state stores.

2/22/2021

This post details Pinterest's operational experience running Kafka at scale, focusing on challenges and solutions related to performance and cost control. Key technical aspects include the transition from magnetic disks to SSDs for improved IOPS and reduced latency during broker recovery, the adoption of a static assignment model ('brokerset's) instead of dynamic rebalancing, and efforts to manage message format versions to reduce CPU load. Cost control measures include rack-aware data transfer strategies, enforcing compression on the producer side, and optimizing topic retention and replication factors. The post also touches upon the use of Singer for log ingestion and Maxwell for database changelogs.

2/17/2021

ksqlDB 0.15 introduces expanded support for message keys, allowing structured data types (ARRAY, STRUCT) and various serialization formats (JSON, AVRO, PROTOBUF, DELIMITED, KAFKA, JSON_SR) in keys. It also enables PARTITION BY and GROUP BY clauses with multiple expressions for tables and streams with multiple key or primary key columns. Additionally, the DROP command now automatically terminates persistent queries associated with a stream or table, simplifying the management of ksqlDB resources.

1/19/2021

This post presents predictions for Apache Kafka and software architecture in 2021, focusing on technical advancements. It highlights the potential for Kafka clusters to support 10 million partitions through Tiered Storage (KIP-405) and ZooKeeper removal (KIP-500). It also predicts rapid Kafka cluster scaling (doubling size in seconds) enabled by Tiered Storage, with architectural benefits including reduced data migration and potential performance gains. The post foresees the proliferation of streaming features in various data products and the rise of event streaming as a dominant architectural paradigm. It also discusses the development of multi-paradigm products and the integration of database concepts into event-driven systems via ksqlDB.

12/17/2020

This post introduces the use of Spring Boot for developing Kafka Streams microservices, detailing its benefits for Kafka integration, configuration, dependency management, and testing. It also describes the deployment of these microservices to Kubernetes using a GitOps workflow, including the use of Kustomize for managing deployments and the integration with the existing streaming-ops project.

12/15/2020

Introduces the Confluent Parallel Consumer, a new client library that provides key-level and unordered parallelism for Kafka message consumption, surpassing the limitations of partition-level parallelism. It details how this library enables higher throughput by allowing more concurrent processing than available partitions and by efficiently handling external service calls (e.g., HTTP requests to legacy services) through non-blocking I/O and optional Vert.x integration. The post also highlights features for client-side work queues and message-level acknowledgment, improving performance for task queue use cases.

12/8/2020

This post provides a beginner's tutorial for Scala developers on producing and consuming records in Apache Kafka. It covers setting up a Scala project with Kafka clients and Confluent serializers, configuring producers with Avro serialization and Schema Registry integration, and demonstrates three production scenarios: batching records with linger.ms, sending single events with event time and headers, and performing transactional writes. It also highlights the use of Avro4s for serialization and pureconfig for configuration.

10/6/2020

Introduces Confluent Platform's Self-Balancing Clusters (SBC) feature, which automates Kafka cluster scaling and load balancing. SBC monitors partition layout and broker load, automatically reassigning partitions to balance replica counts, leader counts, disk usage, and network usage. It handles broker additions, removals, and failures, respecting rack awareness and capacity limits. Rebalancing is throttled to minimize impact on ongoing traffic.

8/21/2020

This post provides a detailed performance comparison of Apache Kafka, RabbitMQ, and Apache Pulsar, focusing on system throughput and end-to-end latency. It details the benchmarking framework used, including fixes made to the OpenMessaging Benchmark Framework (OMB) and its drivers for Kafka, RabbitMQ, and Pulsar. The post presents quantitative results for peak throughput and p99 latency under various load conditions, concluding that Kafka offers the best throughput and lowest latencies. It also discusses durability guarantees in distributed systems and the role of fsync.

6/30/2020

This post introduces strategies for handling deserialization failures (poison pills) in Spring Kafka consumers. It explains the causes of poison pills, including incompatible serializers and deserializers, and demonstrates how to configure Spring Kafka to gracefully manage these errors. The post highlights the importance of robust error handling for production deployments and provides an example project on GitHub for practical implementation.

5/15/2020

This post details the technical plan and roadmap for removing ZooKeeper dependency from Apache Kafka through KIP-500. It explains the problems with external metadata management (duplication, inefficiency, scalability limits, de-synchronization) and proposes storing metadata in a Kafka partition managed by a Raft quorum. It outlines the controller architecture changes, including standby controllers and O(1) topic operations. The roadmap includes removing ZooKeeper from administrative tools, implementing self-managed metadata quorums with a pull-based Raft protocol (KIP-595), introducing KIP-500 mode (initially experimental), and a bridge release strategy for zero-downtime upgrades using new controller APIs like AlterIsr.

5/11/2020

Introduces and explains the incremental cooperative rebalancing protocol for Kafka consumers, contrasting it with the eager rebalancing protocol. Details the mechanics of the JoinGroup and SyncGroup requests, the roles of group coordinator and leader, and the partition assignor's function in the new protocol. Explains how this protocol minimizes downtime by allowing consumers to retain ownership of unassigned partitions, benefiting Kafka Streams and ksqlDB.

5/4/2020

This post details Walmart's implementation of a real-time inventory management system using Apache Kafka. It covers the engineering challenges and solutions for managing diverse data sources through a canonical approach and transformation engine. The post provides in-depth technical details on scaling Kafka by optimizing partitions, producer configurations (linger.ms, batch.size, acks), and consumer scaling strategies. It also addresses database integration challenges, specifically with Cassandra, by detailing partitioning strategies to ensure data consistency and minimize deadlocks. The post includes diagrams illustrating data flow and scaling concepts.

2/25/2020

This post provides a deep dive into understanding and optimizing end-to-end latency in Apache Kafka. It breaks down latency into five components: produce time, publish time, commit time, catch-up time, and fetch time, explaining the factors influencing each. It also clarifies the relationship between end-to-end latency, producer latency, and consumer latency, and discusses strategies for bounding tail latencies by addressing head-of-line blocking and micro-bursting. The post emphasizes tuning client and broker configurations, and scaling Kafka clients for throughput while minimizing queueing delays and congestion.

1/21/2020

This post details Netflix's use of Apache Kafka for distributed streaming within their Content Finance Engineering team. It explains their adoption of an event-driven architecture for microservices, detailing the roles of producers (including CDC and normalized event formats), enrichers (using stream processing like Flink and ksqlDB for data enrichment and materialized views), and consumers (using Spring Boot and Schema Registry/Avro for schematized streams). It also covers strategies for ensuring event ordering (keyed messages, delayed materialization) and message delivery guarantees (idempotency with EVCache).

1/14/2020

This post delves into Kafka's storage fundamentals, focusing on topics and partitions. It explains how topics are unbounded sequences of events, configurable with retention policies, and how brokers are serialization-agnostic. The core concept of partitions is highlighted as fundamental for Kafka's scalability, elasticity, and fault tolerance, enabling parallel reads and fault tolerance through replication. The post details how producers determine event partitioning using a partitioning function, emphasizing the importance of sending related events to the same partition for ordering guarantees within consumer groups. It also provides guidance on choosing the number of partitions and discusses common reasons for events with the same key ending up in different partitions.

12/17/2019

Introduces and demonstrates the usage of TestInputTopic and TestOutputTopic classes from KIP-470 for simplifying Kafka Streams testing with TopologyTestDriver. Provides examples using AssertJ for testing single records (value, key-value, record with timestamp, record with headers) and collections of records (value lists, key-value lists, record lists, key-value maps). Includes guidance on migrating to the new interface and references external testing resources.

10/3/2019

Scrapinghub's AutoExtract service uses Confluent Cloud for its Kafka needs to scale its AI data extraction API. They partition URLs for fetching, rendering, and screenshotting using Kafka, and distribute resource-intensive AI data extraction tasks across multiple instances. The move to Confluent Cloud allowed them to offload Kafka infrastructure management, providing scalability and cost benefits. They detail their evaluation of alternatives like self-hosting Kafka on Kubernetes and Amazon MSK, ultimately choosing Confluent Cloud for its managed nature, consumption-based pricing, and vendor independence. They also describe initial setup, load testing, and workarounds for missing tools like Burrow by using a consumer metrics exporter.

8/29/2019

This post introduces the integration of Neo4j with Apache Kafka for graph processing and stream analytics. It details the use of the Neo4j-Streams plugin to ingest Kafka data into Neo4j, creating nodes and relationships. It also demonstrates how to apply graph algorithms like Common Neighbors to identify potential connections and output these results back to Kafka. The post provides configuration details for connecting Neo4j to Kafka using SASL_SSL and API keys, and includes a GitHub repository with code and setup instructions.

6/26/2019

This post discusses the symbiotic relationship between microservices, domain-driven design (DDD), and Apache Kafka. It explains how Kafka serves as an event streaming platform backbone for microservices, enabling decoupled communication through publish-subscribe, fault-tolerant storage, and real-time processing. The post elaborates on DDD principles like bounded contexts and ubiquitous language, and how Kafka facilitates their implementation by decoupling microservices and allowing teams to own their domains. It highlights Kafka's advantages over traditional middleware and discusses the use of Kafka Streams, ksqlDB, and Kafka Connect for building event-driven applications within DDD contexts. The concept of an anti-corruption layer is explained in the context of mapping data between bounded contexts using event schemas and the Schema Registry. The post also touches upon the operational aspects of managing Kafka clusters, including self-service offerings on PaaS infrastructure like Kubernetes and managed services like Confluent Cloud.

4/24/2019

Introduces the 'Test Machine' library for end-to-end testing of event-driven systems, addressing the limitations of unit tests and existing Kafka Streams testing tools like TopologyTestDriver and Fluent Kafka Streams Tests. Details a testing pattern involving writing input events, waiting for processing, and asserting output. Explains the 'Test Machine' architecture with 'write' and 'watch' commands, portable test execution against various targets (local Kafka, REST Proxy, mock topology), and performance considerations. Provides a Clojure code example using the Test Machine for a word count demo.

4/4/2019

Introduces KStream#branch() for static stream splitting based on predicates, creating new Kafka topics for different event subsets. Details dynamic routing using TopicNameExtractor (KIP-303) for runtime determination of output topic names, enabling routing based on event fields or enriched data from external sources like databases via JDBC Source Connector and GlobalKTable joins.

1/15/2019

Introduces unit testing for Kafka Streams applications using TopologyTestDriver and MockProcessorContext, integration testing with EmbeddedKafkaCluster, and schema compatibility testing with MockSchemaRegistryClient and the Schema Registry Maven plugin. It also mentions testing with Confluent Cloud tools and multi-datacenter scenarios.

12/3/2018

Introduces the concept of prefixed ACLs in Kafka (KIP-290) enabled by Confluent Platform 5.0+ and Apache Kafka 2.0+. Details two patterns for configuring these ACLs: 1) Team-level prefixes (e.g., 'team.fraud.') allowing read/write/create on all topics and groups starting with that prefix, and 2) Application-level prefixes (e.g., 'mobile.payments.fraud') for more granular control. Explains how to apply these patterns to both Kafka Streams (using `application.id`) and KSQL (using `ksql.service.id`), including the specific prefixes for internal KSQL topics (e.g., '_confluent-ksql-team.fraud').

2/22/2018

This post introduces the security aspects of KSQL, detailing how to configure it with SSL for encryption and SASL for authentication when communicating with Kafka clusters. It also covers HTTPS for Schema Registry integration. The post emphasizes the comprehensive security design of the Confluent Platform, including its various components, and provides links to demos and tutorials for practical security implementation.

4/18/2017

This post details three common operational pitfalls encountered in production Apache Kafka deployments: under-replicated partitions, issues with automated liveness checks leading to cluster downtime, and performance degradation when adding new brokers. It provides high-level recommendations for each scenario, emphasizing root cause analysis for under-replicated partitions, the importance of human intervention for automated restarts in stateful systems, and understanding the impact of partition reassignments before execution. It also highlights the benefits of Confluent's auto data balancer for safer data movement.

3/28/2017

This post details the configuration steps for securing the Confluent Schema Registry to work with a secured Kafka cluster. It covers configuring SASL for authentication with Kafka and ZooKeeper, setting up Kafka ACLs for the Schema Registry to access the `_schemas` topic, and enabling HTTPS for the Schema Registry's REST API. It provides specific configuration examples for JAAS files, Kafka properties, and ACL commands, along with an analogy to explain the security concepts.

1/25/2017

This post announces Confluent 3.1.2, a patch release that includes Apache Kafka 0.10.1.1. It details specific bug fixes and general improvements across Kafka brokers, the Streams API, and clients. For brokers, fixes address log cleaner issues, ISR thrashing, tombstone removal, offset retention calculation, replica fetcher thread errors, and NullPointerExceptions. For Kafka Streams, improvements cover user configuration handling, topic not found errors, resetter performance, cache eviction, and KTableSource simplification. Client fixes include consumer deadlocks, throughput regressions, and failures after partition reassignment. A general improvement for Scala 2.12 support is also noted.

12/6/2016

This post details several KIPs impacting Kafka semantics and operability: KIP-95 for incremental batch processing in Kafka Streams, KIP-97 for improved Kafka client RPC compatibility allowing newer clients to connect to older brokers, and the proposed KIP-98 for exactly once delivery and transactional messaging. It also highlights community events and other streaming system developments.

11/15/2016

This post announces Confluent Platform 3.1, which includes Apache Kafka 0.10.1.0. Key new features in Confluent Platform are Automatic Data Balancing to address unbalanced workloads across brokers due to varying topic/partition growth and broker changes, and Multi-datacenter Replication for scenarios like active-active geo-localized deployments, centralized analytics, and cloud migration. Confluent Control Center is enhanced with stream latency and completeness alerting, enterprise authentication integration, and multi-cluster topic monitoring. New open-source components include an Elasticsearch Sink connector with automatic mapping generation and exactly-once delivery, a JDBC Sink connector for exporting data to relational databases, and a new Go Client.

7/30/2016

This post details progress on KIP-4 for client-managed topics (API to create and delete topics voted in, ACL management under discussion), KIP-67 for queryable state in Kafka Streams, and best practices for running Kafka in AWS. It also mentions bugfixes for Kafka 0.10.0.1 and improvements to Spark Streaming.

3/1/2016

This post details several key developments in the Apache Kafka community and Confluent Platform. It announces the release of Apache Kafka 0.9.0.1 and Confluent Platform 2.0.1, highlighting bug fixes and improvements. The next release is designated as 0.10.0, incorporating a protocol update, new file format, and Kafka Streams. Several KIPs were merged: KIP-41 (KafkaConsumer Max Records), KIP-32 (Add timestamps to Kafka message), and KIP-42 (Add Producer and Consumer Interceptors). New KIPs under discussion include KIP-47 (timestamp-based log deletion) and KIP-48 (delegation token support). The Kafka Connector Hub is launched, and Microsoft open-sourced their .NET Kafka client. Community meetups and discussions on stream processing futures are mentioned, along with the early release of "Kafka: The Definitive Guide" and the upcoming Kafka Summit.

2/19/2016

This post announces Apache Kafka 0.9.0.1 and Confluent Platform 2.0.1, which includes 68 issue fixes. Notable bug fixes are detailed for the new Java consumer (rebalancing, offset issues), compatibility, security (ticket renewal, ACLs, SASL packet handling), performance (fetch wait, CPU usage, log cleaner, sensor leaks), and topic deletion. It recommends upgrading to this patch release for improved stability and correctness.

1/13/2016

This post details numerous bug fixes and improvements in Apache Kafka 0.9.0.0, with a strong emphasis on correctness, reliability, and operability over new features. Specific technical details are provided for fixes related to replication (race conditions, min.insync.replica, replication lag), MirrorMaker message loss prevention, log cleanup, log compaction (especially for compressed messages), connection leak detection, broker metadata cache synchronization, request purgatory rewrite, and producer timeout enforcement. It also highlights the addition of a new distributed testing framework with over 100 new test scenarios.

7/1/2015

Introduces a change to Kafka's replica lag detection mechanism, replacing `replica.lag.max.messages` with a time-based approach (`replica.lag.time.max.ms`). This aims to simplify configuration, reduce operational overhead, and prevent false alerts caused by temporary spikes in producer traffic or GC pauses on follower replicas. The new model ensures that replicas are only removed from the ISR if they consistently lag behind the leader for a defined period, improving the stability of replication.