TechPrufer
Get started
Confluent
Confluent logoConfluent · Feature trail

Kafka Broker and Streams Enhancements

126 posts · Jun 2015 – May 2026

What this trail covers

This feature thread tracks the evolution of Kafka Streams and ksqlDB, focusing on enhancing availability, performance, and query capabilities. It covers improvements in serving data from stale stores during rebalances (KIP-535), fetching data from specific partitions (KIP-562), and fixing unnecessary standby restoration bugs (KAFKA-9169). The goal is to enable Kafka Streams to challenge traditional databases by providing high availability and low-latency serving, reducing developer overhead and enabling effective scaling for high-volume data processing through strategies like partitioning, scaling out/up, and fine-tuning configuration parameters.

All Confluent trails
  1. Apache Kafka 4.3 released: 25 KIPs, Kafka Streams, Share Group Controls, Broker Isolation, and moreThis post details the release of Apache Kafka 4.3, highlighting 25 KIPs. Key broker enhancements include follower fetch from tiered offset (KIP-1023), mechanism to cordon brokers and log directories (KIP-1066), group coordinator buffer size configurations (KIP-1196), admin configs for tiered storage (KIP-1208), alignment of num.partitions and default.replication.factor (KIP-1211), KRaft fetch and snapshot byte size configurations (KIP-1219), default min.insync.replicas for \_\_remote\_log\_metadata topic (KIP-1235), additional share group configurations (KIP-1240), assignment epochs for consumer groups (KIP-1251), partition size percentage metrics (KIP-1257), OAuth client assertion support (KIP-1258), and group coordinator assignment batching (KIP-1263). Kafka Streams updates include state store managed changelog offsets (KIP-1035), public API for Bytes utils (KIP-1247), in-memory state store size metrics (KIP-1250), state cleanup on startup (KIP-1259), extended ProcessingExceptionHandler for GlobalThread (KIP-1270), storing headers in state stores (KIP-1271), and DSL opt-in for headers-aware state stores (KIP-1285). Kafka Connect improvements include batch offset translation (KIP-1239) and enhanced configurable component discoverability (KIP-1273). MirrorMaker metrics are updated to use KIP-877 (KIP-1280). Deprecation notices are given for streams-scala, group.coordinator.rebalance.protocols, and existing MirrorMaker metrics.
  2. Add Your First Flink Streaming TransformationThis post introduces the concept and practical implementation of adding Flink streaming transformations to Kafka for real-time data processing. It details the shift from passive to active data streams, contrasting batch processing with continuous streaming. The post outlines the benefits of Flink for real-time data pipelines, including reduced latency, simplified architectures, and centralized transformation logic. It provides a step-by-step guide for implementing a first Flink transformation, focusing on stateless operations like filtering and field projection, and introduces common first transformations such as event filtering, field projection/reshaping, real-time enrichment, and introductory event aggregation. The distinction between stateless and stateful transformations is also discussed.
  3. Agentic Event-Driven Systems ArchitectureIntroduces the concept and architecture of agentic event-driven systems, which are AI-native architectures where software agents continuously sense events, reason over shared state, take actions, and learn from outcomes in real time. This architecture extends traditional event-driven systems by embedding decision intelligence and control loops directly into the event flow, enabling autonomous operation. Key components include an event-driven backbone (e.g., Apache Kafka), stateful stream processing (e.g., Apache Flink), an agent execution layer for reasoning and decision-making, a shared state and context layer, and an orchestration and policy engine. The architecture emphasizes closed-loop control patterns where actions generate outcome events that feed back into the decision process for continuous adaptation and optimization.
  4. RAG and GenAI for Regulated SectorsThis post details the application of event streaming (Kafka) to build a secure, compliant Retrieval-Augmented Generation (RAG) system for regulated sectors. It outlines a streaming RAG ML pipeline that uses Change Data Capture (CDC) for real-time data ingestion from databases, a governance layer for PII masking, schema enforcement via a schema registry, and tokenization. It also describes secure vector index retrieval with Context Scoping, LLM integration with safety guardrails, and an observability layer for response provenance. The post contrasts batch RAG with real-time streaming RAG, highlighting the latter's advantages in data freshness, compliance timing, audit complexity, and policy enforcement for regulated environments.
  5. Feed your data lake with AI-ready tables for 30-50% lower costIntroduces Tableflow, a new capability that directly converts Apache Kafka topics into analytics-ready tables in formats like Apache Iceberg and Delta Lake. This feature aims to reduce data ingestion costs by 30-50% compared to traditional ETL pipelines. It achieves this by reusing Kafka segments stored in Confluent Cloud's Kora engine, converting them to Parquet, automatically handling schema evolution and CDC, compacting small files, and publishing tables to external catalogs. The post details the cost drivers of traditional ingestion stacks and highlights how Tableflow addresses them, including Kafka services and storage, connect infrastructure, lakehouse compute, and engineering overhead. It also provides benchmark data comparing Tableflow to managed lakehouse solutions and discusses architectural benefits like openness, operational simplicity, near-real-time data, and shift-left processing.
  7. New Observability Updates in Confluent Cloud KafkaIntroduces new observability metrics for Confluent Cloud Kafka: `io.confluent.kafka.server/client_limit_milliseconds` to identify users breaching limits and reasons, `io.confluent.kafka.server/max_pending_rebalance_time_milliseconds` to monitor consumer group rebalance duration and frequency, `io.confluent.kafka.server/connection_accept_count` to track connection attempts for elastic cluster scaling, and a new `cleanup.policy` dimension for `io.confluent.kafka.server/partition_count` to differentiate compacted partitions. Also details a refreshed cluster monitoring experience in Confluent Cloud Console.
  8. Kafka Client Migrations With KCP and Confluent Cloud GatewayThis post details the integration of Confluent Cloud Gateway with KCP to automate Kafka client migrations. It explains the challenges of client migration (coordination, authentication, offset management, rollback) and how Gateway addresses them by acting as a proxy and performing authentication translation. The post outlines a four-stage client migration strategy (Deploy, Plan, Onboard, Migrate) and provides the specific KCP commands for initializing the migration, including parameters for source/destination cluster details, Gateway CR YAMLs, and authentication methods. It highlights the benefits of Gateway's authentication swapping feature and the seamless cutover process with minimal downtime.
  9. Introducing Confluent Platform 8.2: Queues for Kafka & Flink SQLThis post introduces Queues for Kafka, a new feature in Confluent Platform 8.2, providing native queue semantics on Apache Kafka with elastic consumer scaling and per-message acknowledgment. It also announces the General Availability (GA) of Flink SQL in Confluent Platform for Apache Flink®, enabling declarative stream management with DDLs, changelogs, and shared compute pools. Ease-of-use enhancements for Flink include multi-Kubernetes cluster support, a new savepoint management UI, and native support for RHEL 10 and OpenShift. Confluent Private Cloud Gateway 1.2 adds intelligent fencing and unfencing for migration and failover, SCRAM auth swapping, and expanded non-Java client support.
  10. Why CISOs Are Building Open Security Lakes with IcebergThis post details how Confluent's data streaming platform, including Freight clusters, WarpStream, and Tableflow, enables the creation of 'Open Security Lakes' using Apache Iceberg. It explains the architectural shift from traditional SIEMs to a decoupled model where data streaming handles ingestion and routing, and Iceberg manages storage for forensic data. The post highlights the cost-effectiveness of Freight clusters for high-throughput logging and WarpStream's BYOC model. It also discusses the integration with Apache Flink for real-time threat detection and the use of Tableflow to convert Kafka topics into Iceberg tables with minimal friction.
  11. Why Batch ELT Breaks at Enterprise ScaleThis post details the limitations of traditional batch ELT pipelines at enterprise scale, highlighting issues like cascading failures, schema drift, resource inefficiency, and orchestration chaos. It advocates for a streaming-first data integration architecture using Kafka as a durable, scalable, and reusable data movement layer. The post explains how Kafka, combined with connectors and Tableflow, Schema Registry, and open table formats like Iceberg/Delta Lake, can unify operational and analytical data, enabling write-once-read-anywhere data ingestion and reducing the complexity and cost associated with traditional ELT.
  12. Introducing Schema IDs in Kafka Headers for Smarter Data GovernanceIntroduces the capability to store Schema Registry schema IDs in Kafka headers, enabling backward-compatible schema attachment to Kafka records. This feature allows for schematizing existing topics without breaking legacy consumers or changing payload formats, reducing data errors, and improving data usability for analytics and AI. It details producer and consumer behavior changes, the new 16-byte GUID format for schema IDs in headers, and broad ecosystem support. Migration paths for different starting points are also outlined.
  13. Post-Quantum Cryptography in Confluent CloudThis post announces the General Availability (GA) of 'Queues for Kafka' on Confluent Cloud and Confluent Platform, coinciding with the Apache Kafka 4.2 release. It introduces the share consumer API and share groups, allowing multiple consumers to cooperatively process messages from the same topic regardless of partition count. It also enables per-message acknowledgment, release, rejection, and renewal. The underlying mechanism is KIP-932, which uses time-limited locks on individual records managed by the broker. Code examples demonstrate both implicit and explicit acknowledgment modes. Confluent Cloud offers a fully managed experience with a dedicated UI and CLI, while Confluent Platform integrates with Control Center and Ansible/Kubernetes deployments. The post clarifies when to use the share consumer API (operational/application workloads) versus the traditional Consumer API (analytical/integration workloads).
  14. Automate Kafka Migration With Kafka Copy Paste (KCP)Introduces KCP, an open-source tool for automating Kafka migrations to Confluent Cloud. KCP orchestrates four key steps: discovery and planning (automating inventory of clusters, topics, ACLs, schemas, connectors, and clients), provisioning infrastructure (generating Terraform scripts for Confluent Cloud resources and networking like PrivateLink), data migration (using Cluster Linking for topic data, ACLs, schemas, and connectors), and client migration (planned for H1 2026). The post details KCP commands for discovery, reporting costs/metrics, and scanning various components. It also highlights the use of Cluster Linking over PrivateLink for direct private connections.
  15. Confluent Cloud Q1 2026: Queues for Kafka, KCP for migrationThis post announces the general availability of 'Queues for Kafka' on Confluent Cloud, introducing the 'share group' and 'share consumer' abstractions to enable elastic scaling beyond partition count for queue-like workloads. It also introduces KCP (Kafka Cloud Platform), an open-source CLI tool for automating migration from hosted Kafka to Confluent Cloud, covering discovery, provisioning, data migration, and client migration. New scalability features include maximum eCKU configuration for serverless clusters, increased eCKU scaling for Enterprise clusters (up to 32), client quotas for Enterprise and Freight clusters, and Fetch-from-Follower for Enterprise clusters. Confluent Intelligence is updated with multivariate anomaly detection for Flink.
  16. Apache Kafka 4.2.0 Released: Share Groups, Streams & MoreIntroduces server-side rebalance protocol to GA for Kafka Streams, adds dead letter queue support in exception handlers, and implements anchored wall-clock punctuation for deterministic scheduling. Also provides explicit control over sending leave group requests on closing.
  17. Building Kafka Client Failover: A 60-Second disaster recovery POC with Confluent Cloud GatewayThis post details a Proof of Concept (POC) for building Kafka client failover using Confluent Cloud Gateway to achieve disaster recovery within 60 seconds. It explores the challenges of RPO/RTO in Kafka outages, discusses server-side replication options like Cluster Linking, and highlights the complexities of client-side failover due to the static `bootstrap.servers` configuration. The POC demonstrates how Confluent Cloud Gateway can act as a traffic control layer to redirect client traffic to a passive cluster during an outage without client restarts. It also addresses the issue of read-only mirror topics and proposes a workflow involving connection dropping, reversing replication, and gateway switching. The architecture of the demo service is presented, emphasizing identity passthrough and OAuth2 support for failover.
  18. Streaming Data Integration with Apache Kafka® vs ETLThis post contrasts streaming data integration with ETL, arguing that Kafka's original use case of a centralized log for broadcasting and sharing data in real-time aligns with modern needs for interconnected data products. It details the limitations of ETL, such as data silos, lack of reusability, batch processing, and high latency, and highlights the advantages of streaming data integration with Kafka, including one-to-many and many-to-one source/destination support, reusable canonical streams, unification of real-time and historical data analysis, and the ability to build richly interconnected data products. It also suggests incorporating ETL best practices like intuitive workflows and reduced toil into streaming data integration.
  19. Migrate From Hosted Kafka to Confluent Cloud in the KRaft EraThis post details the technical aspects of migrating from hosted Kafka to Confluent Cloud, focusing on the KRaft era and the role of Kafka Copy Paste (KCP) and Cluster Linking. It explains how KCP simplifies migration planning and provisioning using Terraform, and how Cluster Linking enables byte-for-byte mirroring of topics, preserving partitioning, ordering, and offsets. The post also touches on the underlying mechanisms of Cluster Linking for continuous data propagation, error handling, and its application in global and hybrid cloud scenarios. It highlights cost savings through features like Kora, Private Network Interface (PNI), and Freight clusters, and emphasizes the reliability benefits of Confluent Cloud's 99.99% uptime SLA.
  20. Apache Kafka® Cluster Rebalancing Costs More Than You ThinkThis post details the hidden costs of manual Apache Kafka cluster rebalancing, including resource drain, operational overhead, downtime risk, and opportunity cost, particularly at scale. It highlights how Confluent's automated and self-balancing clusters eliminate these costs through non-disruptive partition redistribution, continuous monitoring, and predictive scaling, leading to reduced TCO and improved operational efficiency.
  21. How Superwall Scales with WarpStream and ClickHouse CloudThis post details Superwall's migration from a self-managed Kafka and ClickHouse OSS stack to WarpStream (a Kafka-compatible streaming platform) and ClickHouse Cloud. It highlights the use of ClickPipes for managed ingestion from WarpStream to ClickHouse Cloud, addressing challenges with disk space, cost, and operational overhead. The post emphasizes the benefits of a fully managed, elastic, and ops-light data foundation for their monetization platform, enabling them to handle high event volumes and large datasets with confidence.
  22. Cost of Apache Kafka® Migrations: Estimate & Reduce Streaming Platform CostsThis post details the cost drivers and mitigation strategies for Apache Kafka migrations, highlighting the benefits of Confluent Cloud and Cluster Linking for reducing engineering hours, downtime risk, and operational overhead. It emphasizes phased migrations, scope reduction, and parallel testing environments.
  23. The Real Cost of Data Streaming: TCO, ROI, and Hidden TradeoffsThis post delves into the Total Cost of Ownership (TCO) of data streaming, specifically contrasting self-managed Kafka with Confluent Cloud. It quantifies infrastructure, operations, engineering, governance, and opportunity costs, providing a detailed breakdown of monthly estimates for each scenario. Key Confluent Cloud features like eCKUs (Elastic Compute Units for Streaming) and elastic storage are highlighted as drivers of cost efficiency and scalability. The post also analyzes the latency-cost tradeoffs between batch processing and real-time streaming, emphasizing how streaming reduces critical business risks and operational costs.
  24. Secure Cross-Cloud Data Replication with ConfluentThis post introduces Confluent's new capability for secure cross-cloud data replication over private networks, powered by Cluster Linking. It details how this feature enables data movement between AWS, Azure, and Google Cloud without public internet exposure, addressing security and compliance concerns for multicloud strategies. The post contrasts this with the older Confluent Replicator method, highlighting the simplified, managed nature of Cluster Linking for private networking scenarios. It also provides a practical demonstration using Terraform and Confluent CLI for setting up and configuring private cross-cloud replication, focusing on use cases like data sharing and disaster recovery.
  25. Cross-Data-Center Apache Kafka® Replication: Decision Framework & Ops GuideThis post details cross-data-center replication strategies for Apache Kafka using Kafka MirrorMaker. It outlines decision frameworks for replication patterns (active-passive, active-active), scope (within-region vs. across-regions), and recovery goals (RTO/RPO). It provides a step-by-step operational guide for setting up and configuring MirrorMaker 2, including best practices for addressing common issues like configuration errors, replication lag, memory constraints, and broker connectivity failures. Key features of MirrorMaker 2 for resilient multi-cluster architectures are also highlighted.
  26. Scaling Kafka Streams for High-Volume Data ProcessingThis post details strategies for scaling Kafka Streams applications for high-volume data processing. It emphasizes the core principle of parallelism through partitioning in Kafka topics, explaining how partition counts dictate the number of tasks and thus maximum parallelism. It outlines two primary scaling strategies: scaling out (horizontal scaling) by running multiple instances with the same application.id, and scaling up (vertical scaling) by increasing machine resources and the `num.stream.threads` configuration. Key configuration parameters for performance tuning are discussed, including `rocksdb.config.setter`, `cache.max.bytes.buffering`, `commit.interval.ms`, and producer/consumer configs. Special considerations for stateful vs. stateless applications are highlighted, particularly the importance of co-partitioning for stateful operations like joins. Finally, it stresses the importance of monitoring key metrics like consumer lag and CPU utilization to identify bottlenecks.
  27. Ensure Data Quality With Real-Time Validation and MonitoringThis post details how to ensure data quality with real-time validation and monitoring using Apache Kafka and Confluent. It introduces a two-layer approach: validation (ensuring data conforms to structure and business rules in motion) and monitoring (tracking health metrics for trends and anomalies). Key technical components discussed are Schema Registry for schema enforcement at ingestion, and Apache Flink or ksqlDB for real-time business rule checks. It also covers integrating monitoring tools like Grafana or Datadog for visibility and actionable insights. The post emphasizes a 'shift left' strategy for data quality, moving validation closer to the source to prevent bad data from propagating.
  28. How to Build Real-Time Alerts to Stay Ahead of Critical EventsThis post details how to build real-time alerts using Apache Kafka, covering threshold-based, anomaly detection, composite event, and automated action alerts. It emphasizes designing trustworthy alerts by addressing alert fatigue through filtering and prioritization, and discusses the importance of timely and actionable notifications for proactive system monitoring and response.
  29. Kafka Connect in Production: Scaling & Security GuideThis post provides a comprehensive guide to deploying, scaling, and securing Kafka Connect in production. It details the distributed mode architecture, Kubernetes deployment considerations, strategies for scaling workers and tasks, tuning for throughput (data formats, batching, SMTs, worker resources), error handling patterns (backpressure, Dead Letter Queue, retries, idempotency), and security best practices including transport encryption, authentication, authorization, and secret management using Secret Providers.
  30. Why Microservices Need Event-Driven Architectures for Agility and ScaleThis post explains how event-driven architectures, enabled by technologies like Kafka, solve the challenges of microservices communication that arise from synchronous API-based patterns. It details how asynchronous event streams decouple services, prevent cascading failures, reduce latency, and improve scalability. The post also discusses the business benefits and common industry use cases for event-driven microservices, and introduces the concept of data governance for managing event streams at scale.
  31. How to Build Real-Time Apache Kafka® Dashboards That Drive ActionThis post details how to build real-time Apache Kafka dashboards that drive action by leveraging Kafka as an integration hub and stream processing engines like Flink, Kafka Streams, or ksqlDB to create materialized views for immediate consumption by BI tools or custom applications. It emphasizes data modeling for real-time performance, including denormalized schemas and effective Kafka partition strategies, and discusses delivery patterns such as read-optimized views, search-optimized indexes, and app-embedded state using Kafka Streams interactive queries.
  32. Apache Kafka 4.1 Release: New Features & Upgrade GuideThis post announces Apache Kafka 4.1.0, highlighting new features and upgrade paths. Key additions include: Queues for Kafka (preview), a new Streams Rebalance Protocol (early access) based on the consumer group protocol, and improvements to Kafka Broker, Controller, Producer, Consumer, and Admin Client. Notable KIPs include plugin metric registration, consistent transaction error handling, consumer close options for group leaving, rack topology change rebalancing, unified consumer topic metrics, deadlock protection for producer network threads, OAuth JWT-bearer grant type support, endpoint listener name cleanup, and transactional ID pattern filtering for ListTransactions API. Kafka Streams sees KIPs for moving window configurations to SerDes, the new Streams Rebalance Protocol, and enforcing explicit naming for internal topics. Kafka Connect gains support for running multiple versions of connector plugins and KIP-877 for plugin metric registration.
  33. Check out the latest in the Confluent Cloud ConsoleIntroduced a new topic message browser with enhanced capacity (1 million messages), human-readable ISO timestamps, and a histogram for message distribution. Added schema-compliant message production directly within the console. Overhauled the clients overview page to display client IDs, software types/versions, and connection metrics, specifically highlighting deprecated requests related to KIP-896. Enhanced consumer group monitoring with time charts for production, consumption, and lag, and enabled direct offset resetting from the console. Introduced a dedicated Kafka Streams view for application monitoring. Added functionality to export topic lists to CSV, view consumer group protocol types, and display schema IDs in the message browser.
  34. Confluent Platform 8.0: Client-side field level encryption (GA), and MoreConfluent Platform 8.0 introduces KRaft as the default metadata management system, replacing ZooKeeper, which simplifies Kafka architecture, improves scalability (millions of partitions), and enables near-instant controller failover. The next-generation Confluent Control Center is now Prometheus-based, offering significant performance improvements: 2x faster metrics, support for up to 400K partitions, metrics freshness within 2-3 minutes, and 15x faster startup times (1 minute). It also integrates Confluent Manager for Apache Flink (CMF) for centralized management of Flink environments and applications. Client-Side Field Level Encryption (CSFLE) is now Generally Available, providing granular encryption of individual message fields for enhanced data protection and compliance. FlinkSQL is available in Open Preview via REST and CLI. Kafka Queues (Share Groups) are in Early Access, offering a native, cooperative consumption model for queue-style workloads. Confluent Community software will now align more closely with the Kafka release cycle. Passwordless authentication is now supported for Confluent Server and Schema Registry using OAuth client credentials grant type with pre-signed assertions. CFK 2.11.1 and Confluent Ansible 8.0.0 have been released to support these updates, with CFK allowing side-by-side deployment of new and legacy Control Center.
  35. KIP-848: A New Consumer Rebalance Protocol for Apache Kafka® 4.0Introduces KIP-848, a new, optional consumer rebalance protocol for Apache Kafka 4.0+. This protocol shifts coordination from clients to the broker-side group coordinator, enabling server-driven, incremental reconciliation. Key changes include a shift from stop-the-world rebalances to incremental partition assignment/revocation via heartbeats, server-side assignors, and dynamic server-side configuration of session and heartbeat intervals. This results in reduced consumer downtime, faster rebalances, improved stability, and simplified management. The post details the protocol's mechanics, benefits, and adoption steps, including support for live upgrades.
  36. Next Generation of Control Center for Confluent PlatformIntroduces a new Prometheus architecture for Confluent Control Center, enabling support for up to 400K partitions, reducing start-up time to 1 minute, and achieving metrics freshness within 2-3 minutes. Eliminates the need for a separate Kafka cluster for metrics storage.
  37. How to build a multi-agent orchestrator using Flink and KafkaThis post details the architecture and implementation of a multi-agent orchestrator using Apache Flink and Apache Kafka. It explains how Flink can be used for real-time message interpretation and routing, with Kafka serving as the messaging backbone and short-term shared memory. The post outlines a reference architecture for event-driven multi-agent systems, including message production, Flink processing and routing via LLM inference, and agent execution. A practical example of an AI-driven sales development representative (SDR) system is provided, showcasing how Flink SQL can be used to dynamically assign tasks to different agents based on message content and agent descriptions.
  38. Kafka Consumer Offsets Guide—Basic Principles, Insights & EnhancementsThis post introduces KIP-1094, which adds a new constructor method with nextOffsets to ConsumerRecords, enabling more accurate and reliable retrieval of offset and leader epoch information. It also elaborates on manual offset committing strategies (commitSync, commitAsync) and highlights the critical importance of committing the leader epoch alongside the offset to ensure consumer group consistency and prevent issues like OFFSET_OUT_OF_RANGE errors during leader changes and rebalances.
  39. Use Freight Clusters and Lambda for Cost-Effective Streaming PipelinesThis post introduces Confluent Cloud Freight clusters as a cost-effective solution for high-throughput, relaxed latency Kafka workloads, up to 90% cheaper than self-managed Kafka. It details how to integrate these serverless clusters with AWS Lambda functions using various patterns (managed/self-managed connectors, Lambda ESM, EventBridge Pipes) for serverless stream processing. The post provides a specific e-commerce use case for ingesting customer actions and sending survey links, highlighting the use of Lambda's Kafka Event Source Mapping (ESM) with Provisioned Mode for fine-tuned throughput control, VPC-connected Lambda functions for private resource access, and event filtering for cost optimization.
  40. Apache Kafka 4.0 Release: Default KRaft, Queues, Faster RebalancesApache Kafka 4.0 defaults to KRaft mode, deprecates ZooKeeper, and introduces a new consumer rebalance protocol for faster rebalances. Queues for Kafka (early access) enable traditional queue semantics. Eligible Leader Replicas (ELR) and Pre-Vote mechanisms enhance KRaft stability. Client metrics collection is extended, and duration-based offset reset is supported. Group administration is improved with new command-line tools. Client resilience is enhanced with rebootstrap capabilities. Support for old client protocol API versions, message formats v0/v1, and Java 8 is dropped. Kafka Streams allows foreign key extraction from both key and value in KTable joins, custom processor wrapping, and adds a 'retry' return-option to ProductionExceptionHandler.
  41. Four Design Patterns for Event-Driven, Multi-Agent SystemsThis post introduces four design patterns (orchestrator-worker, hierarchical agent, blackboard, market-based) for building event-driven, multi-agent systems. It details how to adapt these patterns using Apache Kafka for agent coordination, communication, and fault tolerance, moving away from traditional request/response models to a reactive, event-driven approach. The post emphasizes the operational simplifications and scalability benefits gained by leveraging Kafka's data streaming capabilities for agent interactions.
  42. Provisioned Mode for Lambda's Kafka ESM: New Controls and AutoscalingIntroduces Provisioned Mode for AWS Lambda's Kafka Event Source Mapping (ESM), enabling granular control over throughput via minimum and maximum event pollers. This mode simplifies networking by eliminating the need for PrivateLink or NAT Gateway, and provides responsive auto-scaling based on offset lag. Performance benchmarks show significant improvements in message processing time compared to the default On-Demand mode, with specific scenarios demonstrating up to 78.57% faster backlog draining.
  43. Automating Podcast Promotion with AI and Event-Driven DesignThis post details the implementation of an event-driven architecture using Kafka and Flink to automate podcast promotion. It covers downloading podcast episodes, converting audio to text using OpenAI's Whisper model, splitting audio files due to size limits, and using an LLM (GPT-4) via Flink SQL for LinkedIn post generation. The architecture decouples the front-end application from the AI workflow, enabling independent scaling and model swapping. It highlights the use of Kafka topics for inter-service communication and Flink SQL for stream processing and AI model integration.
  44. Confluent Cloud Freight Clusters in GAIntroduces Confluent Cloud Freight clusters, a new cluster type optimized for cost-effective, high-throughput, latency-insensitive workloads. Key technical advancements include autoscaling to optimize resource utilization, direct writes to object storage (e.g., S3) to bypass local storage and inter-AZ replication, and the use of Fetch From Follower to reduce cross-zone traffic costs. The post also highlights the integration with Private Networking Interface (PNI) for secure and cost-effective VPC interconnection. Freight clusters are built on the Kora engine and can scale from 2 to 152 eCKUs.
  45. Allium and Confluent: How to Build a Foundational Data Platform for BlockchainThis post details how Allium leverages Confluent's data streaming platform to build a foundational data platform for blockchain data. It covers challenges in accessing and processing blockchain data, and how Confluent's managed connectors, Stream Sharing, and cloud-agnostic nature enable Allium to index more blockchains, provide real-time data to customers, and build real-time applications. The future direction involves shifting processing and governance left using Confluent Cloud for Apache Flink and Tableflow to ensure data quality at the source.
  46. Apache Kafka 3.9 Release: Dynamic KRaft Quorums, Better ZK MigrationIntroduces Dynamic KRaft Quorums (KIP-853) allowing dynamic addition/removal of controller nodes via tooling or AdminClient API, improving operational flexibility for KRaft mode. Enhances ZooKeeper migration tooling, making Kafka 3.9 the final and best iteration before ZooKeeper mode is removed in 4.0. Makes Tiered Storage (KIP-405) production-ready with new features like dynamic disablement (KIP-950), quotas (KIP-956), and exposed remote offsets (KIP-1005). Adds Kafka Streams config for log summary interval (KIP-1049) and an exception handler (KIP-1033). Kafka Connect improves null handling in transformations (KIP-1040), adds offset control in MirrorSourceConnector (KIP-1031), and introduces a health check endpoint (KIP-1017).
  47. Preventing and Fixing Bad Data in Event Streams, Part 1This post delves into the challenges of handling 'bad data' within event streams, particularly in Apache Kafka, due to its immutable log nature. It contrasts batch processing techniques for data quality management (ETL/ELT, full refresh, incremental processing) with the complexities of streaming. Key takeaways include the reactive nature of data engineering, the reliance on deletion and reprocessing in batch, and the difficulty of applying these methods directly to event streams. The post sets the stage for discussing specific strategies for event streams in a subsequent part.
  48. Revolutionizing Computational Chemistry with Data StreamingThis post details how Atomic Tessellator, a computational chemistry startup, uses Confluent and Apache Kafka to build a scalable and flexible platform for molecular research. It describes their transition from a batch-based ETL process using RabbitMQ, Celery, and a Django app to a data streaming architecture. The new architecture enables rapid ingestion of complex molecular data from databases and generative models into Confluent, followed by streaming to downstream "evaluation" applications. This facilitates quick and reliable replication of chemical reactions for catalyst discovery. The post highlights business benefits such as avoiding over-provisioning of compute resources, the ability to tweak pipelines mid-workload, and clear visibility into production pipeline status through JMX monitoring. It also mentions plans to explore Apache Flink for building a "pipeline templates" system.
  49. Kafka Producer Internals: The Producer Request LifecycleThis post details the internal lifecycle of a Kafka producer request on the broker, covering the stages of socket receive buffer, network threads, request queue, I/O threads, page cache and disk writes (including segment structure and flush configurations), purgatory for replication (and its relation to acks and replica fetchers), response queue, network thread handoff, and socket send buffer. It also enumerates specific configuration parameters and monitoring metrics for each stage, such as socket.receive.buffer.bytes, socket.request.max.bytes, num.network.threads, NetworkProcessorAvgIdlePercent, queued.max.requests, queued.max.request.bytes, RequestQueueSize, RequestQueueTimeMs, num.io.threads, RequestHandlerAvgIdlePercent, log.flush.interval.ms, log.flush.interval.messages, log.segment.bytes, cleanup.policy, LogFlushRateAndTimeMs, LocalTimeMs, default.replication.factor, replication.factor, replica.fetch.wait.max.ms, num.replica.fetchers, RemoteTimeMs, ResponseQueueSize, ResponseQueueTimeMS, and ResponseSendTimeMs.
  50. Inside the Kafka Black Box—How Producers Prepare Event Data for BrokersThis post dives into the technical details of Kafka producer behavior, explaining the serialization process (using serializers and Schema Registry), partitioning logic (default hash-based, sticky, and adaptive strategies), and batching configurations (`batch.size`). It provides insights into how producers prepare data before sending it to brokers, offering configuration parameters and their implications for performance and data distribution.
  51. Effectively Migrating Kafka to the Cloud: How Skai Did ItSkai re-architected their Kafka data ingestion to drastically reduce topic count for scalability and cost-efficiency in Confluent Cloud. They moved from a per-table topic model to consolidated topics ('united-topic-single-pipeline', 'united-topic-multiple-pipelines', 'united-topic-heavy-load') based on traffic. This involved developing a dynamic lag handling API to promote tables to higher-tier topics and a caching library for external Kafka clients to manage dynamic topic mappings. They also implemented enhanced monitoring to pinpoint lag causes within consolidated topics.
  52. AK 3.8 Adds 17 KIPs: New Docker Images and much moreApache Kafka 3.8.0 introduces 17 KIPs, including support for specifying compression levels (KIP-390), a preview of the next-generation consumer rebalance protocol (KIP-848), Docker images for GraalVM-based native Kafka brokers (KIP-974) and official Kafka Docker images (KIP-1028). It also deprecates `offsets.commit.required.acks` (KIP-1041) and the Log4j Appender (KIP-719). Kafka Streams gains shareable state stores (KIP-813) and customizable task assignment (KIP-924). Kafka Connect now enforces the `tasks.max` property (KIP-1004).
  53. Learn the confluent-kafka-javascript Client by Building an AppIntroduces and demonstrates the use of the confluent-kafka-javascript client, a new library for interacting with Apache Kafka and Confluent Platform. The post details how to use the client for producing messages with manual offset commits and consuming messages, including handling the last clicked state and emitting updates via WebSockets. It showcases the client's two API styles: one based on node-rdkafka and a promisified API similar to KafkaJS. The example application built is a real-time voting app to settle the tabs vs. spaces debate.
  54. Unlocking the Edge: Data Streaming Goes Where You Go with ConfluentThis post details how Apache Kafka is being used at the edge, addressing challenges related to disconnected, intermittent, and constrained networking, low-latency demands, and cost optimization. It explains Kafka's role as a buffer for local data collection and replication, and how stream processing can optimize data sent to the cloud. The post also highlights Confluent's solutions for deploying and operating Kafka at the edge, including Confluent for Kubernetes, Confluent Ansible, and Health+ for observability.
  55. Kafka Scaling Reinvented with Elastic CKUsIntroduces Elastic Confluent Units (eCKUs) for Confluent Cloud's Enterprise clusters, enabling serverless, instantaneous autoscaling of Kafka resources. This addresses challenges with fluctuating workloads, unpredictable spikes, and rapid growth by automatically allocating and deallocating resources (throughput, partitions, client connections) as needed, eliminating manual capacity planning and over-provisioning.
  56. Transforming T+1 Trade Settlement for Modern Markets With ConfluentThis post applies Confluent's data streaming platform, specifically Kafka, to the financial services domain for T+1 trade settlement. It details how Kafka can ingest, store, and distribute trade data in real-time, overcoming batch processing limitations. Key technical aspects include using Confluent connectors for data ingestion from various financial systems (e.g., Oracle CDC, AMPS), leveraging the Kora engine for elasticity and reliability in handling market volatility, and employing stream processing (e.g., windowed sorting, KTables, point-in-time joins) to enrich and deliver data to siloed processes. It also mentions Role-Based Access Control (RBAC) for security and data governance. The post illustrates how these components enable faster settlement, reduce risk, and meet regulatory requirements.
  57. Introducing Apache Kafka 3.7Introduces early access support for JBOD disk failure handling in KRaft (KIP-858). Adds broker-side support for client-level metrics via a standardized telemetry interface (KIP-714) and a mechanism to configure these metrics (KIP-1000). Previews the next generation of the Consumer Rebalance Protocol for improved efficiency (KIP-848). Optimizes leader discovery for clients to reduce latency during leadership changes (KIP-951). Provides an official Apache Kafka Docker image (KIP-975). Implements exponential backoff for Kafka client retries to reduce overload after broker failures (KIP-580). Adds new metrics for Tiered Storage monitoring (KIP-963). Enables independent stopping of KRaft processes (KIP-979). Adds server-side transaction verification to prevent hanging transactions (KIP-890). Marks old client protocol API versions (pre-2.1) as deprecated for removal in Kafka 4.0 (KIP-896). Allows AdminClient to communicate directly with the KRaft Controller Quorum for operations like DESCRIBE_QUORUM and INCREMENTAL_ALTER_CONFIGS (KIP-919). Enables dynamic reloading of certificates with different DN/SANs (KIP-978). Adds a `CurrentControllerId` metric to identify the current controller (KIP-1001). Introduces 11 new metrics for measuring KRaft performance (KIP-938). Deprecates Java 11 support for Kafka broker and tools, planned for removal in Kafka 4.0 (KIP-1013). In Kafka Streams, introduces rack-aware task assignment strategies (`balanced_subtopology`) (KIP-925). Expands default DSL store configuration to custom types with support for RocksDB and in-memory stores (KIP-954). Relaxes the non-null key requirement for certain left-joins (KIP-962). Adds a new interface for handling standby task updates (KIP-988). Adds `VersionedKeyQuery` and `MultiVersionedKeyQuery` for querying versioned state stores (KIP-960, KIP-968). Adds `reverseRange` and `reverseAll` queries for key-value stores in IQv2 (KIP-985).
  58. Apache Kafka® vs Confluent Cloud: Latency Benchmarking ResultsThis post details benchmarking results comparing Confluent Cloud's Kora engine performance against Apache Kafka. It highlights significant improvements in p99 end-to-end latency, with Confluent Cloud outperforming Apache Kafka by up to 16x under various throughput and partition scaling scenarios. The post also discusses the methodology for measuring latency, including steady-state and real-world conditions, and the benchmark setup, emphasizing the use of CKUs for consistent resource allocation in Confluent Cloud and equivalent hardware configurations for Apache Kafka. It also touches upon Kora's ability to maintain low latency even during cloud provider disruptions.
  59. How Network Automation with Confluent Helps BT Stream TV Signals Across the UKThis post details BT's implementation of Confluent Platform and Apache Kafka for network automation in their TV signal distribution infrastructure. It describes the Vena orchestration system, which leverages Kafka for reliable message distribution and event-driven architectures. The post outlines two key use cases: event-driven network onboarding using LLDP events and syslog connectors, and real-time network monitoring and fault detection. It highlights the technical reasons for choosing Kafka (de facto standard, ecosystem, stream processing) and Confluent (resilience, automated deployment, replication, monitoring, security, support). The post also discusses the challenges of a legacy network and the need for scalability and high availability in handling network events and potential alarm storms.
  60. Best Practices For Managing Enterprise Apache Kafka® ClustersThis post provides guidance on managing enterprise Apache Kafka clusters, advocating for a platform approach to reduce complexity and cost. It details strategies for consolidating siloed Kafka deployments into a more centralized model, discussing the benefits of standardization, improved service quality, and reduced governance risks. The post also outlines the trade-offs of full centralization, such as meeting diverse requirements and organizational transformation challenges. It references data mesh, data products, and domain-driven design as conceptual frameworks for building effective data streaming platforms.
  61. How Cloud Services Are Helping NASA Take Kafka Farther, FasterThis post details NASA's adoption and expansion of Apache Kafka, particularly leveraging Confluent Cloud for its GCN project and future initiatives like the Data and Reasoning Fabric. It highlights the use of data streaming for real-time astronomical alerts and the potential for drone delivery systems, emphasizing the need for immediacy and scalability. NASA's journey involved moving from self-managed Kafka to Amazon MSK and finally to Confluent Cloud, driven by the desire for reduced operational overhead and enhanced capabilities. The post also touches upon NASA's distributed IT structure and the challenges of lowering the barrier to cloud adoption for its mission teams.
  62. Apache Kafka Message CompressionThis post details Apache Kafka's message compression capabilities. It explains how compression works in Kafka producers by batching data to the same partition, the impact of compression on throughput, network traffic, and disk utilization. It lists available compression algorithms (none, gzip, snappy, lz4, zstd) and provides recommendations (lz4 for performance, zstd for gzip-like ratio with less overhead). The post elaborates on configuring `compression.type` at both the topic and producer levels, detailing precedence and common/uncommon combinations. It also covers broker-side decompression scenarios and how consumers handle compressed messages. Finally, it offers advice on avoiding compression with encrypted data and inspecting compression codecs on brokers.
  63. Confluent’s Kora Engine awarded ‘Best Industry Paper’ at VLDBThis post details the Kora Engine, Confluent's cloud-native event streaming platform, which powers Confluent Cloud. It highlights Kora's architecture designed for reliability, elasticity, and cost-efficiency, including its cellular architecture for multi-tenancy, serverless abstraction, self-balancing clusters, automatic failure mitigation for high availability, and seamless replication/backup using object storage. The post also mentions the award received at VLDB 2023 for the Kora paper.
  64. Kafka Connect: How to Increase Throughput on Source ConnectorsThis post provides detailed guidance on tuning Kafka Connect source connectors to increase throughput. It covers what aspects of a connector can be tuned (connector configurations, producer configurations) and what cannot (converter, transformations). Specific producer configurations like batch.size, linger.ms, buffer.memory, and compression.type are explained with examples. It also lists essential JMX metrics for monitoring connector, broker, and producer performance, and provides a walk-through example using a JDBC Source connector to demonstrate the tuning process.
  65. What Is an Apache Kafka Cluster?This post provides a foundational explanation of Apache Kafka clusters, detailing the roles of brokers, partitions, and replication in achieving speed, durability, and scalability. It introduces KRaft as the successor to ZooKeeper for metadata management, explaining its function in leader election and replica failure recovery. The post also briefly discusses on-premise vs. cloud deployment options for Kafka.
  66. Kafka Clusters and Cluster ManagementThis post addresses the decision-making process for creating new Kafka clusters, detailing physical limitations (e.g., 200,000 partitions per cluster, 4,000 partitions per broker) and operational concerns like data privacy, noisy neighbors, and chargeback. It argues for the benefits of a single cluster for real-time data correlation and introduces Confluent Cluster Linking and Schema Linking as mechanisms for inter-cluster data exchange. It also provides a federated governance model for operating large Kafka deployments, distinguishing responsibilities between shared services and line-of-business teams, with a focus on security, capacity assurance, and self-service operations, and suggests Confluent Cloud as a SaaS solution for simplified management.
  67. How BigCommerce Uses Data Streaming to Bring Real-Time Insights to MerchantsThis post details BigCommerce's migration from self-managed Kafka and a Hadoop MapReduce-based system to Confluent Cloud. It highlights the technical challenges of managing Kafka clusters, including provisioning, scaling for seasonal traffic, software upgrades, and the operational burden on their engineering team. The migration strategy involved a dual-write approach to ensure zero downtime and no data loss. The post emphasizes the benefits of Confluent Cloud for real-time data streaming, reduced operational overhead, and automatic scaling capabilities, enabling BigCommerce to provide real-time analytics to merchants.
  68. Kora: The Cloud Native Engine for Apache KafkaThis post introduces Kora, Confluent's cloud-native engine for Apache Kafka, detailing its architecture and benefits. Kora is designed from the ground up for cloud constraints, focusing on multi-tenancy, disaggregation of compute, network, metadata, and storage layers, intelligent data locality management (memory, SSD, object storage), performance optimization for cloud environments, real-time usage capture for operations, and cost optimization. It enables elastic scaling (30x faster), higher availability (99.99% SLA), and lower latency compared to self-managed Kafka. The post also touches on continuous improvements through multi-phase fleet updates and advanced data balancing algorithms.
  69. Streaming Data Quality, Integrity, and Custom Connectors with Kora and ConfluentIntroduces Kora, a cloud-native Kafka engine for Confluent Cloud, offering enhanced elasticity, reliability, and performance. Enhances Stream Governance with Data Quality Rules (Domain Validation, Event-Condition-Action, Complex Schema Migration) for data integrity and contract enforcement. Introduces Custom Connectors for bringing user-defined Kafka Connect plugins to Confluent Cloud without infrastructure management. Launches Stream Sharing for secure, easy real-time data exchange across organizations. Mentions early access to Apache Flink.
  70. With Confluent Cloud, Loggi boosted productivity and cut costsLoggi details their adoption of an event-driven architecture using Apache Kafka and Confluent Cloud to scale their logistics platform. They migrated from a monolithic system, focusing on improving developer productivity and system resilience. Key technical implementations include the transactional Outbox Pattern for producing events with strong transactional guarantees (ensuring database writes and Kafka event production are atomic) and an Event Broker microservice that consumes events from Kafka and delivers them to consumer services via gRPC. This design abstracts Kafka complexity for engineers, allowing them to focus on business logic. They also discuss lessons learned regarding potential bottlenecks in the Event Broker microservice, organizational trade-offs, and the importance of observability and performance tuning, including the use of dedicated Kafka consumers for high-volume events.
  71. Understanding Kafka’s Engineering CostsThis post analyzes the engineering costs of running Kafka, breaking them down into infrastructure, development, and operations. It quantifies these costs by estimating the number of engineers required for different scales of operation and their associated salaries and fully loaded employment costs. It also discusses capacity planning challenges, the need for platform development beyond core Kafka (e.g., connectors, security, monitoring), and the complexities of cluster scaling and ongoing maintenance. The post highlights the opportunity cost of dedicating engineering talent to infrastructure management versus strategic business initiatives and contrasts these self-managed costs with the benefits of managed services like Confluent Cloud.
  72. Delivery Guarantees in Messaging SystemsThis post explains the concepts of at-most-once, at-least-once, and exactly-once (effectively once) delivery guarantees in messaging systems. It uses the Two Generals Problem as an analogy to illustrate the challenges of achieving reliable communication over unreliable networks. The post discusses the implications of each delivery guarantee, drawing parallels to science fiction teleportation scenarios. It concludes by relating these concepts to modern software development, especially in distributed systems and financial transactions, and mentions Apache Kafka's transactional guarantees as a solution.
  73. Uncovering Kafka’s Hidden Infrastructure CostsThis post details the infrastructure costs of running Kafka, highlighting networking as the biggest cost driver (over 50% of infrastructure costs). It breaks down costs for compute, storage, and networking, providing example calculations for different ingress rates. It emphasizes the importance of cross-AZ traffic for producers, consumers, and partition replication as the primary driver of networking costs. The post also touches upon the impact of utilization and introduces a four-part series on understanding and optimizing Kafka costs, with subsequent posts focusing on operations, general cost reduction, and a savings challenge.
  74. How Michelin Cut Kafka Costs by 35% Using ConfluentThis post details Michelin's successful migration from self-managed on-prem Kafka to Confluent Cloud, achieving a 35% cost reduction. It elaborates on the operational burden of self-managing Kafka, the scaling challenges encountered, and the benefits of a managed service for reliability and faster time-to-market. The migration allowed Michelin to reallocate dedicated Kafka teams to innovation and higher-value activities, improving inventory management and customer experience.
  75. Windowing in Kafka StreamsIntroduces and explains the four types of windowing in Kafka Streams and ksqlDB: Hopping, Tumbling, Session, and Sliding. Details configuration parameters for each (window size, advance size, inactivity gap, time difference) and provides examples. Discusses the handling of late events with grace periods and notes differences in grace period defaults between Kafka Streams and ksqlDB.
  76. Configuring Apache Kafka Consumer Group IDsThis post elaborates on the fundamental concept of Kafka consumer group IDs, detailing their role in detecting new data via offsets, enabling work sharing among consumers for parallel processing, and ensuring fault tolerance through rebalancing mechanisms. It introduces the concept of static membership (using `group.instance.id`) as an optimization for applications with large local state to reduce rebalance-related data transfer and improve rolling upgrades.
  77. Diagnose and Debug Apache Kafka Issues: Understanding Increased ConnectionsThis post focuses on diagnosing and debugging increased connection issues in Apache Kafka. It details the types of connections across the cluster (producer, consumer, broker), explains relevant metrics like connection-count, connection-creation-rate, and AcceptorBlockedPercent, and explores common root causes such as undersized workloads, incorrect consumer-per-thread patterns, and multiple KafkaProducer instances within a single process.
  78. Diagnose and Debug Apache Kafka Issues: Understanding Increased Consumer Rebalance TimeThis post focuses on diagnosing and debugging increased consumer rebalance time in Apache Kafka. It details the mechanics of Stop-the-World and Incremental Cooperative rebalancing, introduces Static Group Assignment as a method to reduce rebalance impact, and highlights key consumer metrics (join-rate, records-lag-max) for monitoring rebalance health. It also identifies common root causes for increased rebalance time, including one-consumer-per-thread patterns, undersized KafkaConsumer instances, and overloaded group coordinators.
  79. Diagnose and Debug Apache Kafka Issues: Understanding Increased Request Rate, Response Time, and/or Broker LoadThis post details the client request lifecycle in Kafka (producer.send() and consumer.poll()) and identifies key JMX metrics for monitoring broker health, including RequestMetrics, SocketServer, KafkaRequestHandlerPool, and RequestChannel metrics. It also provides diagnostic questions to help identify root causes of increased request rates, response times, and broker load, such as inefficient batching, client workload sizing, and excessive metadata requests.
  80. Reduced Message Throughput | Diagnose and Debug Kafka Issues SeriesThis post provides a detailed guide to diagnosing reduced message throughput in Apache Kafka. It focuses on producer-side metrics such as batch-size-avg, records-per-request-avg, record-size-avg, record-queue-time-avg, request-latency-avg, and requests-in-flight. It also explains how to tune producer configurations like batch.size, linger.ms, and buffer.memory to optimize batching. Additionally, it suggests investigating topic partitions, request rates, and the number of KafkaProducer instances as potential root causes for throughput issues.
  81. What’s New in Health+: 7 New Dashboards, 70% Monitoring Infra Savings | ConfluentIntroduces seven new Health+ dashboard categories (Kafka Health, System, Producers, Consumers, Throughput, Thread Utilization, Topics, ksqlDB, Connect) to provide deeper insights into Kafka and Confluent Platform health. Explains the telemetry data collection and transmission process to Confluent Cloud, emphasizing that message content is not collected. Discusses security and privacy aspects of Health+ and provides guidance on getting started.
  82. How to Remove Apache Kafka Brokers the Easy WayIntroduces the ability to safely remove multiple Kafka brokers with a single command, addressing previous limitations of removing only one broker at a time and the risk of under-replicated partitions. Details the use of Confluent's Self-Balancing Clusters for intelligent partition reassignment and bin-packing, operation persistence using a state machine and a Kafka topic for resilience, preliminary validations to prevent capacity issues, and a new Broker Replica Exclusion API to handle race conditions with new topic creations.
  83. How to Make Apache Kafka Clients Go Fast(er) on Confluent Cloud | Confluent How to Make Apache Kafka Even FasterThis post focuses on optimizing Kafka client performance on Confluent Cloud for low latency and high throughput. It highlights five key resources that delve into cloud-native Kafka design principles, client configuration parameters (producer and consumer), benchmarking methodologies, and latency-influencing factors within Kafka. It also touches upon the 'time to market' aspect by emphasizing Confluent Cloud's managed nature reducing operational burden.
  84. 5 Common Pitfalls When Using Apache KafkaThis post details common pitfalls when using Apache Kafka, covering client-side issues like setting request.timeout.ms too low and misunderstanding producer retries and retriable exceptions. It also addresses broker-side issues such as leaving key broker metrics unmonitored, going overboard with partitions, and setting segment.ms too low. Specific configuration parameters and their implications are discussed, along with recommendations for optimal settings.
  85. Modern Messaging and Real-Time Data Integration with ConfluentThis post details how Confluent enables modernization of legacy messaging architectures by integrating with existing middleware (RabbitMQ, ActiveMQ, IBM MQ, TIBCO EMS) using Kafka Connectors. It highlights the use of Schema Registry for metadata integration and governance, and ksqlDB for stream processing and analysis of data in motion. The post emphasizes the benefits of a fully managed cloud service for these capabilities, reducing complexity and cost compared to traditional setups. An example walkthrough demonstrates the end-to-end integration process.
  86. Rethinking Distributed Stream Processing in Apache KafkaThis post elaborates on consistency and completeness guarantees in stream processing, specifically within Apache Kafka and Kafka Streams, detailing how persistent logging, idempotent and transactional writes, and speculative processing with revision enable exactly-once semantics and ordered results even with out-of-order data. It highlights the decoupling of latency, throughput, and correctness trade-offs through Kafka's log-based approach.
  87. Error Handling Patterns in KafkaIntroduces four error handling patterns for Kafka applications: stop-on-error, dead-letter queue, retry topic with potential out-of-order processing, and a more complex retry topic pattern that maintains original event order using in-memory stores and redirect topics for tombstone events. Discusses recovery mechanisms for the in-memory store.
  88. Performance Tuning RocksDB for Kafka Streams’ State StoresThis post details performance tuning for RocksDB state stores within Kafka Streams. It explains how Kafka Streams uses RocksDB for stateful operations, introduces RocksDB's internal workings including compaction styles (level and universal), and provides guidance on configuring RocksDB via RocksDBConfigSetter. It also highlights the importance of RocksDB-specific metrics (introduced by KIP-471 and KIP-607) for diagnosing and resolving operational issues with state stores.
  89. How Pinterest Runs Kafka at ScaleThis post details Pinterest's operational experience running Kafka at scale, focusing on challenges and solutions related to performance and cost control. Key technical aspects include the transition from magnetic disks to SSDs for improved IOPS and reduced latency during broker recovery, the adoption of a static assignment model ('brokerset's) instead of dynamic rebalancing, and efforts to manage message format versions to reduce CPU load. Cost control measures include rack-aware data transfer strategies, enforcing compression on the producer side, and optimizing topic retention and replication factors. The post also touches upon the use of Singer for log ingestion and Maxwell for database changelogs.
  90. Announcing ksqlDB 0.15ksqlDB 0.15 introduces expanded support for message keys, allowing structured data types (ARRAY, STRUCT) and various serialization formats (JSON, AVRO, PROTOBUF, DELIMITED, KAFKA, JSON_SR) in keys. It also enables PARTITION BY and GROUP BY clauses with multiple expressions for tables and streams with multiple key or primary key columns. Additionally, the DROP command now automatically terminates persistent queries associated with a stream or table, simplifying the management of ksqlDB resources.
  91. Spring Microservices into Production with Kubernetes and GitOpsThis post introduces the use of Spring Boot for developing Kafka Streams microservices, detailing its benefits for Kafka integration, configuration, dependency management, and testing. It also describes the deployment of these microservices to Kubernetes using a GitOps workflow, including the use of Kustomize for managing deployments and the integration with the existing streaming-ops project.
  92. Introducing Confluent’s Parallel Consumer Message Processing ClientIntroduces the Confluent Parallel Consumer, a new client library that provides key-level and unordered parallelism for Kafka message consumption, surpassing the limitations of partition-level parallelism. It details how this library enables higher throughput by allowing more concurrent processing than available partitions and by efficiently handling external service calls (e.g., HTTP requests to legacy services) through non-blocking I/O and optional Vert.x integration. The post also highlights features for client-side work queues and message-level acknowledgment, improving performance for task queue use cases.
  93. Apache Kafka and Scala - A Beginner’s TutorialThis post provides a beginner's tutorial for Scala developers on producing and consuming records in Apache Kafka. It covers setting up a Scala project with Kafka clients and Confluent serializers, configuring producers with Avro serialization and Schema Registry integration, and demonstrates three production scenarios: batching records with linger.ms, sending single events with event time and headers, and performing transactional writes. It also highlights the use of Avro4s for serialization and pureconfig for configuration.
  94. Get Automatic, Self-Balancing Kafka Clusters in Confluent Platform 6.0Introduces Confluent Platform's Self-Balancing Clusters (SBC) feature, which automates Kafka cluster scaling and load balancing. SBC monitors partition layout and broker load, automatically reassigning partitions to balance replica counts, leader counts, disk usage, and network usage. It handles broker additions, removals, and failures, respecting rack awareness and capacity limits. Rebalancing is throttled to minimize impact on ongoing traffic.
  95. Benchmarking RabbitMQ vs Kafka vs Pulsar PerformanceThis post provides a detailed performance comparison of Apache Kafka, RabbitMQ, and Apache Pulsar, focusing on system throughput and end-to-end latency. It details the benchmarking framework used, including fixes made to the OpenMessaging Benchmark Framework (OMB) and its drivers for Kafka, RabbitMQ, and Pulsar. The post presents quantitative results for peak throughput and p99 latency under various load conditions, concluding that Kafka offers the best throughput and lowest latencies. It also discusses durability guarantees in distributed systems and the role of fsync.
  96. Spring Kafka Beyond the Basics - How to Handle Failed Kafka ConsumersThis post introduces strategies for handling deserialization failures (poison pills) in Spring Kafka consumers. It explains the causes of poison pills, including incompatible serializers and deserializers, and demonstrates how to configure Spring Kafka to gracefully manage these errors. The post highlights the importance of robust error handling for production deployments and provides an example project on GitHub for practical implementation.
  97. Kafka Streams 2.5.0 - Even Higher Availability & Interactive QueriesThis post details the introduction and impact of KIP-535 (allowing state stores to serve stale reads during rebalance), KIP-562 (allowing fetching a key from a single partition), and KAFKA-9169 (fixing unnecessary restoration of standbys) in Kafka Streams 2.5.0. These improvements address challenges at Walmart scale by increasing availability during VM unavailability and rebalances, reducing query latency, and preventing unnecessary resource consumption.
  98. Kafka Needs No Keeper - Removing ZooKeeper DependencyThis post details the technical plan and roadmap for removing ZooKeeper dependency from Apache Kafka through KIP-500. It explains the problems with external metadata management (duplication, inefficiency, scalability limits, de-synchronization) and proposes storing metadata in a Kafka partition managed by a Raft quorum. It outlines the controller architecture changes, including standby controllers and O(1) topic operations. The roadmap includes removing ZooKeeper from administrative tools, implementing self-managed metadata quorums with a pull-based Raft protocol (KIP-595), introducing KIP-500 mode (initially experimental), and a bridge release strategy for zero-downtime upgrades using new controller APIs like AlterIsr.
  99. Cooperative Rebalancing in the Kafka Consumer, Streams & ksqlDBIntroduces and explains the incremental cooperative rebalancing protocol for Kafka consumers, contrasting it with the eager rebalancing protocol. Details the mechanics of the JoinGroup and SyncGroup requests, the roles of group coordinator and leader, and the partition assignor's function in the new protocol. Explains how this protocol minimizes downtime by allowing consumers to retain ownership of unassigned partitions, benefiting Kafka Streams and ksqlDB.
  100. Highly Available, Fault-Tolerant Pull Queries in ksqlDBThis post introduces significant enhancements to ksqlDB's pull query availability and consistency. It details a new failure detection mechanism using heartbeating between ksqlDB servers, reducing detection time to seconds. It also introduces standby routing, allowing pull queries to be served by standby replicas during active server failures, and lag-aware routing, which selects the most up-to-date standby replica based on changelog topic offsets. These changes allow ksqlDB to offer tunable consistency/availability trade-offs, moving from a CP system to an AP system during partitions loss.
  101. Walmart’s Real-Time Inventory Management Powered by KafkaThis post details Walmart's implementation of a real-time inventory management system using Apache Kafka. It covers the engineering challenges and solutions for managing diverse data sources through a canonical approach and transformation engine. The post provides in-depth technical details on scaling Kafka by optimizing partitions, producer configurations (linger.ms, batch.size, acks), and consumer scaling strategies. It also addresses database integration challenges, specifically with Cassandra, by detailing partitioning strategies to ensure data consistency and minimize deadlocks. The post includes diagrams illustrating data flow and scaling concepts.
  102. Tail Latency at Scale with Apache KafkaThis post provides a deep dive into understanding and optimizing end-to-end latency in Apache Kafka. It breaks down latency into five components: produce time, publish time, commit time, catch-up time, and fetch time, explaining the factors influencing each. It also clarifies the relationship between end-to-end latency, producer latency, and consumer latency, and discusses strategies for bounding tail latencies by addressing head-of-line blocking and micro-bursting. The post emphasizes tuning client and broker configurations, and scaling Kafka clients for throughput while minimizing queueing delays and congestion.
  103. Kafka Streams: Turning Data at REST to Data in MotionThis post details the implementation of a near-real-time flight tracking dashboard using Kafka Streams. It describes how a Kafka producer was built to poll data from a REST API of international flight systems, transforming and enriching this data into Kafka topics. The Kafka Streams application then processes these events to perform aggregations (e.g., flights per airline, fastest flights) within 30-second windows, producing aggregated streams for visualization. The architecture decouples components using Kafka and utilizes WebSockets for streaming processed data to a Vue.js-based progressive web app.
  104. Announcing ksqlDB 0.7.0 | New Features and UpdatesThis post announces ksqlDB 0.7.0, introducing highly available state for queries during cluster rebalances by allowing reads from standby replicas, with configurable staleness tolerance. It also adds COUNT_DISTINCT and CUBE functions, expands key types beyond strings to include INT, BIGINT, and DOUBLE, and introduces first-class constructors for native arrays, maps, and structs.
  105. How Netflix Uses Kafka for Distributed StreamingThis post details Netflix's use of Apache Kafka for distributed streaming within their Content Finance Engineering team. It explains their adoption of an event-driven architecture for microservices, detailing the roles of producers (including CDC and normalized event formats), enrichers (using stream processing like Flink and ksqlDB for data enrichment and materialized views), and consumers (using Spring Boot and Schema Registry/Avro for schematized streams). It also covers strategies for ensuring event ordering (keyed messages, delayed materialization) and message delivery guarantees (idempotency with EVCache).
  106. Streams and Tables in Apache Kafka: Elasticity, Fault Tolerance & Advanced ConceptsThis post details how fault tolerance and elasticity are achieved in Kafka Streams and ksqlDB. It explains that fault tolerance for tables is maintained by storing state locally and also in Kafka changelog topics, which act as a durable source of truth. Elasticity is enabled by the same mechanism, allowing tasks and their state to be migrated between instances during scaling operations. The post highlights the benefits of topic compaction for reducing storage footprint and recovery time, and discusses operational aspects like rebalancing, recovery time, and the use of standby replicas to minimize downtime.
  107. Streams and Tables in Apache Kafka: Storage FundamentalsThis post delves into Kafka's storage fundamentals, focusing on topics and partitions. It explains how topics are unbounded sequences of events, configurable with retention policies, and how brokers are serialization-agnostic. The core concept of partitions is highlighted as fundamental for Kafka's scalability, elasticity, and fault tolerance, enabling parallel reads and fault tolerance through replication. The post details how producers determine event partitioning using a partitioning function, emphasizing the importance of sending related events to the same partition for ordering guarantees within consumer groups. It also provides guidance on choosing the number of partitions and discusses common reasons for events with the same key ending up in different partitions.
  108. Easy Kafka Streams Testing with TopologyTestDriver - KIP-470This post introduces KIP-470, which enhances the TopologyTestDriver for Kafka Streams testing. It adds TestInputTopic and TestOutputTopic classes, simplifying the process of feeding records and reading output. The post demonstrates how to use these new classes with AssertJ for testing single records, collections of records, and validating keys, values, timestamps, and headers. It also discusses migrating existing tests and provides links to external resources for further learning.
  109. Why Scrapinghub’s AutoExtract Chose Confluent CloudScrapinghub's AutoExtract service uses Confluent Cloud for its Kafka needs to scale its AI data extraction API. They partition URLs for fetching, rendering, and screenshotting using Kafka, and distribute resource-intensive AI data extraction tasks across multiple instances. The move to Confluent Cloud allowed them to offload Kafka infrastructure management, providing scalability and cost benefits. They detail their evaluation of alternatives like self-hosting Kafka on Kubernetes and Amazon MSK, ultimately choosing Confluent Cloud for its managed nature, consumption-based pricing, and vendor independence. They also describe initial setup, load testing, and workarounds for missing tools like Burrow by using a consumer metrics exporter.
  110. Kafka Graph Processing: Visual Stream Analytics with Neo4jThis post introduces the integration of Neo4j with Apache Kafka for graph processing and stream analytics. It details the use of the Neo4j-Streams plugin to ingest Kafka data into Neo4j, creating nodes and relationships. It also demonstrates how to apply graph algorithms like Common Neighbors to identify potential connections and output these results back to Kafka. The post provides configuration details for connecting Neo4j to Kafka using SASL_SSL and API keys, and includes a GitHub repository with code and setup instructions.
  111. Building Shared State Microservices for Distributed Systems Using Kafka Streams | ConfluentThis post details how Imperva used Kafka Streams to build shared state microservices for distributed systems. It covers the architecture involving a source, processor with a persistent key-value store, and a sink. It also explains how to build a CRUD API on top of Kafka Streams, distribute processing load across a cluster of microservices, and implement task scheduling using the processor context's schedule method. The post highlights the benefits of using Kafka Streams for fault tolerance, high availability, and scalability in managing shared state.
  112. Microservices, Apache Kafka, and Domain-Driven Design | ConfluentThis post discusses the symbiotic relationship between microservices, domain-driven design (DDD), and Apache Kafka. It explains how Kafka serves as an event streaming platform backbone for microservices, enabling decoupled communication through publish-subscribe, fault-tolerant storage, and real-time processing. The post elaborates on DDD principles like bounded contexts and ubiquitous language, and how Kafka facilitates their implementation by decoupling microservices and allowing teams to own their domains. It highlights Kafka's advantages over traditional middleware and discusses the use of Kafka Streams, ksqlDB, and Kafka Connect for building event-driven applications within DDD contexts. The concept of an anti-corruption layer is explained in the context of mapping data between bounded contexts using event schemas and the Schema Registry. The post also touches upon the operational aspects of managing Kafka clusters, including self-service offerings on PaaS infrastructure like Kubernetes and managed services like Confluent Cloud.
  113. Optimizing Kafka Streams Applications | ConfluentThis post introduces the processor topology optimization framework in Kafka Streams (since version 2.1.0). It explains how processor topologies are generated from the Processor API and Streams DSL, illustrates the use of Topology#describe() and external visualization tools, and details the concept of data parallelism and repartition topics for stateful operations. It highlights how the framework allows for optimizations by analyzing the topology graph and introduces techniques like sticky partitioning for improved efficiency.
  114. Testing Event-Driven Systems | ConfluentIntroduces the 'Test Machine' library for end-to-end testing of event-driven systems, addressing the limitations of unit tests and existing Kafka Streams testing tools like TopologyTestDriver and Fluent Kafka Streams Tests. Details a testing pattern involving writing input events, waiting for processing, and asserting output. Explains the 'Test Machine' architecture with 'write' and 'watch' commands, portable test execution against various targets (local Kafka, REST Proxy, mock topology), and performance considerations. Provides a Clojure code example using the Test Machine for a word count demo.
  115. Putting Events in Their Place with Dynamic Routing | ConfluentIntroduces KStream#branch() for static stream splitting based on predicates, creating new Kafka topics for different event subsets. Details dynamic routing using TopicNameExtractor (KIP-303) for runtime determination of output topic names, enabling routing based on event fields or enriched data from external sources like databases via JDBC Source Connector and GlobalKTable joins.
  116. Kafka Streams' Take on Watermarks and Triggers | ConfluentIntroduces the `Suppress` operator for Kafka Streams, allowing for controlled emission of KTable updates based on window lifecycle and grace periods. This addresses use cases requiring non-retractable outputs like alerts or side effects, complementing the existing continuous refinement model. The post details the `untilWindowClose` suppression strategy and the concept of window grace periods for handling late-arriving records.
  117. Stream Processing Tutorial Part 2: Testing Your Streaming ApplicationIntroduces unit testing for Kafka Streams applications using TopologyTestDriver and MockProcessorContext, integration testing with EmbeddedKafkaCluster, and schema compatibility testing with MockSchemaRegistryClient and the Schema Registry Maven plugin. It also mentions testing with Confluent Cloud tools and multi-datacenter scenarios.
  118. Kafka Streams and KSQL with Minimum Privileges | ConfluentIntroduces the concept of prefixed ACLs in Kafka (KIP-290) enabled by Confluent Platform 5.0+ and Apache Kafka 2.0+. Details two patterns for configuring these ACLs: 1) Team-level prefixes (e.g., 'team.fraud.') allowing read/write/create on all topics and groups starting with that prefix, and 2) Application-level prefixes (e.g., 'mobile.payments.fraud') for more granular control. Explains how to apply these patterns to both Kafka Streams (using `application.id`) and KSQL (using `ksql.service.id`), including the specific prefixes for internal KSQL topics (e.g., '_confluent-ksql-team.fraud').
  119. Real-Time Presence Detection at Scale with Apache Kafka on AWS | ConfluentThis post details Zenreach's migration from Python scripts and Spark Streaming to Kafka Streams for their real-time presence detection platform. It outlines their use case of detecting walk-ins, enriching them with contact information, and maintaining walk-in counts. The post highlights the challenges faced with a monolithic Spark application (deployment complexity, monitoring, cost) and the benefits of adopting Kafka Streams, including horizontal scalability via stream threads and tasks, persistent state management using RocksDB and EBS volumes, and improved fault tolerance with standby replicas. It also touches upon deployment on AWS ECS and integration with Prometheus and Rollbar.
  120. Securing KSQL | ConfluentThis post introduces the security aspects of KSQL, detailing how to configure it with SSL for encryption and SASL for authentication when communicating with Kafka clusters. It also covers HTTPS for Schema Registry integration. The post emphasizes the comprehensive security design of the Confluent Platform, including its various components, and provides links to demos and tutorials for practical security implementation.
  121. Running Kafka Streams Applications in AWS | ConfluentThis post details Zalando's experience running Kafka Streams applications in AWS, highlighting challenges and solutions related to EBS I/O credits, application fine-tuning (timeouts, batch sizes), and memory management (JVM heap vs. off-heap RocksDB usage). It also introduces custom monitoring tools like Remora for consumer lag and emphasizes the importance of comprehensive system monitoring (CPU, memory, disk) for identifying and resolving performance bottlenecks.
  122. Using Kafka Streams API for predictive budgetingThis post details how Pinterest built a predictive budgeting system using the Kafka Streams API to prevent ad overdelivery. It explains the problem of overdelivery due to real-time spend data latency and the need for predictive spend. The system requirements included handling tens of thousands of events per second, fanning out updates to thousands of consumers, and achieving end-to-end delay under 10 seconds with 100% uptime. Kafka Streams was chosen for its millisecond delay guarantee and lightweight nature. The architecture involves an 'inflight spend' service that aggregates ad insertion events using Kafka Streams with a 10-second window store per adgroup. Key learnings include the performance impact of window store design (18x improvement by switching from hopping to tumbling windows) and message compression strategies (delta encoding and lookuptable encoding reducing message size by 4x).
  123. Stories from the Front: Lessons Learned from Supporting Apache Kafka ® | ConfluentThis post details three common operational pitfalls encountered in production Apache Kafka deployments: under-replicated partitions, issues with automated liveness checks leading to cluster downtime, and performance degradation when adding new brokers. It provides high-level recommendations for each scenario, emphasizing root cause analysis for under-replicated partitions, the importance of human intervention for automated restarts in stateful systems, and understanding the impact of partition reassignments before execution. It also highlights the benefits of Confluent's auto data balancer for safer data movement.
  124. Securing the Confluent Schema Registry for Apache Kafka | ConfluentThis post details the configuration steps for securing the Confluent Schema Registry to work with a secured Kafka cluster. It covers configuring SASL for authentication with Kafka and ZooKeeper, setting up Kafka ACLs for the Schema Registry to access the `_schemas` topic, and enabling HTTPS for the Schema Registry's REST API. It provides specific configuration examples for JAAS files, Kafka properties, and ACL commands, along with an analogy to explain the security concepts.
  125. Log Compaction | Highlights in the Apache Kafka Community - Feb 2017This post details several Kafka Improvement Proposals (KIPs) that are under active discussion or have been merged, focusing on enhancing Kafka's robustness, manageability, and features. Key KIPs discussed include: KIP-112 and KIP-113 for improved disk failure handling and replica movement in JBOD configurations; KIP-117 for a public AdminClient API for Kafka administrative operations; KIP-98 for exactly-once delivery and transactional messaging; KIP-118 for dropping Java 7 support in Kafka 0.11; KIP-110 for adding ZStandard compression codec; and KIP-109 for deprecating old consumers. It also highlights a performance improvement for 99% latency without user-visible changes (KAFKA-4614) and links to notable blogs and presentations on microservices, event-driven architectures, and compression codec benchmarks.
  126. Confluent Delivers Upgrades to Clients, Kafka Streams and BrokersThis post announces Confluent 3.1.2, a patch release incorporating Apache Kafka 0.10.1.1. It details bug fixes and general improvements across Kafka brokers, the Kafka Streams API, and Kafka clients. Notable broker fixes include issues with the log cleaner, ISR thrashing under replication quotas, tombstone removal, offset retention calculation, and corrupted message handling. Kafka Streams improvements address user config respect, topic not found errors during broker restarts, slow stream resetters, and cache eviction issues. Client fixes include deadlocks between consumer heartbeats and offset commits, consumer throughput regressions, and consumer failures after offset topic partition reassignment. It also mentions Scala 2.12 support.
  127. Log Compaction | Highlights in the Apache Kafka and Stream Processing Community | December 2016 | ConfluentThis post details several KIPs impacting Kafka semantics and operability: KIP-95 for incremental batch processing in Kafka Streams, KIP-97 for improved Kafka client RPC compatibility allowing newer clients to connect to older brokers, and the proposed KIP-98 for exactly once delivery and transactional messaging. It also highlights community events and other streaming system developments.
  128. Announcing Confluent 3.1 with Apache Kafka 0.10.1.0 | ConfluentThis post announces Confluent Platform 3.1, which includes Apache Kafka 0.10.1.0. Key new features in Confluent Platform are Automatic Data Balancing to address unbalanced workloads across brokers due to varying topic/partition growth and broker changes, and Multi-datacenter Replication for scenarios like active-active geo-localized deployments, centralized analytics, and cloud migration. Confluent Control Center is enhanced with stream latency and completeness alerting, enterprise authentication integration, and multi-cluster topic monitoring. New open-source components include an Elasticsearch Sink connector with automatic mapping generation and exactly-once delivery, a JDBC Sink connector for exporting data to relational databases, and a new Go Client.
  129. Unifying Stream Processing and Interactive Queries in Apache KafkaIntroduces Interactive Queries for Kafka Streams, allowing direct querying of application state without external databases. This feature leverages embedded databases (like RocksDB) within Kafka Streams applications to act as materialized views of Kafka topics, enabling hyper-convergence of processing and storage. The post contrasts this with traditional Lambda and Kappa architectures, highlighting benefits like reduced complexity, faster state access, and better application isolation. It also discusses tradeoffs when choosing between interactive queries and external databases.
  130. Log Compaction | Highlights in the Apache Kafka and Stream Processing Community | October 2016 | ConfluentThis post details several Kafka Improvement Proposals (KIPs) relevant to broker and client behavior. KIP-74 introduces a fetch response size limit to control consumer memory usage and allow progress even with large messages. KIP-79 adds timestamp search methods to the new consumer. KIP-82 proposes adding headers to Kafka messages, a significant change to message structure. KIP-83 allows multiple SASL-authenticated Java clients in a single JVM, aiding MirrorMaker. KIP-85 enables dynamic JAAS configuration for clients, beneficial for containerized microservices.
  131. Log Compaction | Highlights in the Kafka and Stream Processing Community | March 2016 | ConfluentThis post details several key developments in the Apache Kafka community and Confluent Platform. It announces the release of Apache Kafka 0.9.0.1 and Confluent Platform 2.0.1, highlighting bug fixes and improvements. The next release is designated as 0.10.0, incorporating a protocol update, new file format, and Kafka Streams. Several KIPs were merged: KIP-41 (KafkaConsumer Max Records), KIP-32 (Add timestamps to Kafka message), and KIP-42 (Add Producer and Consumer Interceptors). New KIPs under discussion include KIP-47 (timestamp-based log deletion) and KIP-48 (delegation token support). The Kafka Connector Hub is launched, and Microsoft open-sourced their .NET Kafka client. Community meetups and discussions on stream processing futures are mentioned, along with the early release of "Kafka: The Definitive Guide" and the upcoming Kafka Summit.
  132. Announcing Apache Kafka 0.9.0.1 and Confluent Platform 2.0.1 | ConfluentThis post announces Apache Kafka 0.9.0.1 and Confluent Platform 2.0.1, which includes 68 issue fixes. Notable bug fixes are detailed for the new Java consumer (rebalancing, offset issues), compatibility, security (ticket renewal, ACLs, SASL packet handling), performance (fetch wait, CPU usage, log cleaner, sensor leaks), and topic deletion. It recommends upgrading to this patch release for improved stability and correctness.
  133. 290 Reasons to Upgrade to Apache Kafka 0.9.0.0 | ConfluentThis post details numerous bug fixes and improvements in Apache Kafka 0.9.0.0, with a strong emphasis on correctness, reliability, and operability over new features. Specific technical details are provided for fixes related to replication (race conditions, min.insync.replica, replication lag), MirrorMaker message loss prevention, log cleanup, log compaction (especially for compressed messages), connection leak detection, broker metadata cache synchronization, request purgatory rewrite, and producer timeout enforcement. It also highlights the addition of a new distributed testing framework with over 100 new test scenarios.
  134. Hands-Free Kafka Replication: A Lesson in Operational Simplicity | ConfluentIntroduces a change to Kafka's replica lag detection mechanism, replacing `replica.lag.max.messages` with a time-based approach (`replica.lag.time.max.ms`). This aims to simplify configuration, reduce operational overhead, and prevent false alerts caused by temporary spikes in producer traffic or GC pauses on follower replicas. The new model ensures that replicas are only removed from the ISR if they consistently lag behind the leader for a defined period, improving the stability of replication.