Confluent Feature Trails

Feature trailActive

AI Developer Tools

Confluent Cloud enhances AI-ready streaming capabilities across data, pipelines, and operations. This includes a dbt adapter for Apache Flink and Materialized Tables, a managed Model Context Protocol (MCP) server and Confluent Agent Skills for AI coding agents, and the General Availability (GA) of Streaming Agents and Real-time Context Engine for event-driven AI. The VS Code extension now integrates with GitHub Copilot chat participant APIs to provide engineers with safe, fast access to data str

9 posts · 2023–2026

Timeline

20232026

Feature trailActive

Real-time Enterprise RAG

Confluent Cloud enhances real-time, personalized customer experiences by integrating in-store and online data for timely offers and product information. The architecture emphasizes high availability, scalability, and low latency using Kafka, ksqlDB, and notification platforms. This iteration focuses on powering AI agents with real-time context from across the business, enabling them to make accurate, situationally aware decisions and take actions. This includes Streaming Agents (Phase 2 - Open P, and AI Model Inference in Flink for real-time recommendations and personalized offers.

67 posts · 2018–2026

Timeline

20182026

Feature trailActive

Flink AI Model Inference

Confluent Cloud for Apache Flink®️ now supports AI model inference, allowing models to be used as resources in Flink SQL. This enables real-time data enrichment by invoking generative AI (GenAI) large language model (LLM) endpoints (e.g., OpenAI, Azure OpenAI, Amazon Bedrock, Google Vertex AI) and custom ML/deep learning models directly from Flink SQL statements for tasks like classification, text generation, and clustering. This integration allows for real-time ML predictions to be applied to streaming data, enhancing applications with context and enabling smarter decision-making. The feature is currently in early access.

22 posts · 2016–2026

Timeline

…20172026

Feature trailActive

Hybrid and Multicloud Architecture Strategy

Confluent Cloud is now accessible through major cloud provider marketplaces (AWS, Azure, Google Cloud), enabling unified billing, utilization of pre-committed cloud spend, and a consistent experience across different cloud environments. This simplifies procurement and adoption for enterprises and emphasizes the portability offered by Confluent Cloud's presence across multiple clouds. Confluent Platform has also been validated on AWS. This post further elaborates on designing for continuous availability and adaptability in hybrid and multicloud environments by emphasizing portable interfaces, clear service boundaries, and separation of concerns, moving beyond simple disaster recovery to true resilience.

18 posts · 2016–2026

Timeline

…20172026

Feature trailActive

Kafka Infrastructure Cost Optimization

Confluent Cloud continues to evolve its cost optimization features for Kafka, focusing on reducing data transfer costs, providing better cost control mechanisms, and enabling more flexible scaling. This includes advancements in follower fetching, Private Network Interface (PNI) networking, enhanced metrics for understanding eCKU scaling drivers, client quotas for Enterprise and Freight clusters, and new 'Queues for Kafka' functionality to consolidate messaging infrastructure. The goal is to enable more efficient and cost-effective Kafka operations through serverless autoscaling (eCKUs) and optimized resource utilization.

2 posts · 2017–2026

Timeline

20172026

Feature trailActive

Kafka KRaft Migration

Apache Kafka 3.8.0 continues the transition to KRaft mode, with KIP-848 (Next Generation Consumer Rebalance Protocol) now in preview. Tiered Storage supports clusters with multiple log directories (JBOD), and JBOD in KRaft is no longer early access. Compression level can now be specified (KIP-390). The Log4J Appender is deprecated. Docker images for GraalVM native Kafka broker and official Docker image are introduced. KIP-1036 enhances RecordDeserializationException for better DLQ/logging. KIP-1037 allows WriteTxnMarkers API with Alter Cluster Permission. `offsets.commit.required.acks` is deprecated. Kafka Streams gains Shareable State Stores (KIP-813), customizable task assignment (KIP-924), and improved StateStore Iterator metrics (KIP-989). Kafka Connect enforces `tasks.max` (KIP-1004).

6 posts · 2015–2026

Timeline

…20172026

Feature trailActive

Data Serialization Formats

Confluent Cloud's Tableflow continuously converts Kafka topics into open table formats (Apache Iceberg or Delta Lake) for analytics engines, eliminating custom ETL pipelines. It handles schematization, type conversions, schema evolution, CDC stream materialization, catalog publishing, and table maintenance, using Schema Registry for governance and Kora storage for conversion to Parquet files. Supported analytics engines include Snowflake, Databricks, AWS Athena, Amazon Redshift, Trino, and BigQuery. This integration is now enhanced to provide bidirectional data flow with Databricks Unity Catalog, enabling real-time operational data to power AI and analytics, and AI-driven insights to flow back into real-time applications, all under unified governance.

10 posts · 2015–2026

Timeline

…20172026

Feature trailActive

Apache Kafka Security Features

Confluent Cloud continues to evolve its security features for Apache Kafka and Confluent Platform, focusing on robust audit logging capabilities. This iteration details the architecture for building a real-time compliance and audit logging pipeline using Kafka, Kafka Connect, stream processing (Kafka Streams/Flink), and tiered storage. Key aspects include ensuring data accuracy and immutability via Kafka's append-only log, guaranteed long-term retention with tiered storage, ironclad security with mTLS and RBAC, and continuous transparency through distributed tracing and structured audit logs. The focus is on enabling event-native governance by embedding policies, controls, and visibility directly into the data stream, ensuring secure, compliant, and reliable streaming systems by design.

4 posts · 2016–2026

Timeline

…20172026

Feature trailActive

Kafka Broker and Streams Enhancements

This feature thread tracks the evolution of Kafka Streams and ksqlDB, focusing on enhancing availability, performance, and query capabilities. It covers improvements in serving data from stale stores during rebalances (KIP-535), fetching data from specific partitions (KIP-562), and fixing unnecessary standby restoration bugs (KAFKA-9169). The goal is to enable Kafka Streams to challenge traditional databases by providing high availability and low-latency serving, reducing developer overhead and enabling effective scaling for high-volume data processing through strategies like partitioning, scaling out/up, and fine-tuning configuration parameters.

126 posts · 2015–2026

Timeline

…20172026

Feature trailActive

Stream Processing Fundamentals

ksqlDB continues to evolve as a powerful stream processing engine, enabling declarative data manipulation and real-time data replication. This release introduces support for Avro data format through integration with Confluent Schema Registry, automatic schema inference, and the ability to convert between Avro, JSON, and delimited formats in real-time. It also enhances joins across different data formats and lays the groundwork for observability features with new DESCRIBE EXTENDED and EXPLAIN sta

84 posts · 2016–2026

Timeline

…20172026

Feature trailActive

Confluent Private Cloud Broker Density and Policy Enforcement

Confluent Private Cloud (CPC) introduces a new deployment model for platform teams, bringing cloud-native operational efficiencies to private infrastructure. It enhances Kafka performance with Intelligent Replication, centralizes governance and policy enforcement with CPC Gateway, and unifies management and monitoring across hybrid and multi-cloud environments with Unified Stream Manager (USM). CPC aims to reduce infrastructure and operating costs by up to 50% by automating lifecycle management, improving client management, and enabling Kafka-as-a-service capabilities.

7 posts · 2020–2026

Timeline

20202026

Feature trailActive

IoT Fleet Management Data Streaming

Confluent Cloud continues to evolve its capabilities for integrating edge devices with data streaming platforms for real-time data ingestion and processing. This includes architectures for connecting IoT devices to cloud platforms like AWS IoT Core and Confluent Cloud, enabling contextualization of raw device data by combining it with business system information. The goal is to enrich data streams for improved decision-making, operational efficiency, and enhanced customer experiences. This post details the use of Confluent Cloud and Apache Flink for creating a Single Patient View (SPV) in healthcare, enabling real-time access to patient data across disparate systems for improved clinical outcomes and powering a real-time experimental treatment-matching service for cancer patients. The solution addresses challenges in interoperability, regulatory compliance (HIPAA), scalability, observability, and disaster recovery by leveraging a HIPAA-ready, fully managed, serverless data streaming platform.

19 posts · 2018–2026

Timeline

20182026

Feature trailActive

Data Privacy and De-identification for Kafka

Confluent Cloud continues to evolve data privacy and de-identification techniques for real-time data streams within Apache Kafka. This iteration focuses on integrating third-party platforms like Privitar to apply privacy policies and transform sensitive data in transit. The focus is on enabling compliance with regulations such as GDPR, CCPA, and HIPAA without sacrificing analytical utility. This includes demonstrating de-identification scenarios, configurable privacy rules, and the optional re-identification of data. The architectural approach to digital sovereignty is further detailed, emphasizing architectural guarantees over policy assurances, the schema as a new sovereignty boundary with integrated sensitivity and jurisdiction tagging, and the importance of open protocols for portability. This post specifically details the implementation of client-side field-level encryption (CSFLE) and client-side payload encryption (CSPE) as architectural controls for data sovereignty, and the use of schema registry for enforcing data contracts and sensitivity tags.

2 posts · 2017–2026

Timeline

20172026

Feature trailActive

Event-Driven Architectures

Confluent Cloud continues to evolve event-driven architectures by enabling real-time alerting and automated response systems. This iteration focuses on designing and implementing intelligent alerts using Apache Kafka, moving beyond simple notifications to create systems that keep business operations resilient and ahead of critical events. This includes threshold-based, anomaly detection, composite event alerts, and automated action triggers, emphasizing the use of stream processing for real-time data.

32 posts · 2014–2026

Timeline

…20172026

Feature trailActive

Observability Integrations

Confluent Cloud Console enhances observability for streaming workloads with new metrics and a refreshed cluster monitoring experience, including client limit breaches, rebalance duration, connection attempts, and partition counts. It also introduces a dedicated Kafka Streams view for consolidated monitoring and debugging of Kafka Streams applications. The console now offers improved client overview with details on client IDs, software versions, and request counts, aiding in the identification of performance bottlenecks. The next generation of Control Center for Confluent Platform integrates with Prometheus, supporting up to 400K partitions, accelerating start-up time to 1 minute, achieving metrics freshness within 2-3 minutes, and eliminating the need for a separate Kafka cluster for metrics storage.

11 posts · 2016–2026

Timeline

…20172026

Feature trailActive

Kafka Client Migration Automation

Confluent Cloud Gateway now fully orchestrates Kafka client migrations with KCP, enabling seamless, safe, and rollback-friendly cutovers. This includes transparent authentication translation (e.g., SASL/SCRAM to API keys), group-based migration for incremental rollouts, and offset preservation via Cluster Linking, minimizing downtime and coordination overhead. The process is streamlined into three KCP commands: init, migrate, and promote. This post demonstrates a POC for disaster recovery failover orchestration using Confluent Cloud Gateway to achieve seamless client failover within 60 seconds, combining Cluster Linking for server-side replication with the gateway for client-side redirection and managing the replication reversal.

1 post · 2026

Timeline

Feature trailActive

Schema Registry Best Practices

Confluent Schema Registry now supports storing schema IDs in Kafka headers, enabling backward-compatible adoption of schema governance without breaking legacy consumers or changing payload formats. This facilitates incremental migration from schemaless data to a governed data plane, improving data quality, powering contextualized AI/analytics, and simplifying lakehouse integration. The new format uses 16-byte GUIDs for schema identification, offering broader compatibility across environments. This post details how data contracts, CEL rules, DLQs, and client-side field-level encryption (CSFLE) can be used to protect PII in Kafka.

17 posts · 2015–2026

Timeline

…20172026

Feature trailActive

Streaming Machine Learning

This feature thread tracks the evolution of integrating machine learning with real-time data streams. It covers transactional machine learning (TML) which combines automated machine learning (AutoML) with data streams using Apache Kafka as the backbone. The focus is on creating frictionless ML processes, building transactional training datasets from streams, applying ML algorithms, and using them for predictive analytics and optimization. The architecture leverages tools like MAADS-VIPER and MAA. This post details how to build, operate, and monitor analytic models in a scalable, mission-critical way by leveraging Apache Kafka® as a central, scalable, mission-critical nervous system plus Apache Kafka's Streams API to build intelligent streaming applications. It discusses potential use cases for machine learning in mission-critical real time applications and outlines a reference architecture for building, operating and monitoring analytic models with Kafka, separating model training from model inference.

11 posts · 2017–2026

Timeline

20172026

Feature trailActive

Kafka Partition Scalability

Skai successfully migrated their large-scale Kafka deployment to Confluent Cloud by re-architecting their topic strategy to drastically reduce the number of topics. They introduced "united-topic" strategies (single-pipeline, multiple-pipelines, heavy-load) to consolidate events from multiple tables into fewer topics, significantly improving manageability, reducing broker count, and lowering operational costs. Enhanced monitoring and lag handling mechanisms were implemented to address challenges with mixed event streams. The solution also addressed external Kafka client integration through a caching library for dynamic topic mapping. This migration resulted in a >90% reduction in topics without performance degradation, paving the way for further cloud adoption.

8 posts · 2015–2026

Timeline

…20172026

Feature trailActive

Kafka Producer and Consumer API Enhancements

This feature thread tracks the evolution of Kafka producer and consumer APIs, focusing on reliability, performance, and advanced features like Exactly Once Semantics (EOS). It covers improvements in producer idempotency, transactional guarantees, and the integration of new connectors to enhance data pipeline capabilities. The thread also highlights advancements in consumer group management and offset handling for robust data processing. This post details the internal workings of Kafka producers, including serialization, partitioning, and batching strategies, and how these prepare event data for brokers. Future posts will cover broker handling of producer requests, consumer fetch preparation, and broker-side fetch handling.

5 posts · 2014–2026

Timeline

…20172026

Feature trailActive

Kafka Producer Partitioning Strategy

This feature thread tracks the evolution of Kafka producer partitioning strategies, focusing on improving latency and efficiency. It covers the introduction and impact of sticky partitioning for null-keyed records, which aims to create larger batches and reduce overhead. The thread also includes performance benchmarks and analysis of CPU utilization to quantify the benefits of these improvements. This post details how a Tier-1 bank achieved sub-5ms p99 latency for trading pipelines by meticulously tuning Kafka brokers, producers, and consumers, leveraging OpenMessaging Benchmark for reproducible results, and focusing on a 'lowest baseline first' methodology with a single partition to isolate latency contributors. Key tuning areas included broker configurations (network/IO threads, replica fetchers, segment size, heap), producer configurations (linger.ms, batch.size, compression, idempotence, acks), and consumer configurations (max.poll.records, max.poll.interval.ms). Infrastructure upgrades and JVM tuning (Generational ZGC) were also critical.

1 post · 2019–2026

Timeline

20192026

Feature trailActive

Data Streaming Cost and ROI Analysis

This feature thread tracks the evolution of understanding and optimizing the total cost of ownership (TCO) and return on investment (ROI) for data streaming platforms. It covers the breakdown of infrastructure, operations, engineering, governance, and opportunity costs associated with streaming systems, comparing different deployment models (self-managed, hosted, cloud-managed) and processing paradigms (batch vs. streaming vs. micro-batch). The goal is to provide guidance on making informed architectural decisions and demonstrating the business value of data streaming.

2 posts · 2023–2025

Timeline

20232026

Feature trailActive

Confluent Cloud Availability and Resilience

Confluent Cloud continues to evolve its Kora Engine, a cloud-native event streaming platform designed for Kafka. This iteration focuses on enhancing reliability, elasticity, and cost-efficiency through a cellular architecture for multi-tenancy, serverless abstractions, self-balancing clusters with tiered storage, automatic mitigation for high availability, and robust data durability leveraging object storage. The platform aims to provide consistent, predictable performance across diverse cloud environments, offering a fundamentally more managed and cost-effective alternative to hosted Kafka solutions by decoupling compute and storage, enabling elastic scaling with ECKUs and diskless clusters, and optimizing networking with Private Networking Interfaces (PNI).

21 posts · 2017–2025

Timeline

20172026

Feature trailActive

Confluent Connectors Portfolio

Confluent Cloud continues to evolve its Kafka Connectors portfolio, focusing on simplifying the migration of self-managed connectors to fully managed Confluent Cloud connectors. This iteration introduces new integrations with Amazon SageMaker, AWS Glue, ClickHouse, Couchbase, Datadog, DiffusionData, Firebolt, Infoview, LittleHorse, MariaDB, Onibex, SingleStore, Weaviate, and YugabyteDB. It also enhances existing integrations with Amazon EventBridge and AWS Lambda, and introduces Tableflow for exposing Kafka topics as open table formats (Iceberg/Delta Lake) for analytics engines. WarpStream also gains a Datadog integration for observability.

99 posts · 2016–2025

Timeline

…20172026

Feature trailActive

Confluent for Kubernetes

Confluent for Kubernetes (CFK) simplifies running Kafka as a cloud-native data system on Kubernetes, enabling programmatic provisioning of Kafka resources with compute, storage, networking, and security options. It provides a consistent operational experience across on-prem and cloud deployments. This thread also tracks the evolution of tiered storage for decoupling Kafka's storage and compute layers, and self-balancing clusters for automated partition rebalancing and optimized resource utilization.

12 posts · 2015–2025

Timeline

…20172026

Feature trailActive

Multi-Kafka Cluster Management with Cluster Registry

Unified Stream Manager (USM) is now generally available with Confluent Platform 8.1, enabling a unified view for governance and monitoring of all Confluent clusters, whether on-prem, in a private cloud, or across public clouds. USM connects on-prem Confluent Platform environments to Confluent Cloud through a secure private network connection (initially AWS PrivateLink), sharing only metadata and telemetry. This provides a single source of truth for schemas, enforcing data contracts and enabling client-side field-level encryption (CSFLE) across hybrid infrastructure. It also consolidates observability, allowing users to monitor the health and performance of all clusters from the Confluent Cloud UI, trace data flow end-to-end, and discover data in a unified catalog. The architecture keeps message data within the private environment, sending only operational metadata to Confluent Cloud.

3 posts · 2019–2025

Timeline

20192026

Feature trailActive

Shared Services Security and Resource Management

Confluent Cloud enhances its capabilities for building secure shared services platforms. This includes Cluster Role-Based Access Control (RBAC) for granular management access to resources like production environments and sensitive clusters, complementing existing ACLs for application access to Kafka resources. Audit Logs are now generally available, enabled by default, to track user/application resource access for anomaly detection and bad actor identification, with logs available in topics. Cloud

4 posts · 2019–2025

Timeline

20192026

Feature trailActive

Stream Lineage

Stream Lineage provides an interactive map of data flows in event-driven architectures, enabling users to understand data origins, transformations, and destinations. It visualizes producers, consumers, connectors, Kafka Streams applications, and ksqlDB streams/tables. The tool supports time-windowed views (up to 24 hours) and search functionality for specific clients, topics, connectors, ksqlDB queries, and consumer group IDs. Use cases include development of data pipelines, troubleshooting of d

4 posts · 2021–2025

Timeline

20212026

Feature trailActive

Multi-Geo Replication

Confluent Platform continues to evolve its multi-datacenter replication capabilities, now enabling secure, private cross-cloud data replication over Confluent's backbone. This enhances existing Cluster Linking functionality by supporting private networking between AWS, Azure, and Google Cloud, facilitating use cases like data sharing, disaster recovery, and aggregated analytics without traversing the public internet. The platform offers native enterprise-grade protections including IP filtering, and now supports Azure Private Link for secure, private data replication between Kafka clusters within private Azure networks.

12 posts · 2016–2025

Timeline

…20172026

Feature trailCooling

Secure Financial Data Exchange

This feature thread tracks the evolution of real-time fraud detection and prevention systems in financial services, leveraging data streaming and machine learning. It details how organizations like EVO Banco integrate Confluent Cloud for real-time transaction processing with Apache Spark for batch processing and ML model training. The system analyzes transaction data, customer information, and account details streamed through Confluent Cloud to train ML models and trigger real-time alerts. This iteration focuses on the patterns enabled by real-time streaming, including anomaly detection, transaction scoring pipelines, identifying composite events, and automating real-time fraud responses. It also emphasizes the importance of governance, data accuracy, and data lineage for meeting regulatory requirements and building trust in the data.

3 posts · 2022–2025

Timeline

20222026

Feature trailCooling

HTTP and REST API Integration with Kafka

Confluent Cloud continues to evolve its integration capabilities for HTTP and REST APIs with Kafka, focusing on enhancing the HTTP Source V2 and HTTP Sink V2 connectors. These connectors now support configuring multiple APIs with the same hostname and authorization, seamless integration with OpenAPI Specification 3.0, API testing during configuration, offset management, and client-side field-level encryption. The HTTP Sink V2 also allows specifying Kafka record fields in the URL, parameters, and headers. This post details the creation of a node.js wrapper for the Kafka REST Proxy, demonstrating how to build applications by exposing Kafka functionality through HTTP. It highlights improvements in the produce API for handling various message formats and serialization, and a more native Node.js-like consumer API using event emitters. An example application is provided that streams Twitter data into Kafka and processes trending topics.

10 posts · 2015–2025

Timeline

…20172026

Feature trailCooling

Incremental Cooperative Rebalancing

This feature thread tracks the evolution of Kafka's consumer rebalance protocol, moving from the classic eager and cooperative strategies to a new, server-driven protocol (KIP-848) in Kafka 4.0+. This new protocol aims to eliminate consumer downtime, reduce rebalance times, improve stability and scalability, and simplify management by shifting coordination logic to the broker-side group coordinator. It supports live upgrades and is designed to become the default in future Kafka releases. Future enhancements include client-side performance improvements and full topic ID support for offsets. Work is also underway to bring these benefits to Kafka Streams via KIP-1071.

2 posts · 2019–2025

Timeline

20192026

Feature trailCooling

Kafka Consumer and Group Management

This feature thread tracks the evolution of Kafka consumer and group management, focusing on offset management, rebalancing, and ensuring data processing reliability and scalability. It covers fundamental principles, advanced insights, and new enhancements like KIP-345 for static membership to reduce consumer rebalances, particularly for heavy-state applications, and KIP-1094 for more accurate offset and leader epoch handling, as well as strategies for manual and automatic offset commits, and the introduction of server-driven rebalancing in Kafka 4.0+.

2 posts · 2016–2025

Timeline

…20172026

Feature trailCooling

Kafka mTLS Authentication System

Confluent Cloud enhances security and private connectivity for its data streaming services. This includes the general availability of Mutual TLS (mTLS) authentication for Dedicated clusters, providing two-way client and broker authentication to protect sensitive data and reduce risk. Additionally, Private Link is now available for Schema Registry and Apache Flink, enabling secure, private access from within customer virtual private clouds (VPCs) without traversing the public internet. This simplifies secure data ingestion into analytical platforms like Amazon Redshift.

1 post · 2021–2025

Timeline

20212026

Feature trailCooling

Schema Registry and Flink Private Connectivity

Confluent Cloud enhances security and private connectivity for its data streaming services. This includes the general availability of Mutual TLS (mTLS) authentication for dedicated clusters, providing two-way client and broker authentication to protect sensitive data and reduce risk. Additionally, Private Link is now available for Schema Registry and Apache Flink, enabling secure, private access from within customer virtual private clouds (VPCs) without traversing the public internet. This simplifies network architecture, reduces exposure to threats, and improves performance for critical components like Schema Registry and Flink state management.

2 posts · 2025

Timeline

20252026

Feature trailCooling

Edge Data Ingestion and Processing

3 posts · 2019–2024

Timeline

20192026

Feature trailQuiet since 2024

IAM Role Integration for Managed Connectors

Confluent Cloud managed connectors now integrate with AWS IAM roles, enabling the use of temporary security credentials. This reduces the risk of long-term credential exposure and simplifies key management by eliminating the need for static access keys. The integration allows for secure, role-based access between AWS resources and Confluent Cloud for data ingestion and transfer, with setup managed via the Confluent Cloud API and AWS IAM console.

1 post · 2024

Timeline

20242026

Feature trailQuiet since 2024

Real-time Insurance Quoting with Data Streaming

This feature thread tracks the evolution of real-time order notifications and logistics management. It details how companies have migrated from siloed, batch-processed data systems to modern microservices architectures powered by Confluent's event streaming platform. Key technical aspects include integrating disparate databases (Postgres, MySQL, SQL Server) using CDC connectors, leveraging ksqlDB for real-time data processing and microservice development, and utilizing Stream Governance for data.

6 posts · 2018–2024

Timeline

20182026

Feature trailQuiet since 2024

Data Mesh Architecture and Practices

Confluent Cloud continues to evolve data mesh architectures and their implementation using streaming data platforms. This iteration focuses on leveraging generative AI to automate the creation of metadata for data governance, thereby improving data quality and enabling the development of more accurate AI-powered applications. This complements existing efforts in decentralizing data ownership, federated governance, and delivering data products as APIs, with a focus on cloud-native principles, scalability, and real-time data processing.

4 posts · 2019–2024

Timeline

20192026

Feature trailQuiet since 2024

JavaScript Kafka Client

Confluent Cloud offers an officially supported JavaScript client for Apache Kafka (CJSK), based on librdkafka. This client provides APIs similar to KafkaJS and node-rdkafka, supports TypeScript, and integrates with Confluent Schema Registry and client-side field-level encryption (CSFLE). It offers both callback and promisified APIs for producer and consumer workflows. This post details building an application using the client to settle a debate, demonstrating both producer and consumer functionality.

1 post · 2021–2024

Timeline

20212026

Feature trailQuiet since 2024

OpenSearch Ingestion Integration

Confluent Cloud continues to evolve its integration capabilities for OpenSearch, focusing on simplifying data ingestion and analytics. This iteration introduces support for Amazon OpenSearch Ingestion as a source, enabling seamless, private connectivity between Confluent Cloud and OpenSearch domains/collections. This complements the existing managed OpenSearch Sink connector, providing a comprehensive solution for streaming data to OpenSearch.

4 posts · 2018–2024

Timeline

20182026

Feature trailQuiet since 2024

Real-time Field Service Management Optimization

8 posts · 2020–2024

Timeline

20202026

Feature trailQuiet since 2024

Kafka Infinite Storage

Confluent Platform introduces Tiered Storage, enabling operators to offload the majority of Kafka data to remote object stores like Amazon S3. This separation of storage and processing enhances scalability, elasticity, and cost-efficiency by reducing the burden of rebalancing, cluster expansion/shrinking, and broker replacement. Tiered Storage allows for infinite data retention, enabling Kafka to serve as a system of record for use cases like materialized views, ML model training, and regulatory compliance. This post details the initial considerations for deploying Kafka on AWS EBS, highlighting the importance of storage performance and reliability for Kafka clusters. It specifically discusses the benefits of EBS volumes for sequential I/O workloads and the cost-effectiveness of st1 and sc1 offerings.

24 posts · 2015–2024

Timeline

…20172026

Feature trailQuiet since 2023

Policy-as-Code for Terraform

This feature thread tracks the evolution of integrating policy-as-code with Confluent's Terraform provider, enabling self-service management of Confluent resources with automated enforcement of security, compliance, and auditing requirements. This release introduces the Confluent Terraform provider, allowing infrastructure to be managed as code through GitOps integration. It also enhances network management by treating Confluent Cloud networks as first-class resources, provisionable via REST APIs and Terraform, with granular access control for network administrators. The connector portfolio has expanded to 70 fully managed connectors, including new source connectors for GCS, S3, and Azure Cosmos DB. Audit logs now include user login monitoring to identify potential breaches. Cluster Linking is simplified with flexible topic naming and a new GUI for easier geo-replication and data movement. Zone selection for networks on AWS and Google Cloud is now available for enhanced high-availability.

3 posts · 2022–2023

Timeline

20222026

Feature trailQuiet since 2022

.NET Kafka Client and TPL Integration

This feature thread tracks the evolution of building streaming applications using Apache Kafka and .NET clients, focusing on leveraging the Task Parallel Library (TPL) for in-process parallelization to maximize throughput. It covers implementing producer and consumer pipelines with Dataflow blocks, managing buffering and parallelism, and ensuring data order guarantees. The goal is to enable .NET developers to build robust and efficient event streaming applications by abstracting low-level concurrency.

5 posts · 2016–2022

Timeline

…20172026

Feature trailQuiet since 2022

Kafka Client Connectivity Troubleshooting

This feature thread tracks the evolution of diagnosing and debugging Apache Kafka issues, focusing on common symptoms like increased connections, reduced message throughput, increased request rates, and longer consumer rebalance times. It aims to help engineers understand the root causes of these problems by examining broker and client metrics, connection types, and common misconfigurations. The series provides a systematic approach to troubleshooting Kafka deployments. This post specifically details how to use Apache Kafka command-line tools with Confluent Cloud, including configuration, producer/consumer usage, and connectivity troubleshooting.

2 posts · 2019–2022

Timeline

20192026

Feature trailQuiet since 2022

Observability for Self-Managed Kafka on Kubernetes

This feature thread tracks the evolution of observability for self-managed Kafka deployments on Kubernetes. It covers integrating monitoring tools like Datadog with Confluent for Kubernetes (CFK) to collect and visualize Kafka telemetry data, including metrics, logs, and traces. The goal is to provide comprehensive visibility into Kafka cluster health and performance, enabling proactive response and troubleshooting in complex, distributed environments.

6 posts · 2020–2022

Timeline

20202026

Feature trailQuiet since 2022

Unified CLI for Hybrid and Multi-Cloud

Confluent CLI v2 unifies the ccloud CLI and Confluent Platform CLI into a single client, simplifying hybrid and multi-cloud management. It enables seamless context switching between Confluent Cloud and on-premises deployments, and introduces features like multi-org support for cloud users and enhanced usability for on-prem users. The update facilitates easier migration to Confluent Cloud and provides a consistent command-line experience for all Confluent users.

4 posts · 2019–2022

Timeline

20192026

Feature trailQuiet since 2022

Real-time Edge Messaging Integration

This feature thread tracks the evolution of integrating edge devices with data streaming platforms for real-time data ingestion and processing. It covers architectures for connecting IoT devices to cloud platforms like AWS IoT Core and Confluent Cloud, enabling contextualization of raw device data by combining it with business system information. The goal is to enrich data streams for improved decision-making, operational efficiency, and enhanced customer experiences. This includes configuration and bidirectional communication between edge devices and Kafka topics via PubNub's Edge Messaging Platform, enabling direct connectivity from web and mobile apps to Kafka without complex gateway infrastructure.

2 posts · 2019–2022

Timeline

20192026

Feature trailQuiet since 2021

GraphQL API for Kafka Data

This feature thread tracks the evolution of using GraphQL as an API layer for accessing data stored in Apache Kafka. It clarifies common misconceptions about both technologies and explores how they can be used together to bridge the gap between backend Kafka data and frontend applications. The post details GraphQL basics, including its query language, schema definition, introspection, and operation types (query, mutation, subscription). It also addresses misconceptions about GraphQL's database requirements and explores its suitability for mobile apps and server-to-server communication. The thread covers integrating GraphQL with Kafka for real-time data access, enabling frontend applications to consume Kafka streams efficiently and with reduced over-fetching.

1 post · 2021

Timeline

20212026

Feature trailQuiet since 2021

Secrets Management with HashiCorp Vault

This feature thread tracks the evolution of building secure shared services platforms. It introduces Cluster Role-Based Access Control (RBAC) for granular management access to resources like production environments and sensitive clusters, complementing existing ACLs for application access to Kafka resources. Audit Logs are now generally available, enabled by default, to track user/application resource access for anomaly detection and bad actor identification, with logs available in topics. Cloud-agnostic secret management with HashiCorp Vault is integrated for secure credential access by Confluent Platform components, particularly Kafka Connect, via Kubernetes Service Accounts and Vault policies. This ensures credentials are only stored in Vault, accessed over secure connections with tight policy-based controls, and held only in memory by the components that need them.

3 posts · 2019–2021

Timeline

20192026

Feature trailQuiet since 2021

Containerized Kerberos and SSH Testing

This feature thread tracks the evolution of building secure testing environments for distributed systems, specifically focusing on containerized setups for authentication protocols like Kerberos and SSH. It covers the configuration of KDC servers, SSH servers, and clients within containers to enable passwordless authentication and facilitate the testing of connectors and applications that rely on these security mechanisms. The goal is to provide a reproducible and efficient way to validate security features in local development and CI/CD pipelines.

1 post · 2021

Timeline

20212026

Feature trailQuiet since 2021

Node.js Kafka Client

2 posts · 2019–2021

Timeline

20192026

Feature trailQuiet since 2020

Messaging System Performance Benchmarking

This feature thread tracks the evolution of understanding and comparing the performance characteristics of various messaging and event streaming systems. It covers detailed benchmarking methodologies, testbed configurations, and workload analysis to evaluate system throughput and latency. The goal is to provide engineers with data-driven insights to select the most appropriate system for their specific use cases, considering factors like hardware utilization, end-to-end latency, and tail latencies.

1 post · 2020

Timeline

20202026

Feature trailQuiet since 2020

Go Binary Code Coverage

This post introduces Bincover, an open-source tool developed by Confluent to measure code coverage for Go binaries, particularly for end-to-end and system tests. It addresses limitations of the built-in `go test` tool by enabling coverage measurement for binaries that execute `main()` and handle panics or `os.Exit()` calls. Bincover instruments binaries to collect coverage profiles, supports passing command-line arguments and environment variables, and merges profiles from multiple test runs. This capability is crucial for ensuring the robustness and testability of Go-based Confluent products like CLIs.

1 post · 2020

Timeline

20202026

Feature trailQuiet since 2020

Go Binary Code Coverage

This feature thread tracks the evolution of tools and techniques for measuring code coverage in Go binaries, particularly for end-to-end and system tests. It addresses the limitations of standard Go testing tools when dealing with binaries that execute main functions, handle panics, or call os.Exit(). The focus is on providing a robust and flexible solution for generating instrumented binaries, running them with custom arguments and environment variables, and merging coverage profiles from multiple test runs. This enables better testing and quality assurance for Go-based products and services.

1 post · 2020

Timeline

20202026

Feature trailQuiet since 2020

API Mocking and Proxying

This feature thread tracks the evolution of API mocking and proxying tools to reduce the 'blocked factor' in API-driven development. It covers solutions like Mox, which acts as a hybrid proxy and mocking framework, allowing developers to proxy requests to a live backend while selectively mocking specific endpoints. This approach aims to balance the flexibility of API mocks with the maintainability of using a live backend, reducing developer downtime and improving iteration speed. The thread explores trade-offs between pure mocking and live backends, and discusses various implementation options like webpack-dev-server, MockServer, Interceptor, and GraphQL Faker.

1 post · 2020

Timeline

20202026

Feature trailQuiet since 2019

Ansible Deployment for Confluent Platform

CP-Ansible playbooks are now open-source, simplifying the deployment of Apache Kafka and Confluent Platform. This release focuses on providing best practices for configuration, security (TLS, Kerberos, SCRAM, SASL PLAIN), and customization. Future enhancements include rolling upgrades, RBAC configuration, and FIPS operational readiness.

2 posts · 2018–2019

Timeline

20182026

Feature trailQuiet since 2019

Kafka Connect Single Message Transforms

This feature thread tracks the evolution of Kafka Connect's Single Message Transforms (SMTs), enabling in-flight message manipulation without custom code. This includes setting message keys, adding metadata for data lineage, masking or dropping fields, and routing messages to different topics based on regex patterns or timestamps. SMTs enhance data pipelines by allowing for pre-processing, enrichment, and flexible routing before data reaches its final destination or is processed by downstream systems like Elasticsearch.

6 posts · 2016–2019

Timeline

…20172026

Feature trailQuiet since 2019

Kafka Client-Broker Compatibility

This feature thread tracks the evolution of Apache Kafka's reliability and operability, focusing on bug fixes and enhancements to core components like replication, MirrorMaker, logging, log compaction, connection handling, metadata management, and the request purgatory. It emphasizes the importance of correctness and stability over new features, with a significant number of bugs fixed in the 0.8.2 release. The thread also covers the introduction of a new distributed testing framework to ensure compatibility and the new producer client.

4 posts · 2014–2019

Timeline

…20172026

Feature trailQuiet since 2015

Kafka Request Purgatory Performance

This feature thread tracks the evolution of Kafka's request purgatory, focusing on performance and memory efficiency. The initial implementation using Java's DelayQueue suffered from O(log n) insert/delete costs and inefficient purging, leading to potential OutOfMemoryErrors and performance degradation under high load. The new design leverages Hierarchical Timing Wheels and doubly linked lists to achieve O(1) insert/delete operations for completed requests and a more efficient purging mechanism. Benchmarks demonstrate significant improvements in request per second (RPS) throughput and reduced CPU usage, especially in high enqueue rate scenarios. The new design also optimizes the driving clock using DelayQueue by enqueuing task buckets instead of individual tasks, further reducing overhead.

2 posts · 2015

Timeline

…20172026