TechPrufer
Get started
Confluent
Confluent logoConfluent · Feature trail

Apache Kafka Security Features

4 posts · Jan 2016 – Jul 2026

What this trail covers

Confluent Cloud continues to evolve its security features for Apache Kafka and Confluent Platform, focusing on robust audit logging capabilities. This iteration details the architecture for building a real-time compliance and audit logging pipeline using Kafka, Kafka Connect, stream processing (Kafka Streams/Flink), and tiered storage. Key aspects include ensuring data accuracy and immutability via Kafka's append-only log, guaranteed long-term retention with tiered storage, ironclad security with mTLS and RBAC, and continuous transparency through distributed tracing and structured audit logs. The focus is on enabling event-native governance by embedding policies, controls, and visibility directly into the data stream, ensuring secure, compliant, and reliable streaming systems by design.

All Confluent trails
  1. Deep Dive into SASL PLAIN and SCRAM in Kafka: Login Modules and Config Hot-ReloadThis post details the operational differences between SASL PLAIN (using PlainLoginModule and FileBasedLoginModule) and SASL SCRAM (using ScramLoginModule) in Kafka regarding credential management and hot-reloading. It explains that PlainLoginModule with static JAAS requires broker restarts for credential changes, while ScramLoginModule (leveraging cluster metadata) and FileBasedLoginModule (watching a credential file) support hot-reloading. It provides a concrete example with three listeners and practical recommendations, emphasizing SCRAM for its security and native hot-reload capabilities.
  2. Event-Native Governance: An Architectural Guide to Secure, Compliant, and Reliable Streaming SystemsThis post introduces the concept of 'event-native governance' for streaming systems, emphasizing that governance (policies, controls, visibility) should be built directly into the core of the streaming platform rather than retrofitted. It details architectural patterns and core principles: 'Define Before Ingest' (using schema registry for contracts), 'Zero-Trust Event Access' (RBAC/ACLs at topic/cluster level), 'Continuous Transparency' (metrics, tracing, audit logs), and 'Automated Lifecycle Control' (retention policies). It breaks down governance into five domains: Schema, Access and Security, Data Lineage and Traceability, Retention and Lifecycle Management, and Operational Governance. A step-by-step implementation guide is provided, focusing on defining data contracts, classifying sensitive fields, enforcing schemas at ingestion, configuring granular access controls, enabling end-to-end encryption, implementing lineage tracking, and setting strict retention policies.
  3. Real-Time Compliance & Audit Logging With Apache Kafka®This post details the architecture for a real-time compliance and audit logging pipeline using Confluent Cloud. It outlines the requirements for such a system, including data accuracy and immutability (leveraging Kafka's append-only log), long-term retention (using tiered storage and object storage), security (encryption, authentication, RBAC), low-latency processing (Kafka Streams/Flink), and data lineage (Schema Registry). The proposed architecture includes an ingestion layer with Kafka Connectors, a processing layer for normalization and enrichment, a storage layer for durable retention, and an access layer for monitoring and querying. It also emphasizes end-to-end governance through RBAC and data lineage.
  4. Confluent Platform 8.0: Client-side field level encryption (GA), and MoreConfluent Platform 8.0 introduces the General Availability of Client-Side Field Level Encryption (CSFLE), enabling encryption of individual fields within messages on the producer side. This feature complements existing security measures like TLS and RBAC. The release also includes passwordless authentication for Confluent Server and Schema Registry, extending the OAuth client credentials grant type with pre-signed assertions.
  5. Confluent Cloud Q4 '24 brings more security to the platformThis post announces the general availability of mTLS authentication for Confluent Cloud Dedicated clusters, providing two-way client and cluster authentication. It also introduces limited availability for AWS PrivateLink for Schema Registry and Egress Private Link for Enterprise clusters on AWS and Azure. Private networking for Confluent Cloud for Apache Flink on Azure is also in limited availability. User-defined functions (UDFs) for Flink on AWS are now generally available, supporting Java scalar and table functions. WarpStream introduces Orbit, a managed replication tool, and BYOC Schema Registry. Follower fetching on AWS is now generally available to reduce cross-AZ traffic costs.
  7. Secure Integrations With Any Cloud, Any Network | Confluent CloudThis post details Confluent Cloud's features for secure networking, focusing on both public and private connection scenarios for fully managed connectors. For public endpoints, it covers Public Egress IPs for firewall rules and Gateway/VNet Service Endpoints for traffic within cloud provider networks. For private endpoints, it explains DNS Forwarding for VPC/VNet peering and Transit Gateway, and Egress Access Points for AWS PrivateLink, enabling secure connections to data sources and sinks without traversing the public internet.
  8. Apache Kafka Security in Confluent Cloud - Project Metamorphosis Month 6This post details Confluent Cloud's security posture, emphasizing defense in depth. It highlights TLS/SSL encryption for transit, at-rest encryption with BYOK support on AWS, and AWS PrivateLink for secure network access. Authentication is enforced via SASL PLAIN and SSO integration. Authorization is managed through Role-Based Access Control (RBAC) for management operations and Access Control Lists (ACLs) for data access. An Audit Log is introduced to track changes to clusters.
  9. Best Practices for a Secure Kafka DeploymentThis post details best practices for securing Kafka deployments, covering authentication (SASL mechanisms like PLAIN, SCRAM, GSSAPI, OAUTHBEARER; mTLS; HTTP services), authorization (Kafka ACLs, Confluent RBAC with MDS), and encryption (SSL/TLS for data in transit). It highlights recommended authentication methods for brokers (mTLS, SASL SCRAM, SASL GSSAPI) and ZooKeeper, and explains how RBAC addresses limitations of ACLs by providing centralized, role-based access control for various Confluent Platform components.
  10. Kafka Streams and KSQL with Minimum Privileges | ConfluentIntroduced prefixed ACLs for Kafka Streams and KSQL, allowing applications to define prefixes for their internal topics (e.g., state store changelog topics). This enables DevOps teams to grant read/write/create access to these prefixed topics and consumer groups using wildcard ACLs (KIP-290), thereby adhering to the principle of least privilege without requiring full administrative access for stream processing applications. Two patterns are presented: one for agile teams with a shared team prefix, and another for application-level security with application-specific prefixes.
  11. Securing KSQL | ConfluentThis post details how to secure KSQL and the Confluent Platform by configuring SSL for encryption, SASL for authentication, and HTTPS for Confluent Schema Registry. It provides sample KSQL server properties and environment variable configurations. It also emphasizes the end-to-end security design of the Confluent Platform, covering clients, brokers, Kafka Connect, Schema Registry, ZooKeeper, and Control Center, and points to demo and tutorial resources for practical implementation.
  12. Secure Stream Processing with the Streams API in Kafka | ConfluentThis post details how the Kafka Streams API natively integrates with Apache Kafka's security features, specifically focusing on encrypting data-in-transit between clients and brokers, and enabling client authentication. It provides configuration examples for SSL encryption and client authentication within StreamsConfig, and explains how misconfigurations would lead to runtime errors. The post emphasizes that securing stream processing applications built with Kafka Streams involves configuring the underlying Kafka producer and consumer clients.
  13. Apache Kafka Security 101 | ConfluentThis post introduces the security features added in Apache Kafka 0.9, including client authentication (Kerberos, TLS), authorization (Unix-like permissions/ACLs), network encryption (TLS), and broker/ZooKeeper authentication. It provides detailed guidance on generating TLS keys and certificates, configuring Kerberos principals, and setting up ZooKeeper and Kafka broker configurations for these security protocols. The post emphasizes that these features are disabled by default and are backward-compatible.