TechPrufer
Get started
Confluent
Confluent logoConfluent · Feature trail

Shared Services Security and Resource Management

4 posts · Aug 2019 – Oct 2025

What this trail covers

Confluent Cloud enhances its capabilities for building secure shared services platforms. This includes Cluster Role-Based Access Control (RBAC) for granular management access to resources like production environments and sensitive clusters, complementing existing ACLs for application access to Kafka resources. Audit Logs are now generally available, enabled by default, to track user/application resource access for anomaly detection and bad actor identification, with logs available in topics. Cloud

All Confluent trails
  1. Real-Time Compliance & Audit Logging With Apache Kafka®This post details building a real-time compliance and audit logging pipeline using Apache Kafka, Kafka Connect, stream processing (Kafka Streams/Flink), and immutable object storage. It outlines the requirements for such a pipeline (data accuracy, immutability, long-term retention, security, low-latency processing, auditability/lineage) and presents a reference architecture. Key components include connectors for ingestion, stream processing for normalization/enrichment/validation, tiered storage for long-term retention, and access layers for monitoring and querying. It also emphasizes end-to-end governance through RBAC, data masking, and data lineage.
  2. Build a Secure Data Streaming Platform Across Shared ServicesThis post introduces several new features for building secure shared services platforms on Confluent Cloud. Centralized identity management is enabled via OAuth support, allowing integration with third-party identity providers for managing application identities and credentials. Enhanced RBAC now extends granular access controls to all Confluent Cloud resources, including Schema Registry, Connect, and ksqlDB, enabling consistent access control across users and applications. Cloud Client Quotas are introduced to control resource utilization in multi-tenant deployments, allowing custom throughput controls for individual applications or service accounts. Cloud Service Quotas provide visibility and budgeting for Kafka resources. AsyncAPI support is added for capturing Confluent Cloud implementations. Self-serve provisioning for Transit Gateway attachment resources is simplified. Terraform support for Cluster Linking is now generally available.
  3. Introducing Cluster Authorization Using RBAC, Audit Logs, and BYOK in Confluent CloudThis post announces the general availability of Cluster Role-Based Access Control (RBAC) and Audit Logs in Confluent Cloud. Cluster RBAC allows for role-based permissions per user to gate management access to resources like production environments, sensitive clusters, and billing details, with specific roles like OrganizationAdmin, EnvironmentAdmin, and CloudClusterAdmin. Audit Logs are enabled by default and capture user/application resource access in a Kafka topic for anomaly detection and analysis. The post also mentions updates to the BYOK offering for AWS (GA) and Google Cloud (preview) for data-at-rest encryption, and reiterates the availability of Private Link for secure networking.
  4. New Confluent Cloud Data Security FeaturesThis post announces the general availability of Bring Your Own Key (BYOK) encryption for dedicated clusters on AWS, allowing customers to use their own AWS KMS master keys for data-at-rest encryption. It also previews Audit Logs, which track user and application access to Kafka clusters and management actions, with events captured in a Kafka topic. Additionally, it previews Cluster RBAC, which will enable granular role-based access control for OrganizationAdmin, EnvironmentAdmin, and ClusterAdmin roles, allowing for more precise management of user permissions across different resources.
  5. Best Practices for a Secure Kafka DeploymentThis post details the security features of Apache Kafka and Confluent Platform, covering authentication mechanisms (SASL with PLAIN, SCRAM, GSSAPI, OAUTHBEARER; mTLS), authorization mechanisms (Kafka ACLs, Confluent Platform's centralized ACLs, Role-Based Access Control (RBAC) powered by Confluent's Metadata Service), and encryption for data in transit (SSL/TLS). It provides configuration examples for Kafka brokers, ZooKeeper, and HTTP-based services (Kafka Connect, Schema Registry, REST Proxy, ksqlDB, Control Center). The post emphasizes the importance of securing these components for enterprise production environments and highlights the benefits of RBAC for managing access across multiple Kafka clusters and services.
  7. Maximize Apache Kafka Security with Secret Protection | ConfluentThis post introduces Secret Protection, a commercial feature in Confluent Platform 5.3, which uses envelope encryption to encrypt secrets within configuration files. It extends previous KIPs for securing passwords in ZooKeeper and Connect configurations to cover all Confluent Platform components. The feature uses a master passphrase to derive a master encryption key, which then encrypts a data encryption key. This data encryption key is used to encrypt the actual secrets. The post details the workflow for generating the master key, encrypting configuration parameters (e.g., `config.storage.topic`), updating secrets, and operationalizing the workflow for production environments, including distributing the master key and secrets file, and restarting services.