TechPrufer
Get started
Confluent
Confluent logoConfluent · Feature trail

Data Privacy and De-identification for Kafka

2 posts · Dec 2017 – May 2026

What this trail covers

Confluent Cloud continues to evolve data privacy and de-identification techniques for real-time data streams within Apache Kafka. This iteration focuses on integrating third-party platforms like Privitar to apply privacy policies and transform sensitive data in transit. The focus is on enabling compliance with regulations such as GDPR, CCPA, and HIPAA without sacrificing analytical utility. This includes demonstrating de-identification scenarios, configurable privacy rules, and the optional re-identification of data. The architectural approach to digital sovereignty is further detailed, emphasizing architectural guarantees over policy assurances, the schema as a new sovereignty boundary with integrated sensitivity and jurisdiction tagging, and the importance of open protocols for portability. This post specifically details the implementation of client-side field-level encryption (CSFLE) and client-side payload encryption (CSPE) as architectural controls for data sovereignty, and the use of schema registry for enforcing data contracts and sensitivity tags.

All Confluent trails
  1. Confluent's architectural approach to digital sovereignty for real-time streamingThis post details Confluent's architectural approach to digital sovereignty for real-time streaming, emphasizing architectural guarantees over policy assurances. It introduces the concept of the schema as a sovereignty boundary, enabling the annotation of sensitivity and jurisdiction tags, and the enforcement of data contracts and encryption directives (CSFLE/CSPE) at produce time. The post also highlights the importance of open protocols (Kafka, Flink) for portability and discusses the operational trade-offs of achieving architectural sovereignty, such as BYOC, self-managed deployments, and key management. The core technical contribution is the articulation of how schema-level controls, combined with BYOK and private networking, provide a robust architectural foundation for data sovereignty in regulated environments.
  2. Data Streaming Privacy, Security, and Compliance in KafkaThis post introduces the integration of the Privitar Data Privacy Platform with Confluent Platform via the Privitar Kafka Connector. It details how Privitar's Privacy Policies and rules are applied in real-time to de-identify sensitive data within Kafka messages, demonstrating specific scenarios with JSON data and various de-identification techniques like tokenization and perturbation. The post also covers the configuration of the Privitar Policy Manager and the setup requirements for the Kafka Connector, including its connection to Privitar hosts, user credentials, job ID, and topic configurations. It highlights the ability to maintain referential integrity and enable controlled re-identification.
  3. Handling GDPR with Apache Kafka: How does a log forget? | ConfluentThis post introduces two primary methods for handling data deletion in Kafka to comply with GDPR's 'right to be forgotten': 1. Using Kafka's compacted topics by sending a message with a null value for a given key to mark it for deletion during compaction. It also details how to handle cases where the deletion key differs from the message key, suggesting composite keys or external mapping systems, and mentions Kafka Streams as a potential solution. 2. An alternative approach involving per-user encryption keys, where deleting the key renders the user's data unintelligible. The post also covers propagating deletes from databases using CDC-enabled Kafka Connectors and provides guidance on tuning compaction settings (log.cleaner.min.cleanable.ratio, log.cleaner.io.max.bytes.per.second, log.roll.hours) and monitoring compaction progress via JMX metrics.