Log Compaction – Highlights in the Apache Kafka ® and Stream Processing Community – May 2017 | Confluent

What this post added

This post introduces the concept of using Kafka's compacted topics to implement GDPR's 'right to be forgotten'. It explains how messages can be marked for deletion via their keys and subsequently removed during the compaction process. Alternative strategies like encrypting data with user-specific keys and deleting those keys, as well as handling deletes in downstream databases via Kafka Connectors, are also discussed. Tuning parameters for compaction are provided.