Automatic Detect PII in Real Time with Machine Learning

This post introduces a PII Detection stream processing app, PII UDFs/UDTFs for ksqlDB, and a PII SMT for Kafka Connect. These solutions leverage NLP and pattern recognition models to identify and redact Personally Identifiable Information (PII) in real-time within data streams. The PII Detection app acts as a pass-through filter, inspecting messages for PII entities and redacting them while retaining other data, and publishing entity metadata events for alerting. The ksqlDB UDFs/UDTFs (`containsPII`, `redactPII`, `extractPiiEntities`, `extractPiiEntityTypes`) provide flexibility for targeting specific fields and complex pipelining. The PII SMT for Kafka Connect removes sensitive information before data reaches Kafka brokers. These solutions support 25 entity types and can be deployed on the edge, on-premises, or in the cloud, integrating with Confluent Cloud's Stream Catalog and offering configurable deny lists and regex rules for custom entity detection.