How Meta discovers data flows via lineage at scale

This post details Meta's implementation of data lineage for Privacy Aware Infrastructure (PAI), focusing on automated discovery of data flows across web, data warehouse, and AI systems. It introduces 'Privacy Probes' for runtime signal collection, comparing source and sink payloads to identify data matches with varying confidence levels (EXACT_MATCH, CONTAINS, NO_MATCH). The system uses static code analysis and runtime instrumentation to build lineage graphs, enabling scalable data flow discovery, efficient privacy control rollout (via Policy Zone Manager - PZM), and continuous compliance verification. The example of tracking 'religion' data in Facebook Dating illustrates the process of collecting signals from web endpoints, logging tables, databases, and propagating to data warehouses and AI systems.