How Meta discovers data flows via lineage at scale

What this post added

This post details Meta's development of a scalable data lineage system to support its Privacy Aware Infrastructure (PAI) initiative. It describes techniques for collecting data flow signals, including static code analysis and runtime instrumentation via 'Privacy Probes,' which capture and compare source and sink payloads. The post illustrates the implementation for tracking 'religion' data in the Facebook Dating app, covering its propagation through web systems and into the data warehouse for offline processing, highlighting the use of Policy Zones for privacy controls.