How Meta understands data at scale

This post details Meta's advancements in data understanding as part of the Privacy Aware Infrastructure (PAI) initiative. Key contributions include the development of a universal privacy taxonomy for standardized data privacy management, a "shift-left" approach integrating data schematization and annotations early in development, and the creation of OneCatalog for discovering, registering, and enumerating data assets. The post highlights solutions to challenges such as understanding data at scale (hundreds of data systems, millions of assets) through shared asset schema formats and managing long-tail systems; inconsistent definitions across diverse data systems by introducing a unified taxonomy of semantic types; missing annotations due to limited contextual information by combining schematization with in-code annotations and multiple classification signals; and organizational barriers through collaborative tooling and workflow integration. A walkthrough of understanding user data for the "Beliefs" feature in Facebook Dating illustrates the five-step approach: schematizing using Thrift IDL for a canonical DataSchema, predicting metadata at scale through the universal privacy taxonomy and data classification, and embedding privacy considerations throughout the data lifecycle.