Inside LinkedIn's data context layer: curating trustworthy context for humans and agents

This post details the engineering effort behind LinkedIn's data context layer, a system that combines automatically generated dataset profiles with team-maintained knowledge about meaning, usage, and limitations to help people and AI agents discover the right assets and understand their context. The system uses an AI enrichment pipeline to generate baseline context from schemas, source code, and query logs, and teams supplement this with human-curated knowledge. The generated layer uses a four-stage pipeline (nomination, collection, generation, persistence) to process signals like query logs, source code, and sample values to create structured dataset profiles and embeddings. The human-curated layer uses a Git repository and pull requests for authoring and review, organizing context into global, team, and analysis scopes.