Data Lineage at Slack

This post details the implementation of data lineage ingestion for Airflow callbacks and dashboard usage. It describes the construction of a central Data Lineage Service that exposes endpoints for ingestion, parsing, and persistence of lineage data into RDS MySQL and subsequently flattened into Hive. The post highlights the use of ANTLR for SQL parsing and Pregel for building lineage graphs in Spark, enabling efficient querying and notification of downstream consumers via an internal Data Portal. Future work includes extending lineage to non-SQL Spark jobs, integrating with graph databases, and improving schema change automation.