Balancing quality and coverage with our data validation framework

This post details the creation of a new data validation framework at Dropbox. The framework was built from scratch after evaluating open-source options like Great Expectations and dbt. Key decisions included adopting an 80/20 approach to coverage, choosing SQL as the primary language for validation checks due to its widespread familiarity and flexibility, and storing validation rules as code in Git for better version control. The framework integrates with Airflow, executing validation queries after data ingestion and alerting engineers on failure. Examples of implemented checks for upstream data issues, profiling alerts, and business logic are provided.