TechPrufer
Get started
Dropbox
Dropbox logoDropbox · Feature trail

Data Validation Framework

2 posts · Jan 2021 – Feb 2023

What this trail covers

Dropbox developed a data validation framework to ensure the quality of data in its 55-petabyte Hadoop-based data lake. The framework uses SQL for checks, is integrated with Airflow for orchestration, and balances data availability with quality by performing checks after ingestion. It alerts on-call engineers via PagerDuty for failures and provides detailed error messages. The system aims to catch data quality issues like duplicates, missing mandatory fields, and incorrect business logic before data is used for analytics or ML.

All Dropbox trails
  1. Balancing quality and coverage with our data validation frameworkThis post details the creation of a new data validation framework at Dropbox. The framework was built from scratch after evaluating open-source options like Great Expectations and dbt. Key decisions included adopting an 80/20 approach to coverage, choosing SQL as the primary language for validation checks due to its widespread familiarity and flexibility, and storing validation rules as code in Git for better version control. The framework integrates with Airflow, executing validation queries after data ingestion and alerting engineers on failure. Examples of implemented checks for upstream data issues, profiling alerts, and business logic are provided.
  2. Why we chose Apache Superset as our data exploration platformDropbox adopted Apache Superset as its primary data exploration platform, consolidating over 10 previous internal tools. The decision was driven by Superset's ability to transform SQL queries into charts, enable quick ad-hoc analysis, facilitate sharing, encourage query reuse, and consume data from Hive/Presto clusters. Key evaluation criteria included security, user-friendliness, and maintainability. Dropbox contributed several features back to Superset, including alert improvements, schema permissions, Slack integration, and Presto/Hive support in CI.