Data logs: The latest evolution in Meta’s access tools

This post details the engineering challenges and solutions for providing users with access to granular 'data logs' from Meta's Hive data warehouse. It describes the infeasibility of direct per-user queries due to scale and partitioning, and the computational/storage costs of caching. The implemented solution involves batching user requests, using Core Workflow Service (CWS) for reliable task execution, and performing an INNER JOIN between user IDs and data rows within Dataswarm. Post-processing with PySpark splits data per user, and Hack language renders it into HTML. Key lessons learned include the critical importance of robust checkpointing for resilience against failures (e.g., out-of-memory errors during full table scans) and the implementation of verification steps (e.g., checking user ID columns) to ensure data correctness, particularly addressing Spark concurrency bugs that could lead to data being returned to the wrong user.