Enabling Security for Hadoop Data Lake on Google Cloud Storage

This post details the security architecture for migrating Uber's on-premise Hadoop data lake to GCP, specifically addressing the challenges of integrating the existing Hadoop security model (Kerberos, Delegation Tokens, HDFS ACLs) with Google Cloud Storage (GCS) and GCP IAM. It introduces a layered access model (Foundations, Data Mesh, Dataset Access) and a custom Storage Access Service to manage authentication and authorization, overcoming GCS IAM limitations and reducing attack vectors. The system bridges HDFS and GCS security models to enable secure data lake migration.