Enabling Security for Hadoop Data Lake on Google Cloud Storage

What this post added

This post details the technical challenges and solutions for migrating Uber's on-premise Hadoop data lake to Google Cloud Storage (GCS) on GCP. It focuses on the security aspects, specifically addressing the integration of Hadoop's Kerberos-based authentication and authorization model with GCS's IAM. Key contributions include the design of a layered access model (Foundations, Data Mesh, Dataset Access) for enhanced security in the cloud, and the development of a Storage Access Service as an intermediary to manage authentication and authorization, overcoming limitations of native GCP IAM for large-scale policy management and reducing attack vectors. The post also elaborates on the existing HDFS security architecture and the authentication flow to highlight the differences and integration points.