TechPrufer
Sign inGet started

Blogs›Uber›Secrets Management Platform

Secrets Management Platform

Secrets Management Platform

2
posts
2025

Uber has developed a comprehensive Secrets Management Platform to address widespread secrets sprawl, insecure sharing, and the challenges of managing over 150,000 secrets across thousands of microservices, databases, and third-party integrations. This initiative involves implementing preventive measures like pre-commit hooks to stop secrets from entering codebases, and remediation strategies including real-time and scheduled scanning of code, Slack conversations, logs, and container images. The platform has been extended to support automated periodic rotation of Kerberos keytabs, addressing the scale and complexity of managing over 100,000 keytabs across thousands of applications and nodes. This rotation process is designed to be failure-domain-aware, orchestrating keytab updates without human intervention and minimizing the risk of service disruption by carefully managing the Kerberos key version (kvno) during AS_REQ and AP_REQ phases. For AS_REQ, the system ensures sufficient TGT validity before rotation and minimizes the fetch time for new keytabs. For AP_REQ, servers are configured to maintain both previous and current key versions during the transition to prevent decryption failures. To further enhance safety, containerized applications have been migrated to use node-specific principals/keytabs to minimize blast radius.

2025

Automating Kerberos Keytab Rotation at Uber

6/18/2025

This post details the implementation of automated, failure-domain-aware rotation for Kerberos keytabs at Uber, building upon the existing Secrets Management Platform (SMP). It describes the integration of the Keytab Distribution Pipeline (KDP) with SMP, where KDP acts as a Secret Provider to generate, rotate, and delete keytabs. The Keytab Manager component, residing with the Kerberos KDC, handles password changes and new keytab generation, uploading them to the Secret Store. The post elaborates on the challenges of keytab rotation at scale (over 100,000 keytabs) and the critical need to minimize authentication failures during the rotation process. It details specific strategies to mitigate failures during AS_REQ (ensuring TGT validity, minimizing fetch time) and AP_REQ (maintaining dual key versions on servers during transition). Additionally, it highlights the measure of migrating containerized applications to node-specific principals/keytabs to minimize blast radius.

Building Uber’s Multi-Cloud Secrets Management Platform to Enhance Security

5/15/2025

This post details the evolution of Uber's Secrets Management Platform, focusing on the multi-cloud aspect and the implementation of automated secret rotation and deletion. Key contributions include: 1. Establishing a unified Secrets Management Standard and a visionary solution. 2. Deploying preventive (Git pre-commit hooks) and remediation (real-time and scheduled scanning of code, Slack, logs) strategies to combat secrets sprawl. 3. Consolidating 25 disparate secret vaults across multiple clouds into 6 centrally managed vaults, reducing blast radius and operational overhead. 4. Developing a core metadata model (secret provider, deployment platform, impact level) and building components for UI (API, CLI, web), lifecycle management, access control, third-party integrations, and an insights dashboard. 5. Enabling automated secret rotation and deletion by designing Secret Provider and Deployment Platform APIs, and integrating with various deployment platforms (Up, Kubernetes, Odin, Athena, YARN, Piper, DSW) to handle hundreds of secret types, including those requiring atomic pair rotation.