TechPrufer
Get started
Uber
Uber logoUber · Feature trail

Secrets Management Platform

2 posts · May 2025 – Jun 2025

What this trail covers

Uber has developed a comprehensive Secrets Management Platform to address widespread secrets sprawl, insecure sharing, and the challenges of managing over 150,000 secrets across thousands of microservices, databases, and third-party integrations. This initiative involves implementing preventive measures like pre-commit hooks to stop secrets from entering codebases, and remediation strategies including real-time and scheduled scanning of code, Slack conversations, logs, and container images. The platform consolidates secret vaults, provides a metadata model for governance, supports CRUD operations across various secret managers, offers user interfaces (API, CLI, web UI), manages secret lifecycle workflows (rotation, deletion, revocation), enforces uniform access control, and integrates with third-party vendors and SaaS applications. It also enables automated secret rotation and deletion by integrating with multiple deployment platforms and handling various secret types, including those that don't support seamless rotation. The system tracks secret inventory and metadata, leveraging heuristics and crowdsourcing for missing information, and provides a Secrets Insights Dashboard for governance metrics.

All Uber trails
  1. Automating Kerberos Keytab Rotation at UberThis post details the implementation of automated, failure-domain-aware rotation for Kerberos keytabs at Uber, building upon the existing Secrets Management Platform (SMP). It describes the integration of the Keytab Distribution Pipeline (KDP) with SMP, where KDP acts as a Secret Provider to generate, rotate, and delete keytabs. The Keytab Manager component, residing with the Kerberos KDC, handles password changes and new keytab generation, uploading them to the Secret Store. The post elaborates on the challenges of keytab rotation at scale (over 100,000 keytabs) and the critical need to minimize authentication failures during the rotation process. It details specific strategies to mitigate failures during AS_REQ (ensuring TGT validity, minimizing fetch time) and AP_REQ (maintaining dual key versions on servers during transition). Additionally, it highlights the measure of migrating containerized applications to node-specific principals/keytabs to minimize blast radius.
  2. Building Uber’s Multi-Cloud Secrets Management Platform to Enhance SecurityThis post details the evolution of Uber's secrets management capabilities, focusing on the development of a centralized Secret Management Platform. Key contributions include the implementation of preventive measures (CLI pre-commit hooks) and remediation strategies (real-time and scheduled scanning of code, Slack, logs, container images) to combat secrets sprawl. It describes the consolidation of 25 disparate secret vaults into 6 managed vaults, reducing secrets distributed to data pipeline containers by up to 90%. The platform's core components are outlined: a metadata model (secret provider, deployment platform, secret impact level), user interfaces (API, CLI, web UI), secret lifecycle management workflows (rotation, deletion, revocation), access management, third-party integrations, and a governance dashboard. The post also elaborates on enabling automated secret rotation and deletion by integrating with various deployment platforms (Up, Kubernetes, Odin, Athena, YARN, Piper, DSW) and handling diverse secret types, including those requiring atomic pair rotation.