Automating Kerberos Keytab Rotation at Uber

What this post added

This post details the automation of Kerberos keytab rotation at Uber, a critical security credential management task. It describes the integration with Uber's Secret Management Platform (SMP) and the Keytab Distribution Pipeline (KDP) to achieve scalable, automated rotation. The post elaborates on the challenges of keytab rotation at scale, including the risk of service disruption due to kvno mismatches during authentication (AS_REQ and AP_REQ). It outlines the architectural solutions and timeline-based strategies implemented to minimize authentication failures for both client and server-side operations, including strategies like ensuring sufficient TGT validity, minimizing keytab fetch times, and maintaining dual key versions on servers during transition phases. The post also touches upon minimizing blast radius by migrating to node-specific principals/keytabs.