TechPrufer
Get started
Uber
Uber logoUber · Feature trail

Automated Kerberos Keytab Rotation

1 post · Jun 2025

What this trail covers

Uber has developed a robust, failure-domain-aware automation system for orchestrating Kerberos keytab rotation without human intervention. This system integrates with the Secret Management Platform (SMP) and the Keytab Distribution Pipeline (KDP) to manage the lifecycle of over 100,000 keytabs across thousands of applications and nodes. Key innovations include minimizing authentication failures during rotation by carefully managing TGT validity and fetch intervals for clients, and ensuring servers can decrypt service tickets using previous key versions during transition phases. This enables safe, scalable keytab rotation.

All Uber trails
  1. Automating Kerberos Keytab Rotation at UberThis post details the architecture and solutions for automating Kerberos keytab rotation at Uber. It describes the integration of the Secret Management Platform (SMP) with the Keytab Distribution Pipeline (KDP) to manage keytab lifecycles. Key technical contributions include strategies to minimize authentication failures during AS_REQ by predicting TGT validity and optimizing keytab fetch intervals, and during AP_REQ by ensuring servers maintain both previous and current key versions to decrypt service tickets. The post also highlights the migration from single keytabs to node-specific principals for containerized applications to minimize blast radius.