How Zendesk Secures Kafka with Self-Hosted mTLS Authentication System

This post details Zendesk's implementation of a self-hosted mTLS authentication system for their Kafka clusters to secure inter-pod communication and enable a global event bus. It covers the problem statement of unauthenticated clients and the need for encrypted, authenticated, and authorized connections. The solution leverages HashiCorp Vault as a private Certificate Authority (CA) for PKI backend operations. It describes the client architecture with a PKI auth manager sidecar for certificate generation and regeneration, and the broker architecture with a TLS monitor for hot-reloading certificates. The post also discusses the trade-offs made regarding certificate revocation due to JVM limitations and emphasizes the importance of robust CA root certificate rotation, outlining the process and requirements for a zero-downtime transition.