Secrets Management with HashiCorp Vault

3/17/2021

This post introduces Confluent's implementation of HashiCorp Vault for secrets management. It details the setup process using Banzai Cloud's Vault operator on Kubernetes, backend configuration with HA Postgres on GKE, and HA testing methodologies using the Slapper load testing framework. It also provides examples of Golang programs for programmatic Vault configuration, such as creating AppRoles, and discusses the coexistence with Terraform for configuration management.

2/25/2021

This post details an end-to-end demo using the Confluent Operator to deploy Confluent Platform to Kubernetes. It demonstrates deploying a Kafka Connect GitHub Source Connector that securely accesses GitHub credentials stored in HashiCorp Vault. The post outlines the background of secret management challenges, the ideal solution for Confluent Platform users, and provides detailed steps for setting up a GKE cluster, deploying and initializing Vault, creating a GitHub repo and access token, storing the token in Vault, configuring permissions using Vault policies and Kubernetes Service Accounts, and finally deploying Confluent Platform with the integrated secret management.

8/14/2019

This post details Confluent Platform's Secret Protection feature, a commercial offering introduced in version 5.3, which uses envelope encryption to protect sensitive configuration parameters like passwords. It explains how to generate a master encryption key using a passphrase, encrypt individual configuration values using the `confluent secret file encrypt` command, and how these encrypted values are referenced in configuration files. The post also covers updating encrypted secrets and provides a workflow for production deployment, including exporting the master key, distributing the secrets file, updating configuration files, and restarting services. It highlights the use of KIP-226 and KIP-297 as foundational concepts.