Building Kafka Client Failover: A 60-Second disaster recovery POC with Confluent Cloud Gateway

This post details a Proof of Concept (POC) for building Kafka client failover using Confluent Cloud Gateway to achieve disaster recovery within 60 seconds. It explores the challenges of RPO/RTO in Kafka outages, discusses server-side replication options like Cluster Linking, and highlights the complexities of client-side failover due to the static `bootstrap.servers` configuration. The POC demonstrates how Confluent Cloud Gateway can act as a traffic control layer to redirect client traffic to a passive cluster during an outage without client restarts. It also addresses the issue of read-only mirror topics and proposes a workflow involving connection dropping, reversing replication, and gateway switching. The architecture of the demo service is presented, emphasizing identity passthrough and OAuth2 support for failover.