Scaling StreamHub: Transitioning from Kinesis to Kafka for 145 Billion Daily Events

This post details the migration of StreamHub from Amazon Kinesis to AWS Managed Streaming for Apache Kafka (MSK) to handle a 150 billion daily event volume. It outlines the limitations of Kinesis at scale, including API reliability, cost of high partitioning, retention limits, and consumer throughput exhaustion. The adoption of Kafka Tiered Storage with a two-tier approach (local EBS for hot data, S3 for historical data) is highlighted as a key cost-saving and retention strategy. Significant challenges encountered at scale with MSK are discussed, such as broker network limits, instance sizing, tiered storage offload throughput, S3 request rate limits during retention changes, storage scaling cooldowns during incidents, managed control plane availability issues, and the slow, irreversible nature of scaling operations. Hardening strategies for StreamHub are also presented.