Effectively Migrating Kafka to the Cloud: How Skai Did It

What this post added

Skai re-architected their Kafka data ingestion to drastically reduce topic count for scalability and cost-efficiency in Confluent Cloud. They moved from a per-table topic model to consolidated topics ('united-topic-single-pipeline', 'united-topic-multiple-pipelines', 'united-topic-heavy-load') based on traffic. This involved developing a dynamic lag handling API to promote tables to higher-tier topics and a caching library for external Kafka clients to manage dynamic topic mappings. They also implemented enhanced monitoring to pinpoint lag causes within consolidated topics.