Streaming Everywhere: Predicting Kafka and Software Architecture in 2021

What this post added

This post presents predictions for Apache Kafka and software architecture in 2021, focusing on technical advancements. It highlights the potential for Kafka clusters to support 10 million partitions through Tiered Storage (KIP-405) and ZooKeeper removal (KIP-500). It also predicts rapid Kafka cluster scaling (doubling size in seconds) enabled by Tiered Storage, with architectural benefits including reduced data migration and potential performance gains. The post foresees the proliferation of streaming features in various data products and the rise of event streaming as a dominant architectural paradigm. It also discusses the development of multi-paradigm products and the integration of database concepts into event-driven systems via ksqlDB.