Streams and Tables in Apache Kafka: Storage Fundamentals

This post delves into Kafka's storage fundamentals, focusing on topics and partitions. It explains how topics are unbounded sequences of events, configurable with retention policies, and how brokers are serialization-agnostic. The core concept of partitions is highlighted as fundamental for Kafka's scalability, elasticity, and fault tolerance, enabling parallel reads and fault tolerance through replication. The post details how producers determine event partitioning using a partitioning function, emphasizing the importance of sending related events to the same partition for ordering guarantees within consumer groups. It also provides guidance on choosing the number of partitions and discusses common reasons for events with the same key ending up in different partitions.