Stream Processing with IoT Data: Best Practices & Techniques

This post details best practices and techniques for stream processing with IoT data, focusing on the challenges of variable connectivity, bursty data, mixed service levels, and diverse data formats. It highlights Apache Kafka as the core technology for durable storage, horizontal scalability, high throughput, and low latency. The post discusses designing a system around Kafka, including raw data ingest via intermediaries (REST/MQTT), handling large messages and backlogs, and implementing a partitioning strategy based on message reception epoch milliseconds for uniform distribution. It also touches upon the need for different mechanisms for storing and processing messages with varying durability and latency requirements.