Stream Processing with IoT Data: Best Practices & Techniques

What this post added

This post details best practices and techniques for stream processing with IoT data, focusing on challenges like variable connectivity, bursty data, long tails of firmware versions, DoS-like events, mixed service levels, and varying data formats. It highlights Apache Kafka as the core technology for durable storage, horizontal scalability, high throughput, and low latency. The post discusses raw data ingest strategies using intermediaries like web servers or MQTT endpoints, and addresses handling large messages from disconnected devices by batching and careful partitioning strategies based on epoch milliseconds. It also touches upon the need for different storage and processing mechanisms for varying message priorities and durability requirements.