Best practices using Spark SQL streaming, Part 1

This post introduces best practices for Spark SQL Streaming, focusing on handling multiple streaming sources with different schemas. It advocates for routing messages from diverse sources (e.g., MQTT) through a well-integrated message queue like Kafka before processing with Spark SQL Streaming, rather than building custom connectors for each source. This approach aims to leverage Spark's reliability and performance guarantees. The post provides two examples: a classic word count using Spark SQL Streaming for messages from a single MQTT queue routed through Kafka, and a more complex scenario involving processing multiple streams with different schemas and disparate sources, demonstrating how to distinguish and process these streams based on prepended bits and route them to different Kafka topics for separate processing logic.