Apache Storm Pipelines

Spotify has built and scaled multiple real-time pipelines using Apache Storm for various use cases, including ad targeting, music recommendation, and data visualization. These pipelines integrate with systems like Kafka and Cassandra, processing billions of events daily. The team has focused on architectural improvements, maintainability through externalized configuration and metrics visualization, and performance tuning of hardware, throughput, and latency. Key strategies include creating small logical topologies, promoting code reusability, parallelizing I/O, and implementing robust deployment and rollback strategies. In-memory caching with Guava's Expirable Cache has been used to manage state within Bolts. The system has evolved to independent Storm clusters and considers Storm on YARN for future resource utilization.