Monitoring at Spotify: Introducing Heroic

This post introduces Heroic, Spotify's new scalable time series database built using Cassandra and Elasticsearch. It details the architecture, including Cassandra for primary storage and Elasticsearch for indexing, and addresses potential data safety concerns with Elasticsearch by enabling rapid index rebuilding. The system supports global federation for high availability. It also describes the ffwd agent for metrics collection and Kafka for buffering, enabling flexible service topology experimentation. The post highlights the approach of storing raw data and performing aggregations via the Heroic API, with a sampling density of typically one sample per time series every 30 seconds or more. It also mentions the integration with custom dashboards and alerting systems, and the open-sourcing of Heroic.