TechPrufer
Get started
Spotify
Spotify logoSpotify · Feature trail

Heroic Time Series Database

2 posts · Oct 2015

What this trail covers

Spotify's monitoring capabilities have been significantly enhanced with the introduction of Heroic, a scalable, in-house time series database. Heroic addresses the challenges of near real-time data collection and presentation at scale by leveraging Cassandra for storage and Elasticsearch for indexing. It supports global federation for improved availability and resilience. The system is complemented by the ffwd agent for metrics collection and Kafka for buffering, enabling rapid experimentation and a push-based approach to monitoring.

All Spotify trails
  1. Monitoring at Spotify: Introducing HeroicThis post introduces Heroic, Spotify's new scalable time series database built using Cassandra and Elasticsearch. It details the architecture, including Cassandra for primary storage and Elasticsearch for indexing, and addresses potential data safety concerns with Elasticsearch by enabling rapid index rebuilding. The system supports global federation for high availability. It also describes the ffwd agent for metrics collection and Kafka for buffering, enabling flexible service topology experimentation. The post highlights the approach of storing raw data and performing aggregations via the Heroic API, with a sampling density of typically one sample per time series every 30 seconds or more. It also mentions the integration with custom dashboards and alerting systems, and the open-sourcing of Heroic.
  2. Monitoring at Spotify: The Story So FarThis post details the evolution of Spotify's operational monitoring from Zabbix and a homegrown RRD-backed system (sitemon) to a new approach focused on 'alerting as a service' and a push-based graphing system. It introduces Lyceum, a library built on Riemann for reproducible trigger testing and deployment. For graphing, it discusses the limitations of Graphite (hierarchical naming, sharding, rebalancing) and the benefits of a tag-based approach. The post announces the development of Heroic, a new time-series database inspired by KairosDB, designed to address performance and stability issues, and to support self-service and distributed operational responsibility.