Backend Infrastructure

4 posts · Feb 2013 – Feb 2022

What this trail covers

Spotify's backend infrastructure continues to evolve, with a focus on enhancing fault tolerance and reliability. This post details the introduction of 'Arrow,' an intelligent daemon that centralizes network communication features like health monitoring, circuit breaking, and metrics gathering. This approach aims to reduce complexity within individual services and improve the overall robustness of the distributed system. Complementary load balancing at the edge further enhances symmetry in managing traffic.

All Spotify trails

Mar 2022My beatThis post introduces 'Arrow,' an intelligent daemon designed to centralize network communication features such as health monitoring, circuit breaking, and metrics gathering for backend services. This aims to reduce complexity within individual services and improve the overall robustness of the distributed system. The post also mentions complementary load balancing at the edge.
Mar 2013Achieving fault-tolerance with intelligent daemonsThis post introduces 'Arrow,' a separate daemon process designed to handle intelligent client-side network communication for services. Arrow centralizes features such as connection pooling, health monitoring, circuit breaking, and metrics collection. By offloading these responsibilities from individual service client libraries to a local daemon, Spotify aims to improve resource utilization, simplify debugging, and enhance deployment flexibility. The post also discusses the complementary role of front-end load balancers for managing incoming traffic and achieving symmetry in fault-tolerance strategies.
Mar 2013Backend infrastructure at SpotifyThis post outlines Spotify's backend infrastructure strategy, emphasizing autonomous squads, transparent code, and self-service infrastructure. It details the partitioning of problems by feature, the challenges of scaling, and the development of infrastructure to support squads efficiently. Specific areas covered include provisioning for hybrid cloud/on-premise environments, storage solutions (Cassandra, PostgreSQL, memcached) with self-service cluster setup, a custom messaging layer, and capacity planning with automated scaling. The post also highlights the importance of service insulation through rate limiting and separate deployments.
Mar 2013How we use Python at SpotifyThis post details Spotify's extensive use of Python for backend services (approximately 80%) and data analysis (approximately 90% of map-reduce jobs). It highlights the preference for async frameworks like gevent for I/O-bound services and strategies for compute-bound services (performance testing, profiling, Cython, native libraries). The data analysis section introduces the Luigi package for building complex batch job pipelines and powering features like Radio and Discover. Other uses mentioned include model-based testing with GraphWalker and general scripting. The post also touches on the community involvement and hiring aspects related to Python.
Advertisement