Declarative Infrastructure Platform

5 posts · Feb 2016 – Apr 2023

What this trail covers

Spotify's declarative infrastructure platform continues to evolve, with this post detailing the development of Tsunami, a service for gradual, controlled rollouts of infrastructure changes. Tsunami addresses the risks associated with large-scale deployments of components like Docker and Helios, enabling gradual upgrades over extended periods to detect and mitigate issues early. It provides features like audit logs, granular control over rollout percentages, and monitoring of service-level object

All Spotify trails

May 2023Fleet Management at Spotify (Part 2): The Path to Declarative InfrastructureThis post details Spotify's journey to declarative infrastructure, moving from on-premise host-centric management (Puppet, Ansible) to a cloud-native approach using Kubernetes custom resources and operators. It outlines the requirements for a new solution (GitOps, introspection, data-driven configuration) and explains how Kubernetes was chosen to meet these needs. The implementation involves dedicated Kubernetes clusters, CI/CD integration with kpt, namespace-per-GCP project mapping, and user-facing tools like Backstage plugins for resource import and manifest generation.
Apr 2023Fleet Management at Spotify (Part 1): Spotify’s Shift to a Fleet-First MindsetThis post introduces the concept of 'Fleet Management' and a 'fleet-first mindset' at Spotify. It describes the problem of maintaining thousands of software components at scale, the challenges of manual, slow migrations, and the benefits of automating changes across the entire fleet. Key aspects discussed include identifying code to change, ensuring version control, mechanisms for applying changes safely (automated refactoring, CI verification, orchestrated rollouts), and increasing trust in automatically verified changes through improved test automation and regular rebuilds. The post highlights results such as reduced developer toil, a more secure codebase (e.g., Log4j fix), and faster availability of new internal framework features.
Mar 2021My beatThis post details the personal experience of a Senior Backend Infrastructure Engineer working with Kubernetes at Spotify. It highlights the engineer's role in helping other engineers with Kubernetes issues, debugging, and developing infrastructure services. A specific contribution mentioned is the development and open-sourcing of a 'k8s plug in' that simplifies the Kubernetes experience for developers, abstracting away underlying platform complexities.
Jun 2017Improving Critical Infrastructure RolloutsThis post introduces Tsunami, a new service designed to manage gradual rollouts of infrastructure changes. Tsunami allows for unattended, gradual deployments of configuration and infrastructure components, such as Docker upgrades, over extended periods (e.g., two weeks). It addresses the challenges of scaling critical infrastructure by enabling early detection of issues affecting only a small portion of the fleet. Key features include an audit log, granular control over rollout percentages with floor/ceiling logic, and monitoring of service-level objectives with automatic rollout termination upon breach.
Mar 2016Managing Machines at SpotifyThis post details the evolution of Spotify's machine management infrastructure, moving from manual processes and centralized operations to a self-service model. Key developments include the automation of DNS pushes, machine ingestion via iPXE and a pxeimage, and provisioning requests with Provgun and later Provcannon. The introduction of Neep (a job queue for machine operations like install/recycle) and Sid (a REST service integrating ServerDb, microservice ownership data, and Neep jobs) enabled engineers to manage machine capacity via a web UI, significantly reducing turnaround times from weeks to hours.
Advertisement