Podcast Publishing Pipeline Reliability

3 posts · Feb 2015 – Jul 2026

What this trail covers

Spotify's Podcast Publishing Pipeline Reliability has been enhanced through improved capacity planning, prioritization mechanisms, and extended rate limiting and backpressure. This post details an incident where video transcoding infrastructure reached maximum capacity due to insufficient headroom, a scheduled batch job, increased per-item processing cost, and a resource scheduling bug. The incident response involved stopping the batch job, deploying a fix, and adding capacity. Future improvements include better capacity planning, improved prioritization, extended rate limiting and backpressure, and faster creator notifications.

All Spotify trails

Jul 2026Content Ingestion & Podcast Video Incident Report | Spotify EngineeringThis post details a specific incident where Spotify's video transcoding infrastructure reached maximum capacity, causing a backlog and delaying video podcast episode publication for several hours. The incident was caused by a combination of insufficient headroom, a scheduled batch processing job, increased per-item processing cost due to recent quality improvements, and a software bug underutilizing compute resources. The response involved stopping the batch job, deploying a fix for the bug, and adding processing capacity. The post also outlines broader improvements to the reliability of the podcast publishing pipeline, including enhanced capacity planning, improved prioritization, and extended rate limiting and backpressure mechanisms.
Apr 2015RAMLfications – Python package to parse RAMLThis post introduces RAMLfications, a new Python package that parses RAML specifications into Python objects. It enables validation against the RAML specification and provides programmatic access to API resources, methods, and parameters. The package was developed to support the creation of Spotify's API console, a tool for interacting with Web APIs. The post also outlines future plans for open-sourcing the API console and developing a RAML-based documentation generator.
Mar 2015Understanding the Spotify Web APIThis post details the Spotify Web API, a RESTful API that allows developers to retrieve and manage Spotify data over the internet. It covers accessing catalog data (albums, artists, tracks, playlists) via HTTP GET requests and JSON responses. It also introduces the concept of authentication and authorization using OAuth, specifically the Authorization Code flow, for accessing user-specific data and managing playlists. The post highlights the registration process for developers, the use of Client IDs and Secrets, and the concept of scopes for defining permissions. It also mentions the existence of a Playlist Miner application as an example of using the API for playlist management.
Advertisement