Content Ingestion & Podcast Video Incident Report | Spotify Engineering

This post details a specific incident where Spotify's video transcoding infrastructure reached maximum capacity, causing a backlog and delaying video podcast episode publication for several hours. The incident was caused by a combination of insufficient headroom, a scheduled batch processing job, increased per-item processing cost due to recent quality improvements, and a software bug underutilizing compute resources. The response involved stopping the batch job, deploying a fix for the bug, and adding processing capacity. The post also outlines broader improvements to the reliability of the podcast publishing pipeline, including enhanced capacity planning, improved prioritization, and extended rate limiting and backpressure mechanisms.