Spotify Feature Trails

Feature trailActive

AI-Assisted Developer Productivity Platform

Spotify's AI-Assisted Developer Productivity Platform continues to evolve with the integration of background coding agents for large-scale software maintenance. This iteration focuses on 'context engineering' for these agents, exploring how to effectively instruct them to perform complex, multi-repository migrations. The platform has moved from early open-source agents and a homegrown loop to leveraging Claude Code for more natural, task-oriented prompting and dynamic task management. Key advancements include the development of a scalable annotation platform that integrates human expertise with LLM capabilities to generate high-quality data for ML model training and evaluation, significantly increasing annotation volume and annotator productivity.

19 posts · 2022–2026

Timeline

20222026

Feature trailActive

Experimentation Platform

Spotify's experimentation platform, Confidence, continues to evolve by introducing the 'fixed-power design.' This design allows experiments to start without a pre-estimated sample size, instead estimating the required sample size from available outcome data and stopping when the current sample size exceeds this estimate. This approach, detailed in Nordin and Schultzberg (2024), enables the use of nonsequential methods for analysis without corrections, ensuring consistent point estimators and asymptotically nominal coverage for confidence intervals, mitigating issues with traditional sequential testing like biased estimators and wasted power.

19 posts · 2020–2026

Timeline

20202026

Feature trailActive

Data Lake Indexing for Point Queries

Spotify's data platform continues to evolve with Random Access Parquet (RAP), an external index that maps keys directly to file locations in the data lake, enabling fast point queries by key over datasets too large for traditional KV stores. This bridges the gap between analytical throughput engines and interactive online use cases, including AI agents. RAP operates on existing Parquet files, reducing query latency and bandwidth by replacing dependent reads with precomputed lookups. Optimization efforts are ongoing. The AI data assistant, Vedder, leverages domain-specific clusters (datasets, pairs, docs) curated by domain experts to provide trustworthy answers to natural language queries. Vedder follows a ReAct loop, generates SQL, and returns answers with sources. Cluster health is monitored continuously, and user conversations feed back into the system for ongoing improvement. Future explorations include knowledge outside of schemas, such as documentation and process definitions.

9 posts · 2013–2026

Timeline

…20172026

Feature trailActive

Podcast Publishing Pipeline Reliability

Spotify's Podcast Publishing Pipeline Reliability has been enhanced through improved capacity planning, prioritization mechanisms, and extended rate limiting and backpressure. This post details an incident where video transcoding infrastructure reached maximum capacity due to insufficient headroom, a scheduled batch job, increased per-item processing cost, and a resource scheduling bug. The incident response involved stopping the batch job, deploying a fix, and adding capacity. Future improvements include better capacity planning, improved prioritization, extended rate limiting and backpressure, and faster creator notifications.

3 posts · 2015–2026

Timeline

…20172026

Feature trailActive

AI-Assisted API Interaction

Spotify is enhancing its AI-Assisted Developer Productivity Platform by integrating LLM evals with online experiments. This creates a feedback loop where LLM evals, acting as automated judges for quality and relevance, precede experiments. The system calibrates LLM judges against real user outcomes from A/B tests, improving the accuracy of evals over time and raising the hit rate of experiments by filtering out less promising candidates. This approach aims to reduce the cost of experimentation by enabling natural language interfaces for complex APIs, such as the Spotify Ads API, through Claude Code plugins. These plugins translate natural language requests into sequences of API calls, leveraging OpenAPI specifications and OpenAPI Links to manage entity hierarchies and workflows. The system also includes pre-flight validation and transparent curl command generation for user auditability and control.

1 post · 2026

Timeline

Feature trailActive

Wrapped Animation System

Spotify's Wrapped experience continues to evolve, with the 2025 iteration introducing the 'Wrapped Archive' feature. This system identifies up to five 'remarkable days' from a user's listening history, generating personalized, LLM-generated narratives grounded in actual listening data. The system uses a priority-ordered set of heuristics to identify these days, such as 'Biggest Music Listening Day', 'Biggest Discovery Day', and 'Most Unusual Listening Day'. A distributed data pipeline aggregates candidate days, and reports are pre-generated asynchronously. Prompt engineering, model distillation, and a fine-tuned production model are used to generate over 1.4 billion reports. The system is designed for high-throughput writes using a column-oriented key-value database and proactive scaling for the global launch. An automated evaluation framework with LLM judges assesses report accuracy, safety, tone, and formatting, feeding into a remediation loop to fix issues and update guardrails.

4 posts · 2020–2026

Timeline

20202026

Feature trailActive

Spotify App Release Management Dashboard

Spotify's Release Manager Dashboard, a Backstage plugin, continues to evolve to centralize and streamline the mobile app release process. This iteration details the weekly release cycle for versions like 8.9.2, emphasizing the coordination between developer merges, alpha/beta testing, bug triage, and phased rollouts. The dashboard's role in providing real-time status, identifying blocking bugs, and assessing release readiness against key metrics (crash rates, test results, sign-offs) is highlighted.

3 posts · 2020–2026

Timeline

20202026

Feature trailActive

Shuffle Algorithm Enhancement

Spotify has evolved its Shuffle feature by introducing a 'Fewer Repeats' mode. This mode preserves statistical randomness while improving perceived variety by scoring multiple random sequences based on recent listening history and selecting the freshest one. A 'Standard Shuffle' mode remains available for pure randomness. The system is being refined to balance freshness, artist diversity, and playlist size without impacting latency. This post details the application of property-based testing to the algorithm and the introduction of a new scoring mechanism based on recency of play.

2 posts · 2015–2025

Timeline

…20172026

Feature trailCooling

Perimeter Reliability and Incident Response

Spotify's perimeter reliability has been enhanced through improved incident response, bug fixes in Envoy filters and Kubernetes memory limits, better configuration rollout strategies, and improved monitoring capabilities. This aims to prevent future outages caused by similar cascading failures. This post details an incident caused by a DNS resolver failure triggered by maintenance on GitHub Enterprise, highlighting the need for improved resiliency against DNS outages and the planned decommission. This post details an outage caused by a bug in a custom Envoy filter triggered by a change in filter order, leading to Envoy crashes and a memory limit misconfiguration with Kubernetes. Mitigation involved increasing perimeter capacity. Future improvements include fixing the bug, correcting the configuration mismatch, improving rollout strategies, and enhancing monitoring.

12 posts · 2013–2025

Timeline

…20172026

Feature trailQuiet since 2024

Dashboard Platform and Quality Framework

Spotify has developed a comprehensive Dashboard Platform and Quality Framework to enhance the creation, management, and consumption of dashboards. This platform leverages Tableau and Looker Studio, supported by a SQL scheduling tool for BigQuery data preparation. The Quality Framework includes automated 'Vital Signs' checks and a manual 'Spicy Dashboard Design Checklist' to ensure dashboards are high-quality and trustworthy, categorizing them as Low, High, or Golden. A Dashboard Portal provides a searchable catalog of all published dashboards, with features for curation, trust and quality indicators, and enhanced context for viewers.

2 posts · 2020–2024

Timeline

20202026

Feature trailQuiet since 2024

In-Car Experience Technical Decision-Making

Spotify is refining its approach to technical decision-making for the fragmented in-car experience. This involves breaking down complex problems into smaller, manageable RFCs, utilizing diagrams and code snippets for clarity, and fostering collaboration across different functions (Product, Engineering, Product Insights) to ensure alignment and comprehensive problem-solving. This iterative process aims to improve the development of features like AI DJ and Jam support in cars.

1 post · 2024

Timeline

20242026

Feature trailQuiet since 2024

Spotify Data Platform

Spotify's Data Platform, a large-scale GCP-based system handling over a trillion events daily, continues to evolve. This post details the adoption of Apache Crunch as a Java-based data processing framework on top of Hadoop, replacing the less performant Python-based Hadoop Streaming API. Crunch offers type-safety, higher performance, and higher-level abstractions for data pipelines. The platform also introduced `crunch-lib` for common operations and a unified execution layer with an Inversion of Control. The Data Collection platform handles over 1 trillion events per day, with its event delivery architecture evolving through numerous iterations, including a move to cloud-managed services like Pub/Sub and Dataflow. Data Management and Processing focus on empowering Spotify to utilize data effectively, ensuring data integrity, traceability, searchability, and accessibility. Over 38,000 actively scheduled pipelines handle hourly and daily tasks, executing on BigQuery, Flink, or Dataflow clusters, often utilizing Scio. Data pipelines generate data endpoints with retention policies, access controls, lineage tracking, and quality checks, managed via custom K8 operators. Monitoring includes alerts for data lateness, failing workflows, and endpoints, with Backstage integration for resource management, cost analysis, and quality assurance. The platform emphasizes building a strong community through documentation, onboarding, automated migrations, and a dedicated support channel.

23 posts · 2013–2024

Timeline

…20172026

Feature trailQuiet since 2024

Unified Header/Footer Component

Spotify is evolving its approach to managing shared UI components across different product segments. This post details the creation of the S4X-Masthead, a facade pattern implementation to unify MastheadHeader and MastheadFooter components for the Spotify for Artists segment. This aims to reduce duplication, prevent implementation drift, and simplify maintenance by centralizing dynamic content loading and exposing a consistent interface for user session and localization contexts. Future work involves unifying the desktop and web player clients by migrating the desktop client to a React-based UI, abstracting platform APIs to support different infrastructures, and achieving a single codebase for both clients. This has led to improved reusability, unified user experience and visual design, and increased speed of delivery.

2 posts · 2021–2024

Timeline

20212026

Feature trailQuiet since 2023

Recursive Clustering and Explainability

Spotify has developed a novel approach to clustering diverse datasets by combining dimensionality reduction (UMAP), hierarchical density-based clustering (HDBSCAN), and recursive application of these techniques to identify sub-clusters. This method is enhanced with a supervised machine learning layer (XGBoost with SHAP values) to provide explainability, allowing data scientists and user researchers to understand the drivers behind cluster formation and iterate faster on user insights. The system aims to make large, complex datasets manageable, discover nuanced user segments, and provide a validated explainability layer for findings.

1 post · 2023

Timeline

20232026

Feature trailQuiet since 2023

App Size Management

Spotify is actively managing and optimizing application size across its mobile platforms. This involves implementing pre-merge checks using tools like Emerge to analyze the impact of code changes on download and install sizes, with a 50KB threshold for additional review. Post-merge, a dedicated Slack channel tracks significant size changes. To address gradual, 'under the radar' growth, Spotify is evaluating compilation unit and resource file sizes, attributing them to owning teams via Emerge's C. This post details a significant modernization of the client-side architecture to improve the listening experience on all devices, particularly addressing performance issues on older phones and slower networks by optimizing how Liked Songs metadata is loaded and stored locally. This involved a substantial refactoring of the client codebase, replacing in-memory sorting with pre-computed tables stored on disk, and implementing batched downloads.

4 posts · 2020–2023

Timeline

20202026

Feature trailQuiet since 2023

Automated Content Marketing Acquisition System

Spotify has developed an automated system for content marketing to acquire users at scale. This system ingests content, ranks it using machine learning and heuristics based on popularity and performance metrics, generates creative ad assets (including static and animated visuals), deploys them to digital ad platforms, and learns from performance data to repeat the cycle. The system leverages tools like Adobe After Effects (via nexrender) for creative generation and machine learning models to optimize ad selection and performance, aiming to improve marketing efficiency and user acquisition.

1 post · 2023

Timeline

20232026

Feature trailQuiet since 2023

Build System Migration

Spotify has successfully migrated its iOS application to the Bazel build system, replacing the legacy Xcode build system. This migration aimed to improve developer productivity by significantly reducing build and CI times, enhancing the feedback loop for engineers. The process involved a phased rollout, side-by-side operation of both systems, and the development of tooling to ensure parity and safety. The integration with Xcode was achieved through rules_xcodeproj, and the release process was adapted. This post details the initial success of integrating Rome, an open-source artifact caching tool, with Artifactory to cache Carthage dependencies, resulting in a 50% reduction in CI build times for the Spotify for Artists app. Future work includes extending this to more projects and addressing local debugging limitations with cached binaries.

6 posts · 2019–2023

Timeline

20192026

Feature trailQuiet since 2023

Nearest-Neighbor Search Library

Spotify has developed and open-sourced Voyager, a new nearest-neighbor search library designed to succeed Annoy. Voyager is based on hnswlib and offers significant improvements in speed, accuracy, and memory usage compared to Annoy. It provides production-ready, fault-tolerant index files and fully supported Python and Java bindings with identical interfaces. Voyager is designed for flexibility, stateless deployments, and cost-effectiveness, addressing the evolving needs of Spotify's personalization, recommendation, and search systems.

1 post · 2023

Timeline

20232026

Feature trailQuiet since 2023

Taste Profile Personalization Control

Spotify has introduced a feature allowing users to exclude specific playlists from influencing their taste profile and recommendations. This addresses the issue of 'functional listening' (e.g., for sleep, parties, or children) and shared account listening, which previously skewed personalization. The system integrates this exclusion logic into core taste profile systems, impacting various recommendation surfaces like Home, Discover Weekly, and Radio. The feature was developed iteratively, starting with a focus on user control over personalization.

2 posts · 2023

Timeline

20232026

Feature trailQuiet since 2023

GKE Node Memory Analysis

Spotify has developed a new method for analyzing volatile memory on Google Kubernetes Engine (GKE) nodes. This approach combines open-source tools (AVML, dwarf2json, Volatility 3) to create kernel memory dumps and build symbol files, enabling detailed inspection of processes and memory activities. This provides an open-source alternative or complement to commercial monitoring solutions for GKE environments.

1 post · 2023

Timeline

20232026

Feature trailQuiet since 2023

Declarative Infrastructure Platform

Spotify's declarative infrastructure platform continues to evolve, with this post detailing the development of Tsunami, a service for gradual, controlled rollouts of infrastructure changes. Tsunami addresses the risks associated with large-scale deployments of components like Docker and Helios, enabling gradual upgrades over extended periods to detect and mitigate issues early. It provides features like audit logs, granular control over rollout percentages, and monitoring of service-level object

5 posts · 2016–2023

Timeline

…20172026

Feature trailQuiet since 2023

Design System Abstraction Layers

Spotify's design system (Encore) has evolved to offer multiple layers of abstraction (Config, Slots, Custom) to cater to diverse user needs, balancing out-of-the-box utility with customization flexibility. This approach enhances reuse, reduces static implementations, and democratizes default configurations by analyzing usage patterns. This post details a collaboration with the accessibility team to improve out-of-the-box accessibility in components and provide guidance where enforcement isn't possible. The post specifically explores the trade-offs between customization (low abstraction, direct CSS/HTML manipulation) and configuration (high abstraction, semantic parameters) for evolving design systems, emphasizing how feature and product maturity, timelines, and reusability influence the choice between these approaches. Spotify's value of team autonomy leads them to generally lean towards customization, while acknowledging configuration as another valuable tool.

3 posts · 2021–2023

Timeline

20212026

Feature trailQuiet since 2023

TV Spatial Navigation

Spotify's TV Spatial Navigation system has been developed to address the lack of native support for directional input in hybrid web applications on big screens. The system evolved from a naive approach requiring manual navigation logic and wrapper components to a more streamlined DX using React Hooks. This allows for automatic detection of navigable elements and reduces DOM complexity, improving developer experience and maintainability.

1 post · 2023

Timeline

20232026

Feature trailQuiet since 2023

ML-driven Personalization Systems

Spotify's ML-driven personalization systems continue to evolve, with a focus on leveraging reinforcement learning (RL) to optimize for long-term user satisfaction. This involves processing vast amounts of user data to build sophisticated models that predict user needs and preferences across various contexts, aiming to create a holistic and fulfilling audio experience. The system also considers factors like time of day, user activity, and device to tailor recommendations, ultimately benefiting both users and artists. The 'Algotorial' approach combines human editorial expertise with algorithmic recommendations to create personalized editorial playlists, balancing broad appeal with individual taste.

9 posts · 2015–2023

Timeline

…20172026

Feature trailQuiet since 2023

Wrapped Load Testing Platform

Spotify's Wrapped campaign requires robust load testing to handle the 'thundering herd problem' of millions of users accessing personalized content simultaneously. The internal Moshpit tool, integrated with Backstage, is used to simulate realistic traffic patterns by sending diverse Protocol Buffer payloads. Key learnings emphasize testing with representative payloads, provisioning for upstream services, and testing across multiple regions and endpoints to ensure global scalability and reliability.

2 posts · 2022–2023

Timeline

20222026

Feature trailQuiet since 2023

Deep Linking and Attribution System

Spotify has undertaken a multi-year effort to repair and enhance its deep linking and attribution systems. This involved migrating to a new solution that significantly reduced broken links and improved engagement with linked content. The ongoing work focuses on achieving comprehensive attribution views and supporting new use cases, continuously improving the underlying technology based on new perspectives.

1 post · 2023

Timeline

20232026

Feature trailQuiet since 2023

Ray Platform for ML

Spotify has introduced a centralized Spotify-Ray platform to democratize ML innovation and support a broader range of ML practitioners, including data and research scientists. This platform leverages Ray to provide flexible, scalable, and accessible infrastructure for ML development, from prototyping to production. It aims to lower the barrier to entry for diverse ML tooling and workflows, including support for novel ML solutions, GPU access, and advanced paradigms like RL and GNNs. The platform also details the evolution of ML infrastructure at Spotify, moving from early custom tooling to standardizing on TensorFlow Extended (TFX) and Kubeflow Pipelines (KFP) for end-to-end ML workflows. This includes managing Kubeflow clusters, providing SDKs and CLIs, and addressing integration challenges between TFX and KFP.

3 posts · 2019–2023

Timeline

20192026

Feature trailQuiet since 2022

Vulnerability Management Platform

Spotify's Vulnerability Management Platform (VMP), codenamed Kitsune, has been developed and implemented to efficiently and scalably reduce security risks. It automates the process of identifying, notifying asset owners, and tracking the remediation of vulnerabilities. The platform integrates with various reactive controls (RCs) and supports manual uploads, providing a centralized source of truth for vulnerabilities. It also offers metrics and visualizations on Backstage to drive engagement and

6 posts · 2018–2022

Timeline

20182026

Feature trailQuiet since 2022

Cloud Carbon Footprint Plugin for Backstage

Spotify has developed and open-sourced the Cloud Carbon Footprint (CCF) plugin for Backstage, an open platform for developer portals. This plugin helps development teams understand the environmental impact of their cloud usage across AWS, Azure, and Google Cloud. It converts cloud utilization into estimated energy usage and carbon emissions, displays metrics and carbon savings, and provides relatable equivalencies. The plugin also offers forecasts, recommendations, and carbon intensity maps, aiming to shift resource management left and integrate climate awareness into daily engineering practices.

1 post · 2022

Timeline

20222026

Feature trailQuiet since 2022

Algorithmic Impact Assessment Process

Spotify has established a process for Algorithmic Impact Assessments (AIAs) to evaluate and mitigate potential algorithmic inequities and harms. This involves a combination of centralized and distributed efforts, encompassing research, product and tech impact collaboration, and external engagement. The process aims to increase accountability and visibility by prompting product owners to review their systems, create roadmaps for addressing potential issues, and share audits with stakeholders. The company has educated a significant number of employees and assessed over a hundred systems, leading to improvements in recommendations, search, data availability, data usage, safety mechanisms, and content moderation.

1 post · 2022

Timeline

20222026

Feature trailQuiet since 2022

Internationalization and Localization Platform

Spotify has significantly expanded its language support, growing from 26 to over 60 languages. This involved a comprehensive technical effort to adapt the existing architecture, automate quality assurance, and update backend services en masse. The project also emphasized strong vendor partnerships, cross-functional collaboration, and clear communication to achieve a rapid, large-scale localization rollout.

2 posts · 2019–2022

Timeline

20192026

Feature trailQuiet since 2022

Web Performance Measurement and Storytelling

Spotify's approach to web performance measurement and storytelling has evolved to emphasize the critical role of data in demonstrating the value of performance improvements. This includes leveraging lab data from tools like Lighthouse CI for early feedback on pull requests and integrating real user data from sources like Google Analytics into dashboards for monitoring performance in production. The focus is on using core Web Vitals metrics (LCP, FID, CLS) and other supported metrics (FCP, TTFB, INP) to identify areas for improvement, validate the impact of changes, and build a compelling narrative for stakeholders. This iterative process, combining lab and field data, allows for targeted optimizations and a better understanding of the user experience.

1 post · 2022

Timeline

20222026

Feature trailQuiet since 2022

Marketing Campaign Management Platform

Spotify has evolved its marketing campaign management capabilities by developing the Spotify Toolbelt, a platform designed to empower distributed local marketing teams to create and manage campaigns across various markets and surfaces. This platform, built upon a microservices architecture with a Node.js Express marketing vertical, utilizes a Scheduler-API powered by JSON blueprints and a decision tree for dynamic content targeting. A reusable 'editor' React component, driven by JSON schemas, provides an intuitive WYSIWYG interface for internal users to configure campaign elements, ensuring guardrails and reducing errors. The system orchestrates translation workflows and approval processes, significantly reducing the time and effort required for campaign creation and enabling more frequent, localized campaigns.

1 post · 2022

Timeline

20222026

Feature trailQuiet since 2022

Software Catalog and Visualization

Spotify's Backstage, a homegrown developer portal, has evolved significantly. Initially developed to address onboarding time and discoverability issues caused by rapid growth and autonomous teams, it now manages over 2,000 backend services, 300 websites, 4,000 data pipelines, and 200 mobile features. Over 200 engineers have contributed to Backstage, with 120+ plugins developed by 50+ teams, demonstrating strong internal adoption and community engagement. The platform provides a centralized location for infrastructure tooling, services, and documentation, offering a unified UI for developers to create, manage, and discover resources. This post details the open-sourcing of Backstage and its core functionalities, including creating new microservices, tracking pull requests, centralizing technical documentation, and reviewing mobile feature performance.

10 posts · 2020–2022

Timeline

20202026

Feature trailQuiet since 2022

Audio-to-MIDI Conversion Model

Spotify has developed and open-sourced Basic Pitch, a lightweight and accurate machine learning model for converting audio to MIDI. This tool is designed for artists and producers, offering polyphonic and instrument-agnostic transcription with pitch bend detection, and can run faster than real-time. The model utilizes a shallow architecture with a harmonic constant-Q transform, jointly modeling onsets, frames, and multipitch information, resulting in a small footprint (<20MB peak memory, <17K parameters). The project also aims to inspire the development of other lean ML models.

1 post · 2022

Timeline

20222026

Feature trailQuiet since 2022

Live Audio Recording and Distribution

Spotify Live has introduced a native recording feature, allowing creators to record live audio streams and distribute them as podcasts. This system handles room creation events, initiates cloud recording, processes raw audio files (.ts and .m3u8), stitches them into a single .mp4 file using FFmpeg, embeds metadata, and uploads the final recording to S3 for distribution to the creator. The system also includes error handling for real-time audio disruptions and silenceremoval during post-processing.

1 post · 2022

Timeline

20222026

Feature trailQuiet since 2022

Spotify Player API and SDK

Spotify's Player API and Web Playback SDK provide developers with programmatic control over Spotify playback, enabling a wide range of integrations from home automation to social listening experiences. The API has evolved to include features like adding to queue and has been enhanced by the Web Playback SDK for in-browser player functionality. Future developments aim to further expand its capabilities.

1 post · 2022

Timeline

20222026

Feature trailQuiet since 2022

Backend Infrastructure

Spotify's backend infrastructure continues to evolve, with a focus on enhancing fault tolerance and reliability. This post details the introduction of 'Arrow,' an intelligent daemon that centralizes network communication features like health monitoring, circuit breaking, and metrics gathering. This approach aims to reduce complexity within individual services and improve the overall robustness of the distributed system. Complementary load balancing at the edge further enhances symmetry in managing traffic.

4 posts · 2013–2022

Timeline

…20172026

Feature trailQuiet since 2022

Natural Language Search for Podcast Episodes

Spotify has introduced Natural Language Search for podcast episodes, moving beyond traditional term-based matching to semantic correlation. This system leverages deep learning, transformer neural networks, and approximate nearest neighbor (ANN) search techniques to enable users to find relevant content using natural language queries, even when exact keywords are not present. The solution involves training dense retrieval models to produce query and episode vectors in a shared embedding space, enabling efficient vector search for retrieval. The system is integrated into the production workflow using the Vespa search engine for offline indexing and online query encoding and retrieval.

1 post · 2022

Timeline

20222026

Feature trailQuiet since 2021

Blend Personalized Playlist

Spotify's Blend feature, which creates personalized playlists combining the tastes of two users, has evolved significantly. Initial development focused on addressing the challenges of scale and balancing multiple attributes like relevance, coherence, equality, and democracy. The team optimized the algorithm for latency by identifying and improving 'hot spots' in the code, leading to a tenfold reduction. Qualitative and quantitative methods, including heuristic reviews and a 'Content Recommendation Scorecard' with a Data Curation team, were used to ensure content quality. Blend Data Stories were developed to communicate taste similarities and differences to users, with a Wrapped Blend experience being a notable application. Ongoing work aims to further enhance the social experience through music.

1 post · 2021

Timeline

20212026

Feature trailQuiet since 2021

Audio Effects Library

Spotify has developed and open-sourced Pedalboard, a Python audio effects library built on JUCE. It allows programmers to use studio-quality audio effects in their code, bridging the gap between professional DAWs and Python. Pedalboard supports built-in effects, VST3/AU plugins, and is significantly faster than existing Python audio effect packages. It has found applications in ML data augmentation, content creation automation, and creative audio manipulation.

1 post · 2021

Timeline

20212026

Feature trailQuiet since 2021

Search Platform Reorganization

Spotify reorganized its Search teams to address challenges of scale and cross-team collaboration. The new structure creates two groups: one focused on the core Search experience for end-users and another dedicated to improving developer happiness and experimentation within the Search domain. This aims to improve both user satisfaction and developer productivity by providing more autonomy and context-driven goals. This post details how Slack's Machine Learning Services team solves search ranking problems and the challenges training their models.

2 posts · 2019–2021

Timeline

20192026

Feature trailQuiet since 2020

Tech Debt and Migration Management

Spotify's approach to managing technology debt and migrations has evolved to minimize engineering team exposure and reduce fragmentation. This involves a company-wide migrations map, productizing migrations with dedicated PMs and automation, and extensive automation to abstract underlying infrastructure changes. The goal is to enable faster innovation by moving up the stack and leveraging cloud capabilities.

3 posts · 2019–2020

Timeline

20192026

Feature trailQuiet since 2020

macOS CI Bare Metal Migration

Spotify's Client Platform tribe continues to enhance the CI/CD experience for mobile apps. This post details the successful migration from macOS virtual machines to bare metal machines for iOS application builds. This transition significantly reduced build times (up to 75% for pre-merge checks) and improved stability by eliminating the performance variability inherent in shared VM environments and SAN storage. The team plans to further increase the adoption of bare metal machines and eventually replace the legacy Xcode build system with Bazel.

2 posts · 2013–2020

Timeline

…20172026

Feature trailQuiet since 2020

Personalized Home Screen Shortcuts

Spotify's Home screen has evolved to include 'Shortcuts,' a dedicated area at the top displaying six personalized tiles (entities) reflecting recent listening habits. This feature aims to improve user experience by providing quick access to familiar music and podcasts, balancing familiarity with freshness. The development involved identifying and quantifying the opportunity, developing recommendation heuristics and models, and performing both offline and online evaluations to reliably recommend content.

1 post · 2020

Timeline

20202026

Feature trailQuiet since 2020

Podcast API

Spotify's Podcast API has been launched to third-party developers, allowing them to manage user podcast libraries, search the catalog, and fetch detailed show/episode information. The API design considered backward compatibility with existing endpoints (like the Player API), consistency with other Spotify Web APIs (error codes, auth, payloads), app terminology, and payload verbosity. The API is monitored using Grafana and Lightstep, with alerts triggering PagerDuty for outages. Prior to public launch, a select group of partners provided feedback to refine the design and onboarding experience, with a focus on clear technical documentation.

1 post · 2020

Timeline

20202026

Feature trailQuiet since 2020

CDN Service Standardization

Spotify has standardized its CDN services by migrating various content types (images, client updates, etc.) to Fastly's edge cloud platform. This initiative aims to reduce complexity, improve efficiency, and provide a unified approach to content delivery across diverse engineering teams. The process involved creating a self-service tool (SquadCDN) for requesting CDN reviews and approvals, leveraging Fastly APIs and VCL for intelligent caching and edge logic, and implementing robust security measures like Edge Dictionaries. The migration has led to over 80 services pushing content through Fastly more efficiently, with over 20% of the R&D organization adopting the new system.

1 post · 2020

Timeline

20202026

Feature trailQuiet since 2019

Test Flakiness Management

Spotify has developed and continues to evolve systems for identifying, tracking, and mitigating test flakiness. This includes visualization tools like Odeneye to distinguish between test flakiness and infrastructure issues, simple reporting tables that have shown significant reductions in flakiness, and automated bots like Flakybot to provide pre-merge confidence in test stability. The ongoing effort aims to improve overall test reliability and engineer confidence in continuous delivery.

1 post · 2019

Timeline

20192026

Feature trailQuiet since 2019

Documentation Platform for Monoliths and Monorepos

Spotify has developed and open-sourced a MkDocs plugin to address the challenges of documenting large monoliths and monorepos. This plugin introduces a new '!include' syntax for incorporating subfolder mkdocs.yml configurations, allowing for a declarative approach to navigation and enabling squads to structure their documentation independently while centralizing it into a single site. It leverages GitHub Codeowners for ownership configuration and brings documentation closer to the code at a folder level, improving developer experience by allowing MkDocs to be invoked on specific documentation sections.

1 post · 2019

Timeline

20192026

Feature trailQuiet since 2019

TypeScript Adoption and Support

Spotify is actively driving the adoption and support of TypeScript across its highly distributed engineering community. This involves establishing best practices, providing resources, and fostering a culture of shared learning to ensure successful integration and utilization of TypeScript within various engineering teams and projects. The `@spotify/web-scripts` project provides opinionated tooling to simplify the creation and maintenance of TypeScript NPM libraries, including build, linting, testing, and release configurations.

2 posts · 2019

Timeline

20192026

Feature trailQuiet since 2019

Embedded Application Programming

This post introduces the concept of embedded application programming at Spotify, distinguishing it from embedded system programming. It outlines the core principles and constraints of embedded application development, focusing on simplicity, elegance, memory management, concurrency, portability, testability, and performance. The goal is to encourage and guide engineers towards writing high-quality, efficient embedded code within Spotify's ecosystem.

1 post · 2019

Timeline

20192026

Feature trailQuiet since 2019

Distributed Tracing for Microservices

Spotify's approach to distributed tracing for microservices continues to evolve, focusing on making traces more valuable beyond just raw data. This involves enhancing the art and science of distributed tracing to redefine how engineers interact with and understand microservice architectures.

1 post · 2019

Timeline

20192026

Feature trailQuiet since 2019

Web Player Architecture and Performance

Spotify's web player has undergone a significant architectural overhaul, moving from an iframe-based, multi-library approach to a unified Single Page Application (SPA) built with React + Redux. This rewrite addressed performance issues, improved maintainability, and enabled faster iteration and experimentation. The new player leverages HTML5 standards, Encrypted Media Extensions (EME) for playback, and modern CI/CD pipelines for immediate user updates. This evolution has led to improved user engagement metrics and the ability to introduce new features like Daily Mixes and Connect more effectively.

1 post · 2019

Timeline

20192026

Feature trailQuiet since 2018

Bigtable Autoscaler

Spotify developed and open-sourced the Bigtable Autoscaler, a service that automatically adjusts the number of nodes in a Bigtable cluster to optimize for CPU utilization and storage capacity. It supports configurable min/max nodes, CPU targets, and overload steps, and includes features for handling batch jobs and exponential backoff for error handling. Future improvements include predictive scaling using Facebook Prophet.

1 post · 2018

Timeline

20182026

Feature trailQuiet since 2018

Cassandra Orchestration Tool

Spotify's Cassandra orchestration and management capabilities have evolved with the introduction and adoption of the Date-Tiered Compaction Strategy (DTCS). DTCS is optimized for time-series data, a common use case for Cassandra, by aligning SSTable age with data timestamps. This improves read performance by reducing the number of SSTables that need to be scanned for time-based queries. DTCS also aids in data removal by efficiently purging tombstones and TTL-expired data from older SSTables. While DTCS is highly effective for time-series and steady-rate write patterns, it is less suitable for highly out-of-order writes or scenarios with frequent repairs that introduce out-of-order timestamps. The post also emphasizes the importance of consistent timestamp formatting and client clock synchronization.

3 posts · 2014–2018

Timeline

…20172026

Feature trailQuiet since 2018

SVG to Coordinate Conversion Tool

Spotify has developed Coördinator, an open-source browser-based tool that converts SVG files into XY coordinates. This tool facilitates the creation of data visualizations and animations by allowing SVGs to be represented as a series of points. It handles multiple SVG paths and shapes, and provides a visual interface for fine-tuning the output. Coördinator was initially developed to create a data visualization for a live interview, and has since been open-sourced to benefit the wider community. It leverages SVG methods like getTotalLength() and getPointAtLength(), and uses Pathologist to handle non-path SVG elements.

1 post · 2018

Timeline

20182026

Feature trailQuiet since 2018

Microservices Testing Strategy

Spotify's approach to testing microservices has evolved from a traditional testing pyramid to a 'Testing Honeycomb' model. This new strategy emphasizes integration tests, with fewer implementation detail tests and ideally no integrated tests. The goal is to provide confidence, fast feedback, and easier maintenance by focusing on testing services at their interaction points and contracts, treating each microservice as a testable component.

1 post · 2018

Timeline

20182026

Feature trailQuiet since 2017

Cloud Security Automation Platform

Spotify is enhancing its cloud security posture by developing and leveraging automated tools for auditing and enforcing security configurations on Google Cloud Platform. This includes building an inventory of resources, running scanners for misconfigurations, and automatically fixing violations. The goal is to scale security hardening efforts alongside development speed and provide better visibility and audit trails for incident investigation.

2 posts · 2017

Timeline

20172026

Feature trailQuiet since 2017

URL State Management with Redux

Spotify has developed Redux-Location-State (RLS), a Redux middleware that synchronizes specific pieces of application state with the browser's URL. This allows for a unified source of truth where both the Redux store and the URL accurately reflect the application's current state, enabling features like bookmarking, sharing, and using browser navigation (back/forward buttons) seamlessly. RLS supports configurable state mapping, initial state loading, and URL cleanup when state is reset.

1 post · 2017

Timeline

20172026

Feature trailQuiet since 2016

Event Delivery System Migration

Spotify is migrating its event delivery system to the cloud, specifically leveraging Google Cloud managed services like Pub/Sub and DataFlow. This post details the architecture and lessons learned from the existing Kafka-based system, highlighting its limitations in terms of latency, data completeness, and flexibility. The new system aims to address these issues by providing more robust and scalable event transportation and processing.

1 post · 2016

Timeline

…20172026

Feature trailQuiet since 2016

SDN Internet Router

Spotify is evolving its network infrastructure by developing and deploying a Software-Defined Networking (SDN) Internet Router. This system aims to optimize internet routing by analyzing traffic patterns and selectively downloading BGP prefixes, allowing the use of more cost-effective and power-efficient switches instead of traditional, expensive routers. The project involves custom tooling for traffic analysis, BGP control, and vendor collaboration to program network hardware.

2 posts · 2016

Timeline

…20172026

Feature trailQuiet since 2015

Latency Based Load Balancing

Spotify's load balancing strategies have evolved to address the challenges of distributing traffic efficiently across backend services. This post introduces the Expected Latency Selector (ELS) as a probabilistic load balancer that considers success latency, failure latency, and outstanding requests to dynamically adjust traffic distribution. ELS aims to outperform traditional methods like Round-Robin and Join the Shortest Queue (JSQ) by providing a more nuanced approach to machine selection, especially under high load or when machines exhibit varying performance or failure rates. Subsequent posts will detail the ELS formula and further benchmarks.

1 post · 2015

Timeline

…20172026

Feature trailQuiet since 2015

Heroic Time Series Database

Spotify's monitoring capabilities have been significantly enhanced with the introduction of Heroic, a scalable, in-house time series database. Heroic addresses the challenges of near real-time data collection and presentation at scale by leveraging Cassandra for storage and Elasticsearch for indexing. It supports global federation for improved availability and resilience. The system is complemented by the ffwd agent for metrics collection and Kafka for buffering, enabling rapid experimentation and a push-based approach to monitoring.

2 posts · 2015

Timeline

…20172026

Feature trailQuiet since 2015

iOS Client Accessibility Improvements

Spotify's iOS client has seen significant improvements in accessibility, driven by a focus on enhancing the user experience for VoiceOver users. This involved refactoring the internal Ceramic framework to group UI elements for better touch targets and visual hierarchy, and transitioning from UITableViews to UICollectionViews to ensure proper accessibility gesture support. The goal is to integrate accessibility as a permanent quality criterion alongside security and other development standards.

1 post · 2015

Timeline

…20172026

Feature trailQuiet since 2015

IPv6 Client Support

Spotify has been progressively enabling IPv6 support across its client applications to address the growing adoption of IPv6 by mobile carriers and to meet potential App Store requirements. This effort began with the Android client to mitigate connectivity issues caused by IPv4-to-IPv6 translation (CLAT) failures, and has since expanded to the iOS client. The long-term goal is to achieve end-to-end IPv6 support, including backend services.

1 post · 2015

Timeline

…20172026

Feature trailQuiet since 2015

Java Dependency Management

Spotify has developed and open-sourced the MissingLink plugin to help mitigate Java dependency hell issues. This plugin analyzes bytecode to verify that invoked methods exist in the final classpath, aiming to catch runtime errors like NoSuchMethodError and NoClassDefFoundError earlier in the development cycle. While not a complete solution, it aids in troubleshooting and identifying potential issues that integration tests might miss. The post also discusses existing challenges with dependency resolution, the limitations of semantic versioning, and other attempted solutions like the Maven Enforcer Plugin and the Shade Plugin.

1 post · 2015

Timeline

…20172026

Feature trailQuiet since 2015

Integer Underflow Bug in Round-Robin

This post details a specific integer underflow bug discovered in Spotify's Java implementation of the round-robin algorithm. The bug occurred when an AtomicInteger, intended to cycle through connection indices, unexpectedly returned Integer.MIN_VALUE instead of a positive value due to an underflow condition. This led to array index out-of-bounds exceptions and crashes. The post explains integer representation, the specific failure mode of the `index()` function, and provides two fixes: one using `Integer.MIN_VALUE` and another using a bitwise AND with `Integer.MAX_VALUE` to effectively reset the sign bit. It also highlights the bug's long lifespan and the specific conditions required for it to manifest.

1 post · 2015

Timeline

…20172026

Feature trailQuiet since 2015

Git Testing Tooling

Spotify has developed git-test, a tool to automate testing of local commits during development and rebasing. This aims to improve code quality and developer workflow by catching issues early and providing a more robust testing loop than manual methods. The tool caches results based on tree hashes to avoid redundant tests and offers features for capturing test output and filtering commits by file.

1 post · 2015

Timeline

…20172026

Feature trailQuiet since 2015

Apache Storm Pipelines

Spotify has built and scaled multiple real-time pipelines using Apache Storm for various use cases, including ad targeting, music recommendation, and data visualization. These pipelines integrate with systems like Kafka and Cassandra, processing billions of events daily. The team has focused on architectural improvements, maintainability through externalized configuration and metrics visualization, and performance tuning of hardware, throughput, and latency. Key strategies include creating small logical topologies, promoting code reusability, parallelizing I/O, and implementing robust deployment and rollback strategies. In-memory caching with Guava's Expirable Cache has been used to manage state within Bolts. The system has evolved to independent Storm clusters and considers Storm on YARN for future resource utilization.

1 post · 2015

Timeline

…20172026

Feature trailQuiet since 2013

Debian Packaging for Python Applications

Spotify has developed and open-sourced dh-virtualenv, a tool that bridges the gap between Debian packaging and Python's virtual environments. This allows for the creation of Debian packages that bundle Python dependencies within a virtualenv, simplifying deployment and development workflows. It addresses the limitations of outdated Python packages in Debian stable and the steep learning curve of traditional Debian packaging for Python developers. The tool integrates with existing Debian build infrastructure (sbuild) and allows for the definition of both system and Python dependencies, enabling easier deployment of complex Python applications like Sentry.

1 post · 2013

Timeline

…20172026

Feature trailQuiet since 2013

Sparkey Key-Value Store

Spotify has open-sourced Sparkey, a C-based key-value store inspired by CDB and Bitcask. It addresses the limitations of CDB's 4GB file size limit and Tokyo Cabinet's overhead and locking issues. Sparkey offers fast lookups, low overhead for small key-value pairs, block-level compression, and fast creation times. It consists of a log file and an on-disk hash table index. The system has been used internally for over a year on the content metadata lookup service and is being adopted by newer services.

1 post · 2013

Timeline

…20172026

Feature trailQuiet since 2013

Workstation Configuration Management

Spotify has evolved its workstation management by adopting and scaling Puppet for both macOS and Windows. This includes automating software installation, configuration, and security measures across over 1500 nodes. Key advancements include implementing local caching for bandwidth optimization, a self-tagging website for user-driven software requests, and desktop notifications for user experience improvements. The future direction involves puppetizing development environments and exploring masterless setups with tools like Boxen.

1 post · 2013

Timeline

…20172026

Feature trailQuiet since 2013

Username Canonicalization and Security

Spotify's approach to username handling has evolved to address security vulnerabilities arising from complex Unicode characters. The system now ensures that usernames are canonicalized correctly and idempotently, preventing account hijacking through clever use of similar-looking characters. This involves rigorous validation of user input and robust handling of Unicode transformations to maintain system integrity and user security.

1 post · 2013

Timeline

…20172026

Feature trailQuiet since 2013

Mature Technology Adoption and Service Discovery

Spotify leverages mature and proven technologies like DNS (Bind, Unbound) for service discovery (SRV records) and configuration management (TXT records), and PostgreSQL for relational data storage. The company also uses Cassandra for specific use cases where its replication and failure handling characteristics are advantageous. The post highlights the economic and temporal aspects of choosing the 'right tool' and the ongoing evolution of service discovery, with an exploration into Zookeeper for dynamic registries.

1 post · 2013

Timeline

…20172026

Feature trailQuiet since 2013

Kernel Driver Debugging and Patching

Spotify's infrastructure team actively debugs and patches kernel drivers to resolve hardware compatibility issues, ensuring the stability and reliability of their server provisioning and data center operations. This involves deep dives into kernel source code, binary searching for problematic changes, and applying targeted fixes to custom kernels.

1 post · 2013

Timeline

…20172026