TechPrufer
Get started
Spotify
Spotify logoSpotify · Feature trail

Data Lake Indexing for Point Queries

9 posts · Apr 2013 – Jul 2026

What this trail covers

Spotify's data platform continues to evolve with Random Access Parquet (RAP), an external index that maps keys directly to file locations in the data lake, enabling fast point queries by key over datasets too large for traditional KV stores. This bridges the gap between analytical throughput engines and interactive online use cases, including AI agents. RAP operates on existing Parquet files, reducing query latency and bandwidth by replacing dependent reads with precomputed lookups. Optimization efforts are ongoing. The AI data assistant, Vedder, leverages domain-specific clusters (datasets, pairs, docs) curated by domain experts to provide trustworthy answers to natural language queries. Vedder follows a ReAct loop, generates SQL, and returns answers with sources. Cluster health is monitored continuously, and user conversations feed back into the system for ongoing improvement. Future explorations include knowledge outside of schemas, such as documentation and process definitions.

All Spotify trails
  1. Indexing the Data Lake for Online Point Queries | Spotify EngineeringThis post introduces Random Access Parquet (RAP), an external indexing technique for data lakes that enables fast point queries by mapping keys directly to file and row locations. It details how RAP overcomes the latency and overhead of distributed SQL engines for online use cases by replacing scan-based lookups with precomputed index lookups. The post elaborates on the structure of the external index, its multimap nature, and its compact entry format. It also discusses optimizations for prepared Parquet files, including concentrating key data (sorting, hash bucketing, co-grouping, coarser partitioning), reducing bytes read (one page per key, ZSTD frame resets within pages, storage alignment), and reducing read operations (blobs, variants, interleaving columns).
  2. Encoding Your Domain Expert: The Context Layer Behind Spotify's Data Assistant | Spotify EngineeringThis post introduces Vedder, an AI data assistant that uses domain-specific 'clusters' (datasets, question-SQL pairs, and documentation) curated by domain experts to provide trustworthy answers to natural language queries against Spotify's data warehouse. It details the ReAct loop used by the agent, the structure of these clusters, the importance of human judgment in curating question-SQL pairs (highlighting that only 12.5% of historical query pairs were deemed suitable), and the continuous monitoring of cluster health via a health score based on data freshness, pair validity, question coverage, and SQL reproducibility. The system logs all conversations and feedback to improve future responses. The architecture is presented as transferable beyond Spotify.
  3. How Spotify Optimized the Largest Dataflow Job Ever for Wrapped 2020This post details the design and implementation of Sort Merge Bucket (SMB) joins within Spotify's data pipelines, specifically for optimizing the Wrapped 2020 Dataflow job. It introduces the SortedBucketSink and SortedBucketSource transforms, explains the role of FileOperations and BucketMetadata, and discusses optimizations such as date partitioning, sharding, and parallelism strategies for joining sources with different bucket counts. It also highlights the SortedBucketTransform for efficient read-transform-write operations and improvements to Beam's external sorter. The post details the adoption of SMB by core data producers and its application in Wrapped 2020.
  4. It’s All Just Wiggly Air: Building Infrastructure to Support Audio ResearchThis post introduces Klio, a new framework for building smarter data pipelines for audio and other media processing. Klio is built on Apache Beam for Python, aiming to provide a more ergonomic and Python-native experience. Key features include streamlined Apache Beam usage, an open graph of job dependencies, integration with cloud processing engines, containerization of custom dependencies, and support for both batch and streaming pipelines. The post highlights Klio's efficiency gains, demonstrated by a six-day catalog downsampling task that previously took weeks, and a four-fold cost reduction.
  5. How We Improved Data Discovery for Data Scientists at SpotifyThis post details the iteration on Spotify's Lexikon platform to improve data discovery for data scientists. It describes the diagnosis of the problem, which stemmed from an explosion of dataset creation in BigQuery and increased hiring of insights specialists, leading to localized impact and difficulty finding datasets. The post outlines two primary areas of improvement: understanding user intent for both low-intent (personalized recommendations on the homepage) and high-intent (popularity-weighted search ranking, new entity pages like schema fields and people) data discovery. It also discusses enabling knowledge exchange through people by mapping expertise and providing supplemental information in collaboration tools, moving beyond a purely codification strategy.
  7. Introducing Chartify: Easier chart creation in Python for data scientistsIntroduced Chartify, an open-source Python library that wraps Bokeh to simplify chart creation for data scientists. Addresses pain points in data visualization workflows by offering a more intuitive and concise syntax for common tasks like axis tick formatting and data manipulation for different chart types. Focuses on the 80% of common chart use cases and provides guidance for users.
  8. TC4D: Data Quality By Engineers, For EngineersThis post introduces TC4D (Test Certified for Data-Focused teams), a gamified program designed to improve testing practices and test coverage for data-focused teams at Spotify. It addresses the lack of standardization in big data quality and the inconsistent test coverage across pipelines. TC4D consists of three certification levels focusing on setup, monitoring, alerting, unit tests, security, and data quality. The program is driven by mentors and aims to encourage adoption of best practices through encouragement and trust in engineers' decision-making.
  9. Solving MapReduce Performance Problems With Sharded JoinsThis post details the implementation of a sharded join strategy in Apache Crunch to address performance problems in a MapReduce pipeline caused by skewed data. When a standard join resulted in overloaded reducers and high memory usage, a sharded join was introduced. This involved splitting the key space of the larger dataset and replicating the smaller dataset to distribute the join load across more reducers. The post provides code examples for implementing this strategy from scratch when the built-in Crunch library had a bug. The solution successfully reduced memory usage by over 75% and revived the pipeline.
  10. Snakebite: a pure Python HDFS clientThis post introduces Snakebite, a pure Python HDFS client designed to overcome the slow interaction speeds of the native Hadoop HDFS client. Snakebite communicates with HDFS using Protocol Buffers, bypassing the need to start a JVM for each operation. This results in significantly faster operations, as demonstrated by a benchmark showing a 10x speed improvement for `ls /` commands. The Snakebite project includes a Python library, a command-line client, and a mini-cluster wrapper for integration testing. Currently, it supports NameNode-only operations, with plans to implement DataNode interactions in the future.