TechPrufer
Sign inGet started

Blogs›Spotify›Data Lake Indexing for Point Queries

Data Lake Indexing for Point Queries

Data Lake Indexing for Point Queries

2
posts
2026

Spotify has developed Random Access Parquet (RAP), an external indexing system to enable fast point queries on large datasets stored in data lakes (like GCS). RAP maps keys directly to file locations and row numbers, allowing precise ranged reads of only the necessary data. This bypasses the overhead of distributed SQL engines for interactive use cases. Optimizations include sorting by key, co-grouping, one page per key, ZSTD frame resets within pages, storage alignment, and storing related fiel. This post details the development of an AI data assistant that leverages a "context layer" built from curated datasets, vetted question-SQL pairs, and business documentation to provide reliable data insights. This layer, organized into "clusters" owned by domain experts, enhances the LLM's ability to generate accurate SQL queries by providing crucial semantic and contextual information beyond raw schemas. The system also incorporates health monitoring for clusters and feedback loops from user interactions to maintain accuracy and relevance.

2026

Indexing the Data Lake for Online Point Queries | Spotify Engineering

7/27/2026

Introduced Random Access Parquet (RAP), an external indexing system for data lakes. RAP uses an external index to map keys to specific file and row locations, enabling direct, precise reads. Discussed optimizations for prepared Parquet files, including concentrating key data (sorting, hash bucketing, co-grouping, coarser partitioning), reducing bytes read (one page per key, ZSTD frame resets, storage alignment), and reducing read operations (blobs/variants, interleaving columns).

Encoding Your Domain Expert: The Context Layer Behind Spotify's Data Assistant | Spotify Engineering

6/10/2026

This post introduces the "context layer" for Spotify's AI data assistant, "Vedder." This layer comprises "clusters" owned by domain experts, containing datasets (schemas and profiling), vetted question-SQL "pairs," and additional business "docs." It details the ReAct loop used by the agent, the importance of human judgment in curating training data (only 12.5% of historical queries were deemed good examples), and a "health score" for clusters based on data freshness, schema changes, and query reproducibility. The system also incorporates user feedback to refine the context.