Indexing the Data Lake for Online Point Queries | Spotify Engineering

What this post added

Introduced Random Access Parquet (RAP), an external indexing system for data lakes. RAP uses an external index to map keys to specific file and row locations, enabling direct, precise reads. Discussed optimizations for prepared Parquet files, including concentrating key data (sorting, hash bucketing, co-grouping, coarser partitioning), reducing bytes read (one page per key, ZSTD frame resets, storage alignment), and reducing read operations (blobs/variants, interleaving columns).