Indexing the Data Lake for Online Point Queries | Spotify Engineering

This post introduces Random Access Parquet (RAP), an external indexing technique for data lakes that enables fast point queries by mapping keys directly to file and row locations. It details how RAP overcomes the latency and overhead of distributed SQL engines for online use cases by replacing scan-based lookups with precomputed index lookups. The post elaborates on the structure of the external index, its multimap nature, and its compact entry format. It also discusses optimizations for prepared Parquet files, including concentrating key data (sorting, hash bucketing, co-grouping, coarser partitioning), reducing bytes read (one page per key, ZSTD frame resets within pages, storage alignment), and reducing read operations (blobs, variants, interleaving columns).