Scaling Like a Boss with Presto

This post details Grab's transition from Amazon Redshift to a data lake architecture using Amazon S3 and Presto for analytics. Faced with scaling challenges due to increasing user numbers and peaky workloads on Redshift, Grab built a data lake using S3, storing data in AVRO format partitioned by timestamp. Spark was used for data pipelines and transformations. To address the limitations of Spark and Zeppelin for SQL-based users, Presto was introduced as the SQL engine over the data lake. The post discusses the challenges of AVRO format incompatibility with Presto, leading to a switch to Parquet. Performance benchmarks comparing Presto against Redshift are presented, highlighting Presto's strengths in partitioned data queries and its ability to support a shared-data multi-cluster architecture. The migration of SQL workloads from Redshift to Presto involved building custom UDFs and learning optimized query writing for Presto.