Shopify's Path to a Faster Trino Query Execution: Custom Verification, Benchmarking, and Profiling Tooling - Shopify

This post details the development of a custom Python-based framework for verifying, benchmarking, and profiling Trino query execution. The framework includes a core library for Trino query orchestration and interfaces for PyTest. For verification, it enables running queries on candidate clusters and comparing results against control clusters, with marks for different test types (correctness, performance). For benchmarking, it utilizes TPC-DS queries across various scale factors to evaluate performance under standardized conditions. The tooling aims to reduce toil by formalizing undocumented upgrade checklists and providing a structured approach to testing changes in a complex, forked Trino environment.