Shopify's Path to a Faster Trino Query Execution: Infrastructure - Shopify

What this post added

This post details the optimization of Shopify's Trino query execution infrastructure by addressing issues related to JVM settings, CPU and memory allocation, and workload-specific cluster configurations. This includes implementing workload-specific clusters for ad-hoc, scheduled, and experimental queries, reducing coordinator lock contention by setting a hard concurrency limit, and tuning JVM settings and resource allocation to improve cluster throughput and query latency.