Keeping Developers Happy with a Fast CI - Shopify

What this post added

This post details improvements to Shopify's CI test selection process. It expands the existing test selection system to include mappings for non-Ruby files, specifically ActiveRecord fixtures, by subscribing to ActiveSupport notifications for SQL queries. This increases the percentage of builds that don't run the full test suite from 45% to over 60% and improves test stability from 88% to 97%. The post also discusses strategies for handling slow or problematic tests, including temporary disabling, which reduced the p95 CI build time by 10 minutes. Additionally, it highlights improvements in agent preparation by addressing Docker start time bottlenecks related to I/O, reducing p95 Docker start time from 90 seconds to 25 seconds, and optimizing dependency building by avoiding unnecessary runs through hash-based caching for database migrations and asset compilation, reducing that job's time from 5 minutes to 3 minutes.