AI Lab: The secrets to keeping machine learning engineers moving fast

This post introduces AI Lab, a pre-production framework at Meta designed to accelerate ML engineer velocity by focusing on Time to First Batch (TTFB). AI Lab continuously A/B tests common ML workflows to proactively improve TTFB and prevent regressions. It addresses challenges like precious GPU capacity by using shrunk models and CPU-only configurations for testing. The post details how AI Lab facilitated a 2x increase in TTFB improvements from the Python Cinder runtime and how it automatically detected and prevented a TTFB regression caused by asynchronous logging. AI Lab integrates into the development cycle at O(Code Changes) and O(Releases) stages, using t-tests and ROC curves to confirm statistically significant changes before marking them as regressions or improvements.