Developing GitLab Duo: How we validate and test AI models at scale

What this post added

This post details the engineering process for validating and testing AI models at scale for GitLab Duo features. It outlines a four-step process: creating a prompt library as a proxy for production, baselining model performance using metrics like Cosine Similarity Score and LLM Judge, iterative feature development with daily re-validation, and a cycle of experimentation using smaller, focused datasets before broader validation. The post emphasizes the importance of data-driven insights for improving AI feature performance and mitigating risks.