How Databricks rolls out frontier models to 12,000 employees on Day 1

This post details the engineering challenges and solutions for rolling out frontier AI models to 12,000 employees on Day 1. It addresses the discrepancy between marketed 'frontier' models and actual performance/cost by implementing a tiered rollout strategy based on model performance and cost metrics. The system uses a combination of internal benchmarking, cost analysis, and a phased rollout to ensure employees have access to the best available models without incurring excessive costs or encountering underperforming models. Key technical decisions include establishing a model evaluation framework, defining cost thresholds, and implementing a mechanism to dynamically switch between models based on these criteria.