Encoding Your Domain Expert: The Context Layer Behind Spotify's Data Assistant | Spotify Engineering

This post introduces Vedder, an AI data assistant that uses domain-specific 'clusters' (datasets, question-SQL pairs, and documentation) curated by domain experts to provide trustworthy answers to natural language queries against Spotify's data warehouse. It details the ReAct loop used by the agent, the structure of these clusters, the importance of human judgment in curating question-SQL pairs (highlighting that only 12.5% of historical query pairs were deemed suitable), and the continuous monitoring of cluster health via a health score based on data freshness, pair validity, question coverage, and SQL reproducibility. The system logs all conversations and feedback to improve future responses. The architecture is presented as transferable beyond Spotify.