Encoding Your Domain Expert: The Context Layer Behind Spotify's Data Assistant | Spotify Engineering

What this post added

This post introduces the "context layer" for Spotify's AI data assistant, "Vedder." This layer comprises "clusters" owned by domain experts, containing datasets (schemas and profiling), vetted question-SQL "pairs," and additional business "docs." It details the ReAct loop used by the agent, the importance of human judgment in curating training data (only 12.5% of historical queries were deemed good examples), and a "health score" for clusters based on data freshness, schema changes, and query reproducibility. The system also incorporates user feedback to refine the context.