How We Generated Millions of Content Annotations
Oct 1, 2024Dana Puleo (Annotation Platform Ops Manager), Meghana Seetharam (Staff Engineer), and Katarzyna Drzyzga (Staff Engineer)
Summary
This post details the development of a scalable annotation platform to support ML and GenAI use cases. It outlines a three-pillar strategy: scaling human expertise (core annotators, quality analysts, project managers), implementing annotation tooling capabilities (custom interfaces, backend management, project metrics, agreement metrics), and establishing foundational infrastructure and integration (flexible data models, APIs, CLIs, UIs, batch orchestration integration). The platform enabled a 10x increase in annotations and a 3x improvement in annotator productivity.
More from the AI-Assisted Developer Productivity Platform trail
- Why Spotify Is Not Using Bayesian A/B Testing | Spotify Engineering
- Portal by Spotify cut my Claude Code token usage by 90% | Spotify Engineering
- AI Changed How Spotify Builds. What We Learned (and Fixed) About Quality at Higher Velocity
- When Can LLMs Replace Humans in A/B Tests? | Spotify Engineering
- Coding Is No Longer the Constraint: Scaling Developer Experience to Teams and Agents at Spotify | Spotify Engineering