How we compare model quality in Cursor

Introduced CursorBench, an internal evaluation suite for measuring AI agent performance in coding tasks. CursorBench sources tasks from actual Cursor usage via Cursor Blame, reducing contamination risk and aligning with real developer outcomes. It measures multiple dimensions of agent performance including solution correctness, code quality, efficiency, and interaction behavior. The suite has evolved to incorporate more challenging tasks, doubling its problem scope in terms of lines of code and mean number of files. CursorBench is designed to provide greater separation between models at frontier levels compared to public benchmarks and its rankings align with online evaluation metrics that measure developer experience. The post also discusses the limitations of public benchmarks and the importance of a hybrid online-offline evaluation process.