DeepsecBench: evaluating model performance in finding cybersecurity vulnerabilities

What this post added

This post introduces DeepsecBench, a new benchmark for evaluating AI models' ability to find cybersecurity vulnerabilities. It details the benchmark's methodology, including the use of a golden set of findings, scoring based on recall-weighted F2, and measures to prevent models from memorizing solutions. The post presents performance data for various AI models, analyzing their scores against cost and time, and discusses strategies for building multi-model scanning programs. It also highlights the role of Vercel's AI Gateway in simplifying the execution of these scans.