DeepsecBench: evaluating model performance in finding cybersecurity vulnerabilities

Introduced DeepsecBench, a benchmark for evaluating AI model performance in finding cybersecurity vulnerabilities. The benchmark uses a secret codebase with a golden set of 231 human-judged findings, running each model three times and publishing median results. It provides recall, precision, cost, and total time metrics, combining recall and precision into a single score weighted by F2. The post details model performance, cost-effectiveness, and time efficiency, and discusses strategies for building scanning programs using multiple models. It also highlights the role of Vercel's AI Gateway in managing diverse model access for security scans.