Measuring LLMs’ ability to develop exploits
May 22, 2026
Summary
This post details the evaluation of Mythos Preview's exploit development capabilities using new academic benchmarks (ExploitBench, ExploitGym) and an updated internal benchmark (SCONE-bench). It quantifies Mythos Preview's performance in finding vulnerabilities, developing exploit primitives, and achieving arbitrary code execution, demonstrating a significant leap over previous models. The post also highlights the technical details of ExploitBench's tiered capability assessment and ExploitGym's broad target set evaluation.
More from the AI Code Generation and Software Engineering Agents trail
- Using Claude Science to produce the first complete map of the sky in UV light
- Project Swap: What happens when agents trade for us?
- Automated researchers can reliably mitigate alignment failures
- How Claude is accelerating protein design and analytical chemistry
- Patterns and problems in multiagent systems