Claude SWE-Bench Performance

This post details the development of an agentic system around Claude 3.5 Sonnet to achieve high performance on the SWE-bench benchmark. It explains the agent's design philosophy of minimal scaffolding and maximal model control, featuring a Bash Tool for command execution and an Edit Tool for file manipulation. The post highlights specific improvements in tool descriptions and error-proofing, such as requiring absolute paths for the Edit Tool and using exact string replacement for edits. It also discusses challenges encountered, including long runtimes, high token costs, grading complexities, and the limitations of current multimodal integration for debugging.