Claude SWE-Bench Performance

This post details the development of an agentic system around Claude 3.5 Sonnet to achieve high performance on the SWE-bench benchmark. It explains the agent's minimal scaffolding, consisting of a Bash Tool and an Edit Tool, and the design philosophy of giving the language model maximum control. Specific technical contributions include detailed tool descriptions designed to preempt model misunderstandings, error-proofing mechanisms for tools (e.g., requiring absolute paths for the Edit Tool), and a string replacement strategy for file edits with exact match requirements. The post also discusses challenges encountered, such as duration/cost, grading accuracy, hidden tests, and the limitations of current multimodal integration for debugging.