Finding bugs with Claude and property-based testing

This post details the development of an AI agent that autonomously writes property-based tests to discover bugs in software projects. The agent infers general code properties by reading type annotations, docstrings, function names, and comments, then generates property-based tests using Hypothesis. It incorporates a self-reflection loop to refine tests and reduce false positives, and has successfully identified hundreds of potential bugs in popular Python packages like NumPy, SciPy, and Pandas, with several already patched. The agent's workflow includes understanding the target code, proposing properties, writing tests, running tests, self-reflecting, and generating bug reports. The post also discusses the evaluation methodology, including manual review and a rubric for ranking bug reports, and highlights specific bug examples found in NumPy, Powertools Lambda, CloudFormation CLI, and Hugging Face tokenizers.