The Agent Said It Was Done. The Database Disagreed.

This post introduces the ThinkingBox benchmark and grading system for evaluating AI agent reliability and consistency. It highlights the critical gap between an agent's reported success (tool calls, final responses) and the actual terminal backend state and side effects it leaves behind. The benchmark runs 507 stateful business workflows 20 times each, measuring pass@1, pass@20, and observed 20/20. Key findings include that many agents fail executable checks despite clean tool calls and no reported errors, with significant discrepancies in field values and unintended side effects. The post details the cost of consistency and provides instructions on how to run the benchmark using OpenEnv and Typesense, and discusses the implications for agent development and deployment.