Fireworks AI

This post introduces and quantifies the 'Agent Execution Tax,' defining it as the ratio of wasted inference to productive inference in agentic AI systems. It presents a benchmark of 720 browser automation tasks across three LLMs (GLM-5, MiniMax M2.5, Kimi K2.5) to measure structured output reliability and step efficiency. Key findings include that Kimi K2.5 had zero parse retries, MiniMax M2.5 offered the best value per successful task due to low retry rates, and GLM-5 provided the highest accuracy but at a higher cost. The post argues that inference infrastructure significantly impacts execution tax through structured output consistency and latency predictability, and proposes 'Reliability-Adjusted Accuracy' as a critical procurement metric. It details the benchmark setup, models used, and a per-site analysis of success and failure modes.