Inside VAKRA: Reasoning, Tool Use, and Failure Modes of Agents

This post introduces VAKRA, a new benchmark for evaluating AI agents' reasoning and tool use in enterprise-like environments. It details the benchmark's structure, including task descriptions (API chaining, tool selection, multi-hop reasoning), evaluation framework, and analysis of observed failure modes. The benchmark comprises over 8,000+ locally hosted APIs across 62 domains, with tasks requiring 3-7 step reasoning chains. It highlights specific tools and API collections used (SLOT-BIRD, SEL-BIRD, REST-BIRD) and discusses challenges like API list length restrictions for agents.