Is it agentic enough? Benchmarking open models on your own tooling

This post introduces a benchmark harness for evaluating agentic use of command-line tools, with a specific focus on the `transformers` library. It details a methodology for measuring the effort (tokens, time, errors) agents expend to complete tasks, beyond just the final answer's correctness. The harness supports different tiers of tool access (bare, clone, skill) and introduces scoring metrics like 'match %', median time, median tokens, error rates, and 'marker adoption' to assess agent performance and guide tool optimization for agentic workflows. The implementation leverages Hugging Face Jobs for parallel execution and a Hugging Face Bucket for results storage.