Quantifying infrastructure noise in agentic coding evals

What this post added

This post introduces the critical issue of 'infrastructure noise' in agentic coding evaluations. It quantifies how variations in resource allocation (CPU, RAM) and enforcement methodologies (strict limits vs. headroom) can significantly impact benchmark scores, sometimes by as much as 6 percentage points. The research demonstrates that beyond a certain threshold (around 3x recommended specs), increased resources actively help agents solve problems, not just avoid infrastructure failures. It also touches upon other confounding factors like time limits and cluster health. The post proposes practical recommendations for benchmark maintainers and users, including specifying both guaranteed allocations and hard kill thresholds for containers, and treating resource configuration as a documented experimental variable to improve the reliability and interpretability of agentic evaluation results.