Quantifying infrastructure noise in agentic coding evals

This post quantifies the impact of infrastructure noise, specifically resource configuration, on agentic coding evaluation benchmarks like Terminal-Bench and SWE-bench. It demonstrates that differences in CPU and RAM allocation can lead to score discrepancies exceeding typical model capability margins (up to 6 percentage points on Terminal-Bench). The core finding is that container runtime enforcement (guaranteed allocation vs. hard kill limit) significantly affects success rates by either causing spurious failures due to transient spikes or enabling agents to utilize more resource-intensive but successful strategies. The post proposes that resource configuration should be treated as a first-class experimental variable, recommending the specification of both guaranteed allocation and hard kill thresholds, and standardized reporting of these parameters to ensure reproducible and interpretable benchmark results.