How SWE-Serve Exposes the Gap Between Local Tests and Live Serving | NVIDIA Technical Blog

This post introduces SWE-Serve, a new benchmark for evaluating AI coding agents on inference-engineering tasks. It highlights the critical gap between local testing and live serving of AI models by demonstrating that a substantial percentage of patches passing other tests fail when validated through a live serving interface. The benchmark comprises 53 tasks derived from 83 merged SGLang pull requests, covering model enablement, decoding, caching, scheduling, serving APIs, and distributed execution. SWE-Serve reveals that tasks spanning multiple runtime domains (request handling, scheduling, model execution, KV-cache management) have a significantly lower pass rate compared to single-domain tasks. The post also presents performance data for 11 AI models across these tasks, showing wide variations in pass@1 scores, cost, and wall time, and notes that no single model excels across all task categories.