Kimi K3 vs GPT-5.6 Sol on DeepSWE: Cost, Coding, and Routing

What this post added

This post provides a detailed comparative analysis of Kimi K3 and GPT-5.6 Sol on the DeepSWE benchmark. It quantifies performance differences in terms of pass@1, pass@k, coverage, and reliability, and analyzes cost per rollout and per solved task. Crucially, it explores the divergence in failure modes and task domain strengths between the two models, proposing and evaluating a routing strategy (Kimi K3 cascade to GPT-5.6 Sol on failure) that leverages this divergence to achieve a higher overall accuracy (85.6%) and task coverage (95.6%) than either model alone, while also optimizing cost. It also details performance by programming language and task type.