Kimi K3 vs GPT-5.6 Sol on DeepSWE: Cost, Coding, and Routing

This post provides a detailed technical comparison of Kimi K3 and GPT-5.6 Sol on the DeepSWE benchmark, focusing on pass@1, pass@k, cost per rollout, and task coverage. It quantifies the performance differences, showing Kimi K3's advantage in pass@k and cost-effectiveness (2.8x more solved tasks per dollar) despite GPT-5.6 Sol's lead in pass@1. The analysis reveals a 0.46 correlation in task success/failure between the models, making them ideal for a cascaded routing strategy. The post demonstrates that a Kimi-first cascade escalating to Sol can achieve 85.6% accuracy, surpassing either model individually and a perfect one-shot router. It also breaks down performance by programming language and task type, and analyzes failure modes, noting Kimi K3's tendency for near misses and Sol's higher rate of breaking baseline tests. The post concludes by emphasizing how Together AI's inference stack can serve open models like Kimi K3 at production scale to enable such routing strategies.