Benchmarking inference at scale: coding agents

This post introduces a new benchmark for coding agent workloads, emphasizing production-scale inference challenges like high concurrency, long context, and TTFT sensitivity. It details the engineering optimizations made to the Together Inference Engine, including ThunderMLA (a fused megakernel for MLA architecture) and custom kernel rewrites, which resulted in 31% more TPS and 2x better TTFT at saturation compared to TensorRT-LLM on the same hardware. The benchmark methodology and results are presented, highlighting the degradation curves of different engines under load. It also touches on cost savings and quality comparisons with other models.