Benchmarking inference at scale: coding agents

What this post added

This post introduces a detailed benchmark for a production coding agent workload, comparing Together Inference Engine against TensorRT-LLM and SGLang. It highlights the engine's superior performance in TPS and TTFT under high concurrency and long context, attributing these gains to full-stack optimization including ThunderMLA (a fused multi-head latent attention kernel) and custom kernel rewrites. The benchmark methodology emphasizes realistic production conditions, such as concurrent requests, long prompts, and prefill-heavy output shapes, and discusses the degradation curves of different engines under load. It also presents cost savings achieved with their models compared to alternatives like Claude Opus.