How to serve trillions of tokens for trillion-parameter coding agents | Modal Blog

This post details Modal's optimizations for serving trillion-parameter coding agents, focusing on achieving high throughput and interactivity for LLM inference. Key contributions include: optimizing inference performance by 2.8x per user and 5.6x across users on the replica; detailing the workload structure of coding agents, which involves long input sequences relative to output sequences due to iterative session history accumulation; and discussing the use of Moonshot AI's Kimi K2.6 model, its trillion parameters, and the NVFP4 micro-scaling format for hardware acceleration on Blackwell GPUs. The post also highlights upstreamed patches to the SGLang inference engine to improve prefill and decode performance, and discusses the trade-offs between interactivity and throughput in serving such models at scale.