Kimi K2 Thinking at 140+ TPS on NVIDIA Blackwell

This post details the technical work to achieve state-of-the-art performance for Kimi K2 Thinking on NVIDIA Blackwell GPUs. Key contributions include: 1. Converting INT4 quantized weights to NVFP4 by de-quantizing to BF16 and then re-quantizing using NVIDIA TensorRT Model Optimizer, addressing compatibility with Blackwell GPUs. 2. Configuring a mixed Tensor Parallelism (TP) and Expert Parallelism (EP) strategy within the inference engine to effectively utilize an 8xB200 node, balancing latency and throughput. 3. Implementing KV-aware request routing to maximize KV cache re-use, significantly reducing time to first token (TTFT) for use cases like agentic tool use and code generation.