How we built production-ready speculative decoding with TensorRT-LLM

This post details the productionization of speculative decoding with TensorRT-LLM, focusing on reducing latency for code generation tasks. Key contributions include addressing inefficient batching by implementing a custom C++-based GRPC server and fixing TensorRT-LLM's request scheduler to account for KV cache re-use and chunked prefill. It also details synchronized draft-target model execution to avoid resource contention and improve TTFT, and provides patches for TensorRT-LLM bugs related to batching and draft token decoding. The post highlights the importance of these optimizations for achieving reliable speed improvements in production LLM inference.