Introducing our Speculative Decoding Engine Builder integration for ultra-low-latency LLM inference

Introduces a Speculative Decoding Integration for the TensorRT-LLM Engine Builder, enabling ultra-low-latency LLM inference. This feature combines a larger target LLM with a smaller 'draft' model to predict tokens more efficiently. The integration provides pre-optimized configurations and allows users to tune parameters like the number of draft tokens and GPU memory allocation. It is particularly beneficial for low-batch, latency-sensitive applications such as code generation, where it has demonstrated halved latencies without compromising output quality. The post highlights the challenges of configuring speculative decoding from scratch and Baseten's two-tiered approach to balance ease of use with control. It also notes scenarios where speculative decoding might not be ideal, such as under very high load or with already lightweight LLMs.