What's New in Inference Engineering — Philip Kiely, Baseten

Philip Kiely's talk introduces several key advancements in inference engineering. It highlights the trade-offs of quantization (4-bit weights vs. compressed cache) for data center vs. local inference. For caching, it discusses learned bottlenecks that compact KV cache entries. The most significant contribution is in speculative inference, detailing the evolution from small draft models to trained drafters, the introduction of diffusion drafters (DFlash) for proposing multiple tokens, and the impact of continuous speculator retraining (DSpark) on acceptance rates.