What Is an Inference Engine, Anyway? — Charles Frye, Modal

This talk provides a deep dive into the inner workings of inference engines, explaining their architecture and the engineering challenges involved in achieving low latency and high throughput. It details the request lifecycle from server IO to tokenization, scheduling, model execution, and detokenization, highlighting bottlenecks like the scheduler. Performance techniques such as KV caching, CUDA graphs, and speculative decoding are explained. The talk also emphasizes the importance of observability, logging, and metrics for debugging and production readiness, using SGLang and vLLM as examples.