What Makes Open Models Fast in Production — Sujee Maniyam, Nebius
Oct 4, 2026AI Engineer
Summary
This talk details the engineering challenges and solutions for serving open LLMs in production with low latency and cost. It covers hardware optimizations (NVFP4), engine selection, cache-aware routing for LLMs, speculative decoding with custom draft models, KV cache offloading, disaggregated prefill and decode, and finding the optimal quantization sweet spot. The discussion highlights the trade-offs between closed APIs and self-hosting, and the full loop of inference, data, post-training, and deployment.
More from the Low-Latency Inference Engineering Playbook trail
- How Inference Keeps Up with Our Insatiable Need for Tokens — Cerebras & Gimlet Labs
- Taming the AI Hardware Cambrian Explosion — Abdul Dakkak, Modular
- Same Model, Different Speed: Why Your Inference Provider Matters — FriendliAI
- Latency Is a Budget. Humanlike Is the Goal. — Jesse Hall, LiveKit
- How MiniMax M3 Was Built: Sparse Attention and Native Multimodality — Olive Song