Same Model, Different Speed: Why Your Inference Provider Matters — FriendliAI

This talk details how to achieve faster and more reliable LLM inference by focusing on the entire inference stack, not just the model. It highlights the importance of open-weight models being 'good enough' and that the differentiator is inference provider quality. Key technical contributions include: prefix caching with cache-aware routing, a tiered cache hierarchy (GPU, CPU, NVMe) for long, repetitive agent inputs, and hybrid speculative decoding that adapts to live traffic. It also emphasizes reliability as a critical component of speed for agentic workloads, citing results of up to 7x faster responses with fewer errors.