How Inference Keeps Up with Our Insatiable Need for Tokens — Cerebras & Gimlet Labs

This talk details techniques for accelerating LLM inference to keep pace with exploding token demand. Daniel Kim from Cerebras covers runtime optimizations including why decoding is memory-bound, strategies for splitting models across GPUs, continuous batching, KV cache management (including paged KV cache), prompt caching, and chunked prefill for long contexts. Michelle from Gimlet Labs discusses system-level optimizations such as prefill/decode disaggregation, speculative decoding, attention/FFN disaggregation, Gimlet's multi-silicon cloud, and the use of AI agents to generate optimized kernels for specific hardware.