TechPrufer
Get started
AI Engineer
AI Engineer logoAI Engineer · Feature trail

Low-Latency Inference Engineering Playbook

11 videos · Sep 2026 – Oct 2026

What this trail covers

This post contributes to the growing body of knowledge around optimizing AI inference for low latency. It details specific techniques, architectural patterns, and tooling strategies employed by leading organizations to achieve faster inference times, crucial for real-time AI applications and agentic systems. This includes optimizations in model loading, execution, caching, and network communication within inference infrastructure, as well as the use of autoresearch for kernel optimization and ba

All AI Engineer trails
  1. How Inference Keeps Up with Our Insatiable Need for Tokens — Cerebras & Gimlet LabsThis talk details techniques for accelerating LLM inference to keep pace with exploding token demand. Daniel Kim from Cerebras covers runtime optimizations including why decoding is memory-bound, strategies for splitting models across GPUs, continuous batching, KV cache management (including paged KV cache), prompt caching, and chunked prefill for long contexts. Michelle from Gimlet Labs discusses system-level optimizations such as prefill/decode disaggregation, speculative decoding, attention/FFN disaggregation, Gimlet's multi-silicon cloud, and the use of AI agents to generate optimized kernels for specific hardware.
  2. Taming the AI Hardware Cambrian Explosion — Abdul Dakkak, ModularThis talk introduces Modular's approach to addressing the AI hardware Cambrian explosion by rebuilding the software stack from the ground up. It presents Mojo, a Pythonic systems language designed for high performance across CPUs, GPUs, and accelerators, aiming to match CUDA's performance with open-sourced kernels that claim to outperform vendor libraries and FlashAttention 4. The talk also introduces MAX, an inference framework that rethinks PyTorch-era assumptions to support thousands of model architectures and run across heterogeneous hardware (multi-GPU, multi-node, mixed hardware). Modular Cloud is presented as a tunable glass box to tie these components together. Benchmarks are shared comparing performance against vLLM and torch.compile on various hardware, demonstrating sub-second image generation and performance on models like Kimi K2.5, FLUX.2, AMD MI355, and Gemma 4.
  3. Same Model, Different Speed: Why Your Inference Provider Matters — FriendliAIThis talk details how to achieve faster and more reliable LLM inference by focusing on the entire inference stack, not just the model. It highlights the importance of open-weight models being 'good enough' and that the differentiator is inference provider quality. Key technical contributions include: prefix caching with cache-aware routing, a tiered cache hierarchy (GPU, CPU, NVMe) for long, repetitive agent inputs, and hybrid speculative decoding that adapts to live traffic. It also emphasizes reliability as a critical component of speed for agentic workloads, citing results of up to 7x faster responses with fewer errors.
  4. Latency Is a Budget. Humanlike Is the Goal. — Jesse Hall, LiveKitThis talk addresses the critical gap between measured latency and perceived latency in voice AI pipelines, arguing that humanlike interaction requires a focus on perceived latency and a 'latency budget.' It highlights how errors compound across pipeline stages (STT, LLM, TTS) and introduces audio-based turn detection for more natural conversation flow. The talk proposes a target of ~600ms for humanlike feel and introduces LiveKit's open-source benchmark and a voice-tuned Gemma model for optimizing these pipelines.
  5. How MiniMax M3 Was Built: Sparse Attention and Native Multimodality — Olive SongThis talk details the architecture and training of MiniMax M3, focusing on its Sparse Attention (MSA) mechanism and native multimodal capabilities. MSA is presented as a novel sparse attention design that uses an index branch to select relevant blocks and a sparse branch to attend only to those, achieving approximately 9x faster prefill and 15x faster decoding than full attention. The talk explains why existing sparse attention designs like DeepSeek's DSA were not suitable for MiniMax's architecture due to grouped-query attention and GPU memory access patterns. It also covers the benefits of training text and vision modalities from step zero, contrasting it with adding vision later, and discusses the use of 3D attention within the vision transformer for improved visual understanding. The implementation details include block-level retrieval and efficient kernels optimized for GPUs.
  7. From 15% to 90% GPU Utilization: Fix the Data Pipeline, Not the ModelThis post details how to significantly improve GPU utilization during multimodal AI model training by addressing data pipeline bottlenecks. It demonstrates a transition from 15% to 90% GPU utilization by optimizing data loading, decoding, and resizing. Key techniques include using asyncio and Ray actors for concurrency, prefetching data to ensure it's ready for the trainer, and employing Ray's object store for zero-copy tensor transfers. At scale, a network card bottleneck was resolved by distributing workers across nodes and implementing zero-copy reads, leading to a 50% throughput increase. The talk highlights that multimodal training is often CPU- and IO-bound, not GPU-bound, and discusses the trade-offs between asyncio, threads, and processes around the GIL.
  8. What Is an Inference Engine, Anyway? — Charles Frye, ModalThis talk provides a deep dive into the inner workings of inference engines, explaining their architecture and the engineering challenges involved in achieving low latency and high throughput. It details the request lifecycle from server IO to tokenization, scheduling, model execution, and detokenization, highlighting bottlenecks like the scheduler. Performance techniques such as KV caching, CUDA graphs, and speculative decoding are explained. The talk also emphasizes the importance of observability, logging, and metrics for debugging and production readiness, using SGLang and vLLM as examples.
  9. Research to Reality with Google DeepMind — Paige Bailey, Google DeepMindThis talk introduces Google DeepMind's Gemma 4 open model family, emphasizing local inference capabilities for both browser-based applications and mobile devices (Android/iOS) via Google AI Edge Gallery. It highlights the practical benefits of smaller model sizes (e.g., 2B parameters fitting under 1GB) and quantization for achieving fast, on-device inference without API calls. The presentation also touches upon responsible AI, managed agents in Linux sandboxes, and the broader research mission of DeepMind in areas like AlphaFold and robotics, underscoring the trend towards decentralized and efficient AI deployment.
  10. What Makes Open Models Fast in Production — Sujee Maniyam, NebiusThis talk details the engineering challenges and solutions for serving open LLMs in production with low latency and cost. It covers hardware optimizations (NVFP4), engine selection, cache-aware routing for LLMs, speculative decoding with custom draft models, KV cache offloading, disaggregated prefill and decode, and finding the optimal quantization sweet spot. The discussion highlights the trade-offs between closed APIs and self-hosting, and the full loop of inference, data, post-training, and deployment.
  11. Autoresearch Made Our Models 3x Faster — Tejas Bhakta, MorphThis post details how autoresearch, combined with human-driven ideas and bare-metal hardware tweaks, can make AI models up to 3x faster. It highlights the effectiveness of autoresearch for optimizing GPU kernels, providing agents with hardware and model context, and the importance of identifying and avoiding reward hacking. The talk also emphasizes that kernel gains compound and that bare-metal optimizations can add significant performance over cloud VMs, while acknowledging that a high percentage of autoresearch attempts are unsuccessful.
  12. AI Engineer Paris 2026 Main Stage: Google DeepMind, ElevenLabs, Hugging Face & Stripe | Day 2Charles Frye's talk, 'The Low-Latency Inference Engineering Playbook,' directly addresses the engineering challenges of achieving minimal latency in AI inference. This contribution is expected to provide practical, actionable strategies and insights into the methodologies and technologies used to optimize inference speed, likely covering aspects such as efficient model loading (potentially building on concepts like vLLM's loading mechanisms), optimized execution paths, and infrastructure design tailored for rapid response times.