Building the foundation for running extra-large language models
Apr 16, 2026Michelle Chen, Kevin Flansburg, Vlad Krasnov
Summary
This post details the foundational work on hardware configurations and inference engine optimizations to support extra-large language models, including disaggregated prefill/decode, prompt caching, KV-cache optimization across GPUs, speculative decoding, and the Infire inference engine with multi-GPU support.
More from the Artificial Intelligence Integration & Impact trail
- Introducing Clef-omni with full multimodality, plus a faster Clef and a cheaper Clef-flash
- Introducing Web Search API via AI Gateway
- Introducing Clef: our open-source decision models, and new RL fine-tuning platform
- One year later: Sovereign AI and the fight for choice
- AI Search is now generally available