Building the foundation for running extra-large language models

What this post added

This post details the technical foundation for running extra-large language models on Cloudflare's Workers AI platform. It introduces hardware configurations optimized for agentic use cases, focusing on fast input token processing and tool calling. Key technical contributions include: Prefill Decode (PD) disaggregation for independent scaling and tuning of prefill and decode stages, token-aware load balancing, prompt caching using `x-session-affinity` headers to improve cache hit ratios, KV-cache optimization leveraging Moonshot AI's Mooncake Transfer Engine and Store for multi-GPU sharing and NVMe extension, and speculative decoding using NVIDIA's EAGLE-3 draft model for faster token generation. It also highlights the enhancements to Cloudflare's proprietary inference engine, Infire, including multi-GPU support (pipeline and tensor parallelism) and reduced memory overhead.