How we built the most efficient inference engine for Cloudflare’s network

What this post added

This post details the development of Infire, a new LLM inference engine written in Rust, designed to maximize memory, network I/O, and GPU utilization for Cloudflare's edge network. It addresses limitations of existing solutions like vLLM by enabling co-hosting of multiple models on a single GPU, secure execution without additional virtualization layers, and optimized performance for dynamic workloads. Key technical contributions include efficient model loading with Page Locked memory and CUDA asynchronous copies, parallel kernel compilation, an OpenAI compatible HTTP server, and a batcher employing continuous batching with chunked prefill to optimize matrix operations and KV cache utilization.