How we built the most efficient inference engine for Cloudflare’s network

This post introduces Infire, a new LLM inference engine written in Rust, designed to optimize GPU, memory, and network I/O utilization for Cloudflare's edge network. It details architectural challenges of running LLM inference at the edge, including the need for efficient resource utilization, dynamic workload handling, and strong security sandboxing. Infire employs continuous batching with chunked prefill and optimized KV cache handling. It also utilizes Page Locked memory with CUDA asynchronous copy over multiple streams for faster model loading onto GPUs. The engine is currently powering the Llama 3.1 8B model for Workers AI.