How Cloudflare runs more AI models on fewer GPUs: A technical deep-dive

This post details the internal platform 'Omni' designed to improve GPU utilization for AI models on Cloudflare's edge nodes. Omni achieves this by: 1. Spawning multiple models from a single control plane (scheduler) using lightweight process isolation and Python virtual environments managed by uv. 2. Implementing a virtual filesystem using FUSE for /proc/meminfo to accurately reflect per-model memory limits, preventing OOM errors. 3. Over-committing GPU memory by injecting a CUDA stub library that intercepts memory allocation calls and forces them into unified memory mode, allowing more models to share a GPU at the cost of potential latency during memory transfers. 4. Enforcing memory limits by overriding CUDA runtime APIs to expose only a subset of memory to each model. Omni integrates with Workers AI, KV, and various inference engines (vLLM, Infire) to provide a unified layer for model management and inference.