How Cloudflare runs more AI models on fewer GPUs: A technical deep-dive

What this post added

Introduced Omni, an internal platform for running and managing AI models on Cloudflare's edge nodes. Omni enables spawning multiple models from a single control plane, implements lightweight process isolation with filesystem separation, and over-commits GPU memory using a CUDA stub library and unified memory mode to run more models on a single GPU. It also addresses Python memory allocation issues by using FUSE to create a virtual /proc/meminfo reflecting per-model limits and leverages uv for Python virtual environments.