Every request, every microsecond: scalable machine learning at Cloudflare

This post details the evolution of Cloudflare's machine learning feature serving infrastructure. It describes the transition from a Unix Domain Socket (UDS) based system (Gagarin) to using memory-mapped files for inter-process communication (IPC). The challenges with UDS, including high tail latency, suboptimal resource utilization, and scalability constraints, are highlighted. The new approach leverages memory-mapped files for reduced latency and CPU contention, combined with wait-free synchronization and efficient data deserialization techniques to serve millions of ML features per request.