Every request, every microsecond: scalable machine learning at Cloudflare

What this post added

This post details the evolution of Cloudflare's machine learning feature serving infrastructure. It describes the transition from a previous system using Gagarin and Unix Domain Sockets (UDS) to a new approach leveraging memory-mapped files for inter-process communication (IPC). The new system addresses issues of high tail latency, suboptimal resource utilization, and scalability constraints by employing wait-free synchronization and efficient data structures to serve hundreds of float features for millions of dimension keys with ultra-low latency.