How Cloudflare runs machine learning inference in microseconds

What this post added

This post details optimizations applied to the machine-learning detections within Cloudflare's bot management module, which accounts for approximately 15% of the latency added by bot detection. By porting from Lua to Rust, eliminating memory allocations, and optimizing parsers, the P50 latency of the bot management module was reduced by 79μs (a 20% reduction). Specific techniques include using fixed-size buffers on the stack, choosing algorithms that operate in-place, and leveraging testing tools like `dhat` to ensure zero allocations. Optimizations were also applied to the CatBoost ML library, including optimizing for single-document evaluation and reusing buffers for categorical features, leading to up to 15% performance gains in production models.