How Cloudflare runs machine learning inference in microseconds

This post details optimizations applied to the machine-learning models within Cloudflare's bot management module, reducing P50 latency by 20% (from 388μs to 309μs) and P99 latency by 14% (from 940μs to 813μs). Key optimizations include: 1. Engineering for zero allocations by using stack allocation for fixed-size buffers and re-using pre-allocated buffers. 2. Choosing algorithms that operate in-place and store state on the stack, such as using iterators for case-insensitive string comparison. 3. Implementing automated testing for memory allocations using `dhat`. 4. Optimizing CatBoost for single-document evaluation by passing references to contiguous memory instead of allocating vectors. 5. Rewriting Rust bindings to accept slices (`&[f32]`, `&[&str]`) instead of owned `Vec`s and pre-computing hashes for categorical features to enable buffer reuse across multiple model executions.