Smaller, faster, safer: running Kimi and GLM at scale

This post details three techniques for optimizing the serving of large, long-context, mixture-of-experts AI models (Kimi K-series and GLM) on Cloudflare's Workers AI platform: 1. KV cache quantization from BF16 to FP8, which doubles the context length and increases concurrent requests by 41% at a lower cost per token, with no measurable accuracy degradation. 2. Model weight compression from FP8 to INT4, reducing checkpoint size by 40% and per-GPU memory, leading to a 55% speedup in decode latency at low concurrency, with model accuracy within 0.8 points. 3. KV cache integrity checking, a defense layer that adds a tag to cache pages and checks mappings before decode operations, incurring less than 1% overhead on throughput and tail latency. The post also highlights the use of SGLang for inference serving and the collaboration with its team.