Smaller, faster, safer: running Kimi and GLM at scale

What this post added

This post details three specific techniques used to optimize the serving of large language models (Kimi and GLM) on Cloudflare Workers AI: 1. Quantizing the KV cache from 16-bit precision (BF16) to 8-bit floating point (FP8) to double the context size and increase concurrent requests. 2. Compressing model weights from 8-bit floating point (FP8) to 4-bit integers (INT4) to reduce checkpoint size and improve decode speed. 3. Implementing KV cache integrity checking with page tags to prevent data corruption from shared cache pages, with a measured cost of under 1% on throughput and latency. The post also highlights the use of SGLang for inference serving and the decision to use different precision/compression levels for prefill and decode phases based on whether they are compute-bound or memory-bound.