Making Workers AI faster and more efficient: Performance optimization with KV cache compression and speculative decoding

This post details performance optimizations for Cloudflare's Workers AI platform. It introduces upgraded hardware (12th generation compute servers) enabling higher throughput for Llama 3.1 and 3.2 models. It also describes a novel, open-source KV cache compression technique using PagedAttention, which allows for up to 64x compression and a 5.18x increase in throughput by efficiently managing memory allocation for attention heads and sequence lengths. Additionally, speculative decoding, specifically prompt-lookup decoding, is implemented to increase generation speed by up to 70% for Llama 3.1 70B models, albeit with a potential trade-off in output quality.