Making Workers AI faster and more efficient: Performance optimization with KV cache compression and speculative decoding

What this post added

Introduced two performance optimization techniques for Workers AI: KV cache compression using PagedAttention to increase throughput by up to 8x and speculative decoding (specifically prompt-lookup decoding) to improve generation speed by up to 40% for Llama-3.1-8B-instruct and up to 70% for the 70B model. Also announced upgraded hardware supporting newer GPUs and larger models like Meta Llama 3.2 11B and Llama 3.1 70B, offering 2-3x throughput for these models.