Workers AI gets a speed boost, batch workload support, more LoRAs, new models, and a refreshed dashboard

This post details significant engineering advancements in Cloudflare's Workers AI platform. Key contributions include the implementation of speculative decoding for 2-4x faster inference, leveraging unused GPU compute by running a smaller draft model concurrently with the main model. Prefix caching has been introduced to reduce 'pre-fill' time for requests reusing context, benefiting RAG, code generation, and chatbots. An asynchronous batch API has been launched to handle large workloads that are not latency-sensitive but require completeness guarantees, preventing immediate errors due to capacity constraints and improving overall inference server utilization. LoRA support has been expanded to 8 models with larger ranks (up to 32) and file sizes (up to 300MB). The post also mentions the addition of new models and a refreshed dashboard for usability.