Cloudflare’s bigger, better, faster AI platform

This post details significant upgrades to Cloudflare's AI platform, focusing on Workers AI, AI Gateway, and Vectorize. For Workers AI, it announces the addition of more powerful GPUs enabling faster inference and support for larger models (e.g., Llama 3.1 70B), expansion of GPU presence to over 180 cities, and a new unit-based pricing model replacing 'neurons'. It also introduces the 'Run Any* Model' feature in closed beta for greater model flexibility. For AI Gateway, new features include persistent logs (open beta) with increased storage capacity, enhanced logging interface with cost/duration insights and human feedback annotations, custom metadata tagging, and Logpush export. AI evaluations with human-in-the-loop feedback are also in open beta. Vectorize is announced as generally available with expanded index sizes and faster queries. The post also references a technical blog post detailing the optimizations for Workers AI speed, including KV cache compression, speculative decoding, and new hardware design.