Cloudflare’s bigger, better, faster AI platform

What this post added

This post announces significant upgrades to Cloudflare's AI platform for Birthday Week 2024. Key contributions include: 1. Workers AI upgrades: more powerful GPUs enabling faster inference and support for larger models (e.g., Llama 3.1 70B), expanded GPU footprint to over 180 cities, improved performance metrics (80+ TPS for 8b models, 300ms TTFT), support for larger context windows (128K), Day 0 support for Meta's Llama 3.2 models, and the introduction of 'Run Any* Model' for dynamic model loading (closed beta). A technical blog post details KV cache compression, speculative decoding, and new hardware design. 2. Workers AI pricing: transition from 'neurons' to a unit-based pricing model (tokens, image resolution/steps, audio seconds) with updated free tier allocations. 3. AI Gateway enhancements: introduction of persistent logs (open beta) for up to 10 million logs per gateway with enhanced logging interface (cost, duration, human feedback, custom metadata), Logpush integration for exporting logs, and AI evaluations (open beta) using human-in-the-loop feedback for scoring model performance, speed, and cost. 4. Vectorize: General Availability (GA) with expanded index sizes and faster queries.