Introducing Cerebras Inference: AI at Instant Speed - Cerebras

This post introduces Cerebras Inference, highlighting its ability to deliver 1,800 tokens/sec for Llama3.1 8B and 450 tokens/sec for Llama3.1 70B, which is claimed to be 20x faster than NVIDIA GPU-based hyperscale clouds and 2.4x faster than Groq for Llama3.1-8B. The core technical innovation is the Cerebras Wafer Scale Engine (WSE-3) which integrates 44GB of SRAM on-chip, providing 21 petabytes/s of aggregate memory bandwidth. This eliminates the need to move model parameters from external memory for each token generation, a key bottleneck in GPU inference. The post explains that for models exceeding single-wafer capacity, they are split at layer boundaries and mapped to multiple CS-3 systems. It also emphasizes the use of native 16-bit model weights for accuracy, contrasting with 8-bit quantization used by some competitors. The Cerebras Inference API is available via chat and API access, built on the OpenAI Chat Completions format, with initial free token offerings and competitive pricing for scaled deployments. The post also touches on how fast inference enables new techniques like scaffolding for more thoughtful AI responses.