Cerebras Inference now 3x faster: Llama3.1-70B breaks 2,100 tokens/s - Cerebras

This post details a 3x performance improvement for Cerebras Inference, achieving 2,100 tokens/sec for Llama3.1-70B. This was achieved through re-writing and optimizing critical kernels (MatMul, reduce/broadcast, element-wise ops, activations), streamlining wafer IO to run asynchronously from compute, and implementing speculative decoding. The post also highlights the performance advantages in terms of output speed per user, time to first token, and total response time compared to GPU-based solutions and hyperscale clouds.