TechPrufer
Get started
Cerebras
Cerebras logoCerebras · Feature trail

Wafer-Scale AI Accelerator Architecture Evolution

1 post · Mar 2025 – Aug 2026

What this trail covers

This post details the evolution of Cerebras's wafer-scale AI accelerator architecture, focusing on the WSE-3. It highlights the cluster-scale capabilities within a single chip, featuring 900,000 AI cores and 27 PB/s aggregate fabric bandwidth. The design incorporates a disaggregated memory architecture with MemoryX, enabling up to 1.2PB of external memory for trillion-parameter models. The CS-3 system is a 15U server consuming up to 23kW of power, offering 125 petaflops of AI performance and significantly improved scalability with up to 2048 systems linked via SwarmX interconnect. It can train models up to 24 trillion parameters and offers 2x performance over CS-2 at the same power and cost. The Condor Galaxy 3 supercomputer, powered by 64 CS-3 systems, exemplifies exa-scale performance with single-device simplicity. Cerebras also partners with Qualcomm for accelerated inference.

All Cerebras trails
  1. Ultrafast Frontier Inference | Cerebras Hot Chips 2026This post details the architectural advancements in Cerebras's CS-4, CS-5, and CS-6 AI accelerators. For CS-4, it elaborates on the Nexus rack-scale platform, modular compute backpacks, and specific improvements in power delivery (closer AC/DC converters) and cooling (integrated water-conditioning systems with sensors and quick-disconnects). It also highlights the centralized power infrastructure at the front of the rack with various redundancy configurations. For CS-5, it projects significant increases in token generation speed and efficiency. For CS-6, it introduces the concept of 3D integration by stacking DRAM with wafer-scale SRAM. The post contrasts Cerebras's on-wafer fabric bandwidth with traditional multi-GPU NVLink bandwidth, emphasizing reduced communication latency, power, cost, and failure points.
  2. Introducing Cerebras CS-4: The Fastest AI Gets FasterIntroduces the Cerebras CS-4 system, featuring the Wafer Scale Engine 3 Turbo processors and a redesigned Nexus rack-scale platform. Highlights up to 30x faster inference than GPU systems, improved economics, and hyperscale deployment capabilities. Details innovations in wafer-to-wafer communication (2 microsecond latency for >1000 tokens/sec on 10T models), higher throughput per watt (10x vs CS-3), and native support for disaggregated inference. The Nexus platform emphasizes modularity with a rear-mounted, self-contained Wafer-Scale Backpack for compute, power, and I/O, enabling faster manufacturing and deployment. Introduces high-density power delivery (100x closer to processors) and a new programmable I/O subsystem doubling bandwidth and slashing latency with RoCE v2 RDMA and Direct Wafer Links.
  3. CerebrasThis post introduces Batch Tiling on Attention (BTA) as a technique to improve compute utilization for sparse Mixture-of-Experts (MoE) models on the Cerebras Wafer Scale Engine (WSE). BTA decouples batch size requirements between attention and expert layers by splitting the attention input batch into tiles, allowing for smaller batches for attention (reducing activation memory) and larger batches for expert processing (improving compute utilization). Experiments with Qwen3 demonstrate that BTA prevents significant throughput degradation as MoE models become sparser (more experts, lower top_k), maintaining high throughput comparable to dense models, unlike conventional batching which suffers severe drops.
  4. Cerebras Architecture Deep Dive: First Look Inside the HW/SW Co-Design for Deep Learning [Updated] - CerebrasThis post provides a deep dive into the Cerebras architecture, focusing on its core design, scale-up, and scale-out strategies for deep learning. It details the core architecture's small size, distributed SRAM (48KB per core) with high bandwidth (equal to datapath bandwidth), and hardware support for tensor instructions. It highlights the ability to run matrix operations at full performance across all BLAS levels, enabling unstructured sparsity acceleration. The fine-grained dataflow scheduling mechanism is explained, which triggers computation based on data arrival and filters out zero data, leading to native sparsity acceleration and power savings. Micro-threading is introduced to maintain high utilization by switching between independent tensor contexts. The post also touches upon the scale-up and scale-out aspects of the architecture, aiming for order-of-magnitude improvements in clustering and performance.
  5. Introducing Condor Galaxy 1: a 4 exaFLOPS Supercomputer for Generative AI - CerebrasThis post announces the deployment of Condor Galaxy 1 (CG-1), a 4 exaFLOPS, 64-node AI supercomputer, representing a significant scaling of Cerebras's wafer-scale AI accelerator architecture. It details the underlying technologies: Cerebras Wafer-Scale Cluster for connecting multiple CS-2 systems as a single logical accelerator with decoupled memory, and weight streaming for simplified data parallelism in training large models. The post highlights CG-1's specifications (4 exaFLOPS, 54M cores, 82TB memory) and its availability via Cerebras Cloud, emphasizing the dramatic reduction in code complexity for training large models compared to GPU-based solutions. The roadmap for deploying nine such supercomputers totaling 36 exaFLOPS by 2024 is also presented.
  7. Announcing the Cerebras Architecture for Extreme-Scale AI - CerebrasIntroduces a new extreme-scale AI cluster architecture that disaggregates memory and compute. Key innovations include 'weight streaming' where model weights are stored in MemoryX and streamed to CS-2 systems, enabling independent scaling of model size and training speed. SwarmX fabric provides efficient parallel training across up to 192 CS-2 systems with near-linear performance scaling. The architecture also accelerates sparse neural networks through fine-grained dataflow scheduling, saving power and achieving 10X speedup for weight sparsity.
  8. An AI Chip With Unprecedented Performance To Do the Unimaginable - CerebrasThis post introduces the Wafer Scale Engine 2 (WSE-2), Cerebras's latest AI accelerator. It details the WSE-2's architecture, which includes 850,000 specialized AI compute cores on a single 7nm wafer-scale chip, providing 220Pb/s of aggregate fabric bandwidth. Key technical advancements include: cluster-scale compute on a single chip, proprietary interconnects for reduced communication overhead, 40GB of on-wafer memory with 20 PB/s bandwidth, and sparsity harvesting capabilities to optimize computations with sparse tensors. The uniform fabric across the wafer simplifies software development and improves time-to-solution by eliminating the need for placement optimization. The WSE-2 is designed for high utilization and performance scaling for AI workloads.
  9. Cerebras CS-3 vs. Nvidia B200: 2024 AI Accelerators Compared - CerebrasThis post introduces the Cerebras CS-3, powered by the third-generation wafer-scale engine (WSE-3), featuring 900,000 AI cores and 125 petaflops of FP16 AI compute. It highlights the disaggregated memory architecture with MemoryX, offering up to 1.2PB of external memory, which is crucial for training trillion-parameter models and simplifies model development by presenting a unified memory block. The CS-3's on-wafer fabric provides 27 PB/s of aggregate bandwidth, significantly exceeding GPU interconnects. The post also emphasizes the CS-3's superior performance per watt compared to the Nvidia B200.
  10. Cerebras CS-3: the world’s fastest and most scalable AI accelerator - CerebrasIntroduces the Cerebras CS-3, the third-generation wafer-scale AI accelerator. Key advancements include: 2x performance increase over CS-2 with 4 trillion transistors, 2x FP16 SIMD units per core, and boosted non-linear arithmetic operations. Scalability is dramatically improved with SwarmX interconnect supporting clusters of up to 2048 CS-3 systems (10x improvement over CS-2), delivering 256 exaflops. Memory scaling is enhanced with MemoryX options up to 1,200 TB, enabling training of models up to 24 trillion parameters. The Condor Galaxy 3 supercomputer (64 CS-3 systems, 8 exaflops) is highlighted as an example of exa-scale performance and single-device simplicity. A partnership with Qualcomm is announced for 10x faster LLM inference.