TechPrufer
Get started
Cerebras
Cerebras logoCerebras · Feature trail

PyTorch Support on Cerebras Wafer-Scale Engine

1 post · Mar 2025

What this trail covers

This post details advancements in PyTorch support on the Cerebras Wafer-Scale Engine (WSE), specifically with the release of CSoft 1.2. This version extends support for PyTorch 1.10.2 using a lazy tensor backend with XLA for optimal mapping of full model graphs onto the WSE-2. It enables users to train, evaluate, and fine-tune a wide range of NLP models with significantly higher performance. Additionally, CSoft 1.2 introduces the "Weight Streaming" execution mode, allowing for seamless scaling o Release 2.0 brings PyTorch 2.0 support to the Cerebras platform, providing generalized capability for developers to implement models, optimizers, and training scripts. A new backend based on Lazy Tensor Core (LTC) replaces the old PyTorch/XLA implementation, enabling faster support for new PyTorch versions. The Cerebras PyTorch API provides direct, generalized access to PyTorch for the first time on Cerebras hardware, allowing users to write fully custom training scripts and models. Contributions were made to the PyTorch backend for accelerators, integrating LTC with Torch-MLIR and improving PyTorch's LTC path. The release also introduces dynamic memory offload for activations, improving throughput by 50% on large GPT models, and adds support for Diffusion Transformers (DiT) and Sparse Pretraining and Dense Finetuning (SPDF). Expanded model support includes Jais, BTLM, LLaMA v2, Falcon 40B, StarCoder 15B, and U-Net 3D.

All Cerebras trails
  1. Supporting PyTorch on the Cerebras Wafer-Scale Engine - CerebrasThis post introduces the Cerebras implementation for supporting PyTorch on its Wafer-Scale Engine (WSE). It details the strategy of leveraging the WSE's scale for different model sizes (multi-replica, pipelined, weight streaming). The technical contribution lies in the integration approach: using a modified XLA backend and custom ATen operator wrapping to lower PyTorch graphs to CIRH, followed by optimization and compilation via the Cerebras ML backend. It also describes the runtime architecture and minimal user-facing API changes.
  2. Cerebras Software Platform R1.2 is Out! - CerebrasCSoft 1.2 extends PyTorch 1.10.2 support using a lazy tensor backend with XLA for full model graph mapping onto the WSE-2. It enables training, eval, and fine-tuning for NLP models like BERT, GPT2, Transformers, and T5 with improved performance. The release also introduces the "Weight Streaming" execution mode, allowing independent scaling of model size and training performance, with support for models up to 120 trillion parameters and the ability to train GPT3-XL (1.3 billion parameters) on a single CS-2 system. New public documentation and reference AI model implementations are also provided.
  3. Cerebras Software Release 2.0: 50% Faster Training, PyTorch 2.0 Support, Diffusion Transformers, and MoreRelease 2.0 introduces PyTorch 2.0 support via a new backend based on Lazy Tensor Core (LTC) and the Cerebras PyTorch API, enabling direct, generalized access for custom training scripts and models. Contributions were made to the PyTorch backend for accelerators, integrating LTC with Torch-MLIR. Dynamic memory offload for activations is introduced, improving training throughput by 50% on large GPT models. Support for Diffusion Transformers (DiT) and Sparse Pretraining and Dense Finetuning (SPDF) is added. Expanded model support includes Jais, BTLM, LLaMA v2, Falcon 40B, StarCoder 15B, and U-Net 3D.