Cerebras Software Release 2.0: 50% Faster Training, PyTorch 2.0 Support, Diffusion Transformers, and More

Release 2.0 introduces dynamic memory offload for activations, allowing for larger batch sizes and a 50% performance improvement in training GPT-3 models. It also integrates PyTorch 2.0 support through a new backend based on Lazy Tensor Core (LTC) and Torch-MLIR, providing a standard PyTorch 2.0 interface. New model support includes diffusion transformers (DiT) with AdaLN-Zero, and expanded support for Jais, BTLM, LLaMA v2, Falcon 40B, StarCoder 15B, and 3D U-Net. The release also enhances dynamic sparsity support with the introduction of Sparse Pretraining and Dense Finetuning (SPDF).