TechPrufer
Get started
Cerebras
Cerebras logoCerebras · Feature trail

Large Model Training on Cerebras Hardware

42 posts · Mar 2025 – Jul 2026

What this trail covers

This post details advancements in using Maximal Update Parameterization (μP) to improve AI model training stability and efficiency across different model scales. μP enables stable hyperparameters, reduces tuning costs, and improves loss at large scale by ensuring training dynamics are invariant to model width. This leads to more predictable scaling and robust research by alleviating the 'Parameterization Lottery'. The post provides a practical guide to μP's mathematical underpinnings and implementation, including a port to nanoGPT.

All Cerebras trails
  1. CerebrasThis post details the debugging of a Mixture-of-Experts (MoE) model training process, specifically addressing issues with learned routing. It identifies a critical bug where setting `top_k=1` for expert selection prevents the router from receiving gradients from the cross-entropy loss, leading to ineffective routing. The solution proposed is to introduce a 'null expert' (a phantom expert with a bias) to provide a contrast for the router, enabling proper gradient flow and improved performance. The post demonstrates this fix by showing restored gradient flow and improved model performance compared to a dense baseline, making MoE training more accessible and effective.
  2. CerebrasThis post introduces Batch Tiling on Attention (BTA) as a technique to improve compute utilization in Mixture-of-Experts (MoE) models on the Cerebras Wafer Scale Engine (WSE). BTA addresses the issue of idle experts and inefficient attention layer processing by splitting the attention input batch into smaller tiles, processing them independently, and then concatenating the results for expert processing. This allows for smaller batches for attention layers (reducing activation memory) and larger batches for expert networks (improving compute utilization). Experiments with the Qwen3 model demonstrate that BTA significantly mitigates throughput degradation as MoE models become sparser, maintaining high throughput comparable to dense models, unlike conventional batching which shows severe degradation.
  3. MoE Fundamentals: Why Sparse Models Are the Future of AIThis post introduces the fundamental concepts of Mixture-of-Experts (MoE) models, explaining why they are crucial for scaling AI beyond current dense model limitations. It details how MoE achieves efficiency gains through conditional parameter activation, contrasting it with dense transformers. The post highlights the logarithmic improvement in loss with increasing expert count and discusses the implementation of MoE layers, including the role of the router and different routing strategies. It references key research papers and provides a roadmap for future discussions on MoE routing strategies and scaling.
  4. CerebrasIntroduces REAP (Router-weighted Expert Activation Pruning), a one-shot pruning method for MoE models. REAP removes low-impact experts by considering both router activation frequency and expert output magnitude, outperforming expert merging for generative tasks. Demonstrates significant compression (up to 50%) on trillion-parameter models with minimal quality degradation, retaining over 96% of baseline capabilities on coding and tool-use tasks. Open-sources codebase and pruned model checkpoints.
  5. SlimPajama: A 627B token, cleaned and deduplicated version of RedPajamThis post details the creation of SlimPajama, a 627B token dataset derived from RedPajama by extensive cleaning and deduplication. It introduces a novel, scalable infrastructure for MinHashLSH deduplication on trillion-token datasets, implemented using producer-consumer patterns and optimized datasketch logic for distributed and memory-efficient processing. The post also describes the data cleaning steps, including filtering short, low-quality documents and applying MinHashLSH with a Jaccard similarity threshold of 0.8 on 13-grams. The resulting dataset and the data processing library are open-sourced.
  7. Introducing Condor Galaxy 1: a 4 exaFLOPS Supercomputer for Generative AI - CerebrasIntroduces Condor Galaxy 1 (CG-1), a 4 exaFLOPS AI supercomputer, expanding the scale of Cerebras's large model training capabilities. Details the architecture of CG-1, including its 64 CS-2 systems, 82 terabytes of memory, and 386 terabits of internal fabric bandwidth. Outlines the roadmap for the Condor Galaxy series, culminating in 36 exaFLOPS across nine centers. Highlights the simplified programming model for training large models (up to 100 billion parameters natively, extendable to 100 trillion) with a standard GPT implementation requiring only 1,200 lines of code. Mentions native support for 50,000-token sequence length training.
  8. Introducing gigaGPT: GPT-3 sized models in 565 lines of code - CerebrasIntroduces gigaGPT, a 565-line implementation of nanoGPT that enables training GPT-3 sized models (up to 175B parameters) on Cerebras hardware. It leverages the hardware's large memory capacity (MemoryX) and compute power to stream model weights, eliminating the need for complex distributed training frameworks like Megatron or DeepSpeed. The post validates gigaGPT by training models of various sizes (111M, 13B, 70B, 175B) and highlights the simplicity of scaling training across 1 to 192 CS-2 systems using standard mini-batches (data parallelism). It also details the use of `cerebras_pytorch` for tracing compute graphs and managing distributed execution.
  9. The Practitioner’s Guide to the Maximal Update Parameterization - CerebrasThis post introduces and explains Maximal Update Parameterization (μP) as a technique to improve AI model training. It details four key benefits: stable hyperparameters across scale (μTransfer), improved loss at large scale due to better HP tuning, significantly decreased danger of instability at large scale, and more predictable scaling. The post provides a mathematical overview of controlling activation magnitudes, gradients, and weight updates to ensure these dynamics are invariant to model width. It also highlights a practical implementation port to nanoGPT.
  10. Sparsity Made Easy – Introducing the Cerebras PyTorch Sparsity Library - CerebrasIntroduces a new Cerebras PyTorch Sparsity Library that simplifies the use of unstructured sparsity for training large language models on Cerebras hardware. The library provides a modular API for implementing static and dynamic sparsity algorithms, including GMP, SET, and RigL, and demonstrates ease of integration with existing PyTorch models and the Cerebras Model Zoo. Benchmarks show improved model quality with dynamic sparsity over static methods on a Llama2-like model. The library's extensibility is highlighted by enabling the GraNet algorithm.
  11. Unlocking High-Resolution Computer Vision with Wafer-Scale Technology - CerebrasThis post details how Cerebras' wafer-scale technology and Weight Streaming execution mode accelerate high-resolution computer vision tasks, specifically CNNs for image segmentation and classification. It highlights the architectural advantages of the CS-2 system, including the Wafer-Scale Engine (WSE) with its large on-chip SRAM and high-speed fabric, which enables monolithic training and eliminates the need for large GPU clusters. The post explains how Weight Streaming disaggregates compute from memory, allowing independent scaling. Key advantages for CV models include fitting all compute and activations for segmentation/classification on a single WSE, the Cerebras Graph Compiler's ability to scale from small to large images/volumes without code changes, and efficient handling of bandwidth-bound operations like normalization, which can be performance bottlenecks on GPUs. The post also mentions successful demonstrations of segmentation tasks up to 7K2 resolution.
  12. Fine-Tuning with Cerebras AI Model Studio Launchpad - CerebrasThis post introduces the Cerebras AI Model Studio Launchpad, a CLI tool for fine-tuning foundation models. It presents experimental evidence showing that fine-tuning smaller models (e.g., GPT-NeoX 1.4B) on domain-specific datasets (TRC2 Corpus) can achieve performance comparable to or exceeding larger out-of-the-box models (e.g., GPT-J 6B). The post details evaluation metrics (loss, accuracy, perplexity) and discusses the trade-offs between fine-tuning and training from scratch, emphasizing resource and cost benefits of fine-tuning smaller models. It also provides a practical guide to using the Launchpad CLI for dataset addition and hyperparameter configuration.
  13. Getting Started with PyTorch BERT Models - CerebrasThis post details how to run PyTorch BERT models on Cerebras CS-2 systems. It introduces convenience wrappers in the Cerebras Software Platform (CSoft) for porting existing PyTorch scripts with minimal code changes. Key performance optimizations include multi-worker coordination for data loading and step closures for queuing functions between training iterations. A reference implementation of BERT in PyTorch is provided, including model, dataloader, and configuration YAMLs for BERTBASE and BERTLARGE variants. It outlines the process for running training and evaluation on CPU, compiling for WSE, and executing on a CS system using provided scripts.
  14. Training Multi-Billion-Parameter Models on a Single Cerebras System is Easy - CerebrasThis post demonstrates the ease of training multi-billion-parameter models on a single Cerebras CS-2 system by abstracting away the complexities of distributed training. It highlights how configuration changes, rather than code modifications, allow for seamless scaling from 1.3B to 6.7B parameters, contrasting this with the intricate and challenging 3D parallelism required for GPU clusters. The Cerebras Software Platform (CSoft) and tools like `csrun_wse` simplify job launching and compilation, enabling researchers to focus on model development.
  15. Supporting PyTorch on the Cerebras Wafer-Scale Engine - CerebrasThis post details the integration of PyTorch support onto the Cerebras Wafer-Scale Engine (WSE-2). It outlines the strategy of adhering to PyTorch's design principles, minimizing user surprises, and integrating with its ecosystem. The implementation leverages a modified XLA backend and MLIR to lower PyTorch graphs to the Cerebras Intermediate Representation (CIRH). Different execution modes (Multi-Replica, Pipelined, Weight Streaming) are described for various model sizes. The Cerebras ML backend performs graph rewrites and automatic kernel generation. The runtime architecture separates the neural network execution on the CS-2 from the chief worker and input workers. User experience is simplified with minimal code modifications under the _cerebras_torch_ namespace.
  16. When Time is Money: Accelerating NLP Model Training at a Leading Financial Institution - CerebrasThis post details a case study where Cerebras's CS-2 system accelerated NLP model training for a financial institution. The CS-2 system delivered compute performance equivalent to over 120 AI-optimized GPUs, reducing training time for a complex BERTLARGE model by 15X compared to a leading 8-GPU server. This resulted in dramatic improvements in model prediction confidence and almost halved energy consumption. The post highlights how this performance enables financial institutions to rapidly experiment with and develop better domain-specific NLP models, which was previously computationally intensive and out of reach.
  17. To Bfloat or not to Bfloat? That is the Question! - CerebrasThis post details the benefits of using the bfloat16 data format for automatic mixed precision training of deep learning models, specifically GPT-style models. It explains how bfloat16, with its 8 exponent bits, offers a greater dynamic range than float16, leading to improved training throughput (18% increase observed for GPT-3 XL), reduced weight growth, and better evaluation scores. The post also highlights that bfloat16 is less prone to numerical instability (underflows, overflows) compared to float16 and can be a near drop-in replacement for float32, often negating the need for loss scaling. Configuration details for enabling bfloat16 in mixed precision mode on Cerebras CS-2 are provided.
  18. BTLM-3B-8K: 7B Performance in a 3 Billion Parameter Model - CerebrasThis post introduces BTLM-3B-8K, a 3 billion parameter language model trained on Cerebras hardware that achieves 7B performance. It details the training process, including the use of SlimPajama dataset, SwiGLU nonlinearity, learning rate decay, and maximal update parameterization. It also highlights the model's long sequence inference capabilities using ALiBi and variable sequence length training. The post emphasizes the scalability and reliability of the Cerebras CS-2 system and the Condor Galaxy 1 supercomputer for training such models, noting the absence of hardware failures and the ease of scaling.
  19. Accelerating Large GPT Training with Sparse Pre-Training and Dense Fine-Tuning [Updated] - CerebrasThis post introduces Sparse Pre-training and Dense Fine-tuning (SPDF) for large GPT models. It demonstrates that pre-training GPT-3 XL (1.3B parameters) with up to 75% unstructured sparsity on the Cerebras CS-2 system reduces training FLOPs by 60% compared to dense training. Subsequent dense fine-tuning preserves downstream task metrics on data-to-text NLG tasks (E2E, WebNLG, DART) with minimal BLEU score degradation (-0.44 to -0.75). The post also shows that simpler NLG tasks tolerate higher sparsity than complex summarization tasks (Curation Corpus), and larger models are more amenable to higher sparsity. The Cerebras CS-2's hardware acceleration for unstructured sparsity is highlighted as a key enabler for these large-scale sparsity studies.
  20. More Pixels, More Context, More Insight! - CerebrasThis post details how the Cerebras CS-2 system, powered by the Wafer-Scale Engine (WSE), enables training large deep neural networks on high-resolution images (up to 50 megapixels) without the memory limitations typically encountered with GPUs. It highlights the WSE's 40GB of on-chip SRAM and unique memory access capabilities, allowing direct processing of large images without tiling or downsampling, thus preserving contextual information and prediction accuracy. The post also mentions an update to R1.8 of the Cerebras Software Platform, increasing support for image segmentation on 50-megapixel images.
  21. Variable Sequence Length Training for Long-Context Large Language Models - CerebrasIntroduces Variable Sequence Length (VSL) training, a two-stage method to reduce training time for long-context LLMs. VSL involves training primarily with short sequences (Stage-1) and then adapting with longer sequences (Stage-2). This method requires no architectural changes or hyperparameter tuning and can reduce FLOPs by up to 29% while maintaining model performance. The Cerebras Software Platform (CSoft) is highlighted as a simple way to implement VSL training by modifying data subset paths.
  22. Multi-Billion-Parameter Model Training Made Easy with CSoft R1.3 - CerebrasCSoft 1.3 enables training and fine-tuning of GPT-J 6B parameters model using weight streaming execution mode on a single CS-2 system. It expands PyTorch support with reference implementations for BERT fine-tuning tasks (sequence classification, extractive summarization, question answering) and upgrades to PyTorch 1.1. Variable Tensor Shape (VTS) computations are introduced for efficient computation on data samples with heterogeneous shapes. Multi-replica data parallel distribution is expanded to TensorFlow for smaller transformer models, boosting training speed.
  23. Context is Everything: Why Maximum Sequence Length Matters - CerebrasThis post details how Cerebras's Wafer-Scale Engine (WSE) architecture, through its massive processing power and high-bandwidth on-chip communication, enables the practical training of transformer models with significantly longer sequence lengths (up to 50,000 tokens) using dense attention. This overcomes the quadratic complexity limitations of traditional architectures on GPUs. The post explains the weight streaming execution mode as the key mechanism for achieving this by managing memory and enabling longer sequences at full performance. It also presents experimental results showing improved accuracy in next token prediction with longer sequences and discusses the implications for various applications.
  24. GlaxoSmithKline and Cerebras are Advancing the State of the Art in AI for Drug Discovery - CerebrasThis post details the application of Cerebras's Wafer-Scale Engine (WSE) to accelerate AI model training for drug discovery, specifically for epigenomic language models (EBERT). The WSE's massive compute power (400,000 cores on a single chip) enables training complex models like EBERT in 2.5 days, compared to an estimated 24 days on a GPU cluster. This speedup allows for exploration of architecture variations and hyperparameters, leading to improved prediction accuracy on benchmarks like ENCODE-DREAM. The latest CS-2 system shows further throughput improvements for smaller models and maintains throughput for larger models. The collaboration with GSK highlights the practical application of Cerebras hardware in advancing AI for drug discovery.
  25. Scaling Up and Out: Training Massive Models on Cerebras Systems using Weight Streaming - CerebrasThis post introduces and details the 'weight streaming' execution mode for the Cerebras Wafer-Scale Engine (WSE), enabling the training of AI models with over 100 trillion parameters, far exceeding the WSE's on-chip memory capacity. It explains how model parameters are managed in an off-chip memory tier with 4 TB/s bandwidth, while the WSE continuously processes streams of weights and gradients. The post also elaborates on the WSE's architecture, its ability to leverage 90% unstructured weight sparsity for a 10x acceleration, and its full-performance execution of all linear algebra operations (GEMM, GEMV, vector-vector) which is critical for both training and low-latency inference.
  26. A Big Chip for Big Science: Watching the COVID-19 Virus in Action - CerebrasThis post details the application of Cerebras CS-2 hardware for accelerating the training of a Convolutional Variational Autoencoder (CVAE) used for steering scientific simulations of the SARS-CoV-2 virus. The CVAE, which was a bottleneck due to its quadratic time and space complexity, achieved significantly higher training throughput on a single CS-2 system (24,000 samples/s) compared to 1,536 GPUs on the Summit supercomputer. The post highlights the 'out-of-the-box' performance and ease of use of the CS-2 system for organizations lacking large GPU clusters, enabling researchers to study microseconds of molecular motion instead of tens of nanoseconds.
  27. An AI Chip With Unprecedented Performance To Do the Unimaginable - CerebrasThis post introduces the Wafer Scale Engine 2 (WSE-2), Cerebras' second-generation wafer-scale AI processor. It highlights the WSE-2's architecture, featuring 850,000 AI cores, 46,225mm² of silicon area, 220Pb/s aggregate fabric bandwidth, and 40GB of on-wafer memory. Key technical details include the proprietary interconnect for stitching die, sparsity harvesting capabilities to reduce computations, and the benefits of on-chip memory for reduced latency and power consumption. The uniform fabric bandwidth across the wafer simplifies programming and improves time-to-solution. The post emphasizes the system's ability to handle large models without performance degradation and its optimization for various BLAS levels, including matrix-vector and vector-vector operations.
  28. Cerebras Software Platform R1.2 is Out! - CerebrasCSoft 1.2 introduces "Weight Streaming" for extreme-scale AI models, enabling independent scaling of model size and training performance, and supporting models up to 120 trillion parameters. It adds support for training GPT3-XL (1.3 billion parameters) on a single CS-2 system. The release also expands PyTorch support to version 1.10.2 using a lazy tensor backend with XLA, allowing for efficient mapping of full model graphs onto the WSE-2 and enabling training, evaluation, and fine-tuning of NLP models like BERT, GPT2, Transformers, and T5 with significantly improved performance.
  29. Cerebras Sets Record for Largest AI Models Ever Trained on Single Device - CerebrasThis post details the challenges and complexity of training large AI models (up to 20 billion parameters) on distributed GPU clusters, requiring months of engineering effort for hybrid parallelism (data, pipeline, and tensor model parallelism). It contrasts this with the Cerebras CS-2 system, which allows for the training of these large models on a single device with minutes of setup time, simplifying the process and making it accessible to a wider range of users. The post emphasizes that the primary challenge in training large models is often distributed computing, not the machine learning itself.
  30. How to Harness the Predictive Power of GPT-J | CerebrasThis post details the fine-tuning of the 6B parameter GPT-J model on a single Cerebras CS-2 system. It highlights the GPT-J model's architectural differences from GPT-3, including parallel decoder architecture and Rotary Position Embeddings, noting their impact on throughput and model quality. The primary contribution is demonstrating how Cerebras's weight streaming execution mode simplifies the fine-tuning of large models like GPT-J, which would otherwise require multiple accelerators due to memory constraints (e.g., 96GB for Adam optimizer states). The post emphasizes that this capability makes it easy to load and tune GPT-J checkpoints on a single CS-2 with custom datasets, addressing the intractable compute demands of such models on conventional infrastructure.
  31. Creating Sparse GPT-3 Models with Iterative Pruning - CerebrasIntroduced iterative pruning to create highly sparse GPT-3 models (1.3B parameters) on the Cerebras CS-2 system. Demonstrated an 83.8% sparse model with a 3x reduction in inference FLOPs and 4.3x reduction in parameters, while maintaining competitive validation loss on the Pile dataset and matching zero-shot performance on LAMBADA and WikiText103 compared to the dense model. The method sparsifies projection and feed-forward network layers separately and involves retraining after pruning.
  32. The Cerebras AI Model Studio brings Wafer-Scale Cluster Acceleration to the Cloud - CerebrasThis post announces the launch of the Cerebras AI Model Studio, a cloud-based service offering access to Cerebras Wafer-Scale Clusters for training large language and generative AI models. It highlights the challenges of LLM training (cost, time, complexity) and positions the Model Studio as a solution. Key features include simplified access via Cirrascale Cloud Services, pre-configured environments, model-based pricing for predictability, and support for various GPT-class models. It also mentions add-on capabilities for enhanced training and inference.
  33. Genomics in Unparalleled Resolution: Cerebras Wafer-Scale Cluster Trains Large Language Models on the Full COVID Genome Sequence - CerebrasThis post details the successful training of large language models (LLMs) on full genomic sequences, such as the SARS-CoV-2 genome, using Cerebras Wafer-Scale Clusters. It highlights overcoming memory and compute limitations of traditional GPU clusters, enabling training of models with billions of parameters and sequence lengths up to 10,240 tokens. The post emphasizes the ease of use of the Cerebras appliance workflow for distributed LLM training and demonstrates linear throughput scaling up to 16 CS-2 nodes, achieving significantly faster time-to-solution compared to GPU-based supercomputers for these specific genomics tasks. The work achieved a time-to-solution of less than one day for GenSLMs with 123M and 1.3B parameters, and demonstrated the ability to train GPT-J models up to 25B parameters with MSL 10,240, achieving linear throughput scaling up to 16 CS-2s.
  34. Cerebras Software Platform R1.7 is Out! - CerebrasThis release enables training large language models with sparse weights in PyTorch, providing scripts to convert dense PyTorch checkpoints to sparse versions. It also introduces unique support for 2D segmentation with UNet on images up to 5K x 5K pixels without tiling, and extends PyTorch support for GPT-J and GPT-3 models up to 20B parameters, with up to 1.5x performance improvement for GPT-style models.
  35. What is Appliance Mode? - CerebrasIntroduces 'Appliance Mode' for the Cerebras cluster, simplifying the process of training large language models. This mode abstracts away the complexities of distributed job launching, resource management, and parameter tuning. Users can now specify their model and desired CS-2 nodes, and the system handles the orchestration. It supports both pipelined and weight streaming execution modes, allowing for training models larger than the memory capacity of a single wafer. The mode is more "pythonic," uses standard TensorFlow/PyTorch wheels, and integrates with common datacenter orchestration tools like Kubernetes. It contrasts with the previous, more complex, single-CS-2, script-based deployment that relied on Slurm and custom containers.
  36. Linear Scaling Made Possible with Weight Streaming - CerebrasIntroduces weight streaming via MemoryX and SwarmX, enabling linear scaling of LLMs up to trillions of parameters using only data parallelism. This disaggregates memory and compute, simplifying distributed compute challenges and offering push-button allocation of work. The Cerebras Wafer-Scale Cluster can scale from a single CS-2 to 192 CS-2s with a single keystroke. The WSE-2's large size allows fitting large layers onto a single device, and MemoryX provides unbounded storage for parameters and optimizer states by streaming weights to the CS-2 and gradients back. This allows a single CS-2 to train models up to 10 billion parameters, and larger models for fine-tuning or development.
  37. Cerebras-GPT: A Family of Open, Compute-efficient, Large Language Models - CerebrasThis post introduces Cerebras-GPT, a family of seven open-source GPT models (111M to 13B parameters) trained using the Chinchilla formula on Cerebras CS-2 systems. It establishes a new scaling law based on the public Pile dataset, predicting model performance for downstream tasks. The training leveraged Cerebras's data-parallel weight streaming architecture on a 16x CS-2 cluster, simplifying distributed training and enabling researchers to focus on ML design.
  38. Cerebras Announces Fine-Tuning on the Cerebras AI Model Studio - CerebrasIntroduces the Cerebras AI Model Studio for fine-tuning large language models (GPT-J, GPT-NeoX, CodeGen), offering faster (up to 8x) and cheaper training compared to traditional cloud GPUs. Highlights cost savings (e.g., $0.00096/1k tokens for GPT-J 6B) and reduced training time (e.g., 2.3 days for GPT-NeoX 10B tokens on Cerebras vs. 19 days on cloud). Emphasizes data security and user ownership of trained weights. Provides a simplified path for users to transition from fine-tuning to training from scratch.
  39. Accelerating Large Language Model Training with Variable Sparse Pre-training and Dense Fine-tuning - CerebrasIntroduces Variable SPDF, an extension of the SPDF framework, which incorporates densification during pre-training (Stage-2) after an initial sparse pre-training phase (Stage-1). This approach aims to improve task-agnostic representations and provide fine-grained control over total pre-training FLOPs. Experiments show that Variable SPDF can match downstream metrics of dense baselines with 64% fewer pre-training FLOPs for 6.7B parameter models, outperforming the original SPDF framework at the same effective sparsity level.
  40. Efficient Large-Scale GPT Training Using a Cerebras Wafer-Scale Cluster - CerebrasThis post details the replication of pre-training a GPT-3 XL (1.3 billion parameters) model on the Pile dataset for 112B tokens using a Cerebras Wafer-Scale Cluster. It highlights the ease of scaling on an 8 x CS-2 cluster with a data-parallel weight streaming architecture and reports zero-shot evaluation accuracy across 24 downstream tasks, demonstrating comparable results to a state-of-the-art external baseline.
  41. Introducing Sparse Llama: 70% Smaller, 3x Faster, Full Accuracy - CerebrasIntroduces Sparse Llama, a 70% sparser and 3x faster LLM with full accuracy recovery. Details the collaboration with Neural Magic, combining Cerebras CS-3's native unstructured sparsity support with Neural Magic's DeepSparse engine for accelerated inference. Explains the sparse fine-tuning process involving one-shot pruning, sparse pretraining on SlimPajama, and task-specific fine-tuning. Highlights the limitations of GPU sparsity (2:4 structured sparsity) versus Cerebras's arbitrary unstructured sparsity support.
  42. Cerebras CS-3 vs. Nvidia B200: 2024 AI Accelerators Compared - CerebrasThis post compares the Cerebras CS-3 with the Nvidia B200 for large AI model training. It details the CS-3's hardware architecture, including its wafer-scale engine with 900,000 AI cores and 125 petaflops of FP16 AI compute. It emphasizes the CS-3's disaggregated memory architecture with MemoryX, offering up to 1.2PB of external memory, which is crucial for training trillion-parameter models and simplifies the programming model by presenting a unified memory block. The post also highlights the CS-3's on-wafer interconnect fabric providing 27 PB/s of aggregate bandwidth and its superior performance per watt compared to the B200.
  43. Cerebras CS-3: the world’s fastest and most scalable AI accelerator - CerebrasIntroduces the Cerebras CS-3, a third-generation wafer-scale AI accelerator. Highlights its 4 trillion transistors, 2x performance increase over CS-2, and support for clusters of up to 2048 systems via SwarmX interconnect. Details memory scaling with up to 1,200 TB MemoryX units for training 24 trillion parameter models. Mentions the Condor Galaxy 3 supercomputer (64 CS-3 systems, 8 exaflops). Notes a partnership with Qualcomm for 10x faster LLM inference.
  44. Cerebras Software Release 2.0: 50% Faster Training, PyTorch 2.0 Support, Diffusion Transformers, and MoreRelease 2.0 introduces dynamic memory offload for activations, allowing for larger batch sizes and a 50% performance improvement in training GPT-3 models. It also integrates PyTorch 2.0 support through a new backend based on Lazy Tensor Core (LTC) and Torch-MLIR, providing a standard PyTorch 2.0 interface. New model support includes diffusion transformers (DiT) with AdaLN-Zero, and expanded support for Jais, BTLM, LLaMA v2, Falcon 40B, StarCoder 15B, and 3D U-Net. The release also enhances dynamic sparsity support with the introduction of Sparse Pretraining and Dense Finetuning (SPDF).
  45. Jais: a New Pinnacle in Open Arabic NLPThis post introduces Jais, a 13-billion parameter Arabic-centric LLM trained on the Condor Galaxy 1 (CG-1) supercomputer. It highlights the model's strong performance on Arabic and English tasks, achieved through a bilingual vocabulary, ALiBi positional encodings, SwiGLU activation functions, and hyperparameter optimization using muP. The training dataset comprises approximately 33% Arabic data, balancing linguistic and cultural nuances with reasoning capabilities from English and code data. The post also details the technical overview of the model architecture and the pretraining process.
  46. How we fine-tuned Llama2-70B to pass the US Medical License Exam in a weekThis post details the fine-tuning of Llama-2 70B on a custom medical dataset to pass the US Medical License Exam (USMLE). It highlights the challenges of fine-tuning large models (70B parameters) on traditional GPU hardware due to memory constraints (requiring ~1TB for parameters and optimizer states), contrasting it with the ease of training on Cerebras CS-2 systems. The Cerebras platform's disaggregated memory architecture (12TB available) easily accommodates the model's memory requirements, allowing for straightforward training without complex memory management techniques like LoRA. The project was completed within a week, with the 70B model fine-tuning starting Friday and finishing Tuesday, achieving a 72% score on USMLE sample questions, outperforming Med-PaLM and ClinicalCamel-70B. The post emphasizes the simplicity and efficiency of the Cerebras AI training platform for large model fine-tuning.
  47. Harnessing the Power of Sparsity for Large GPT AI ModelsThis post details how unstructured sparsity can be leveraged for both faster training and faster inference of large GPT models. For training, it demonstrates pre-training with high sparsity at a fraction of the training FLOPs while preserving downstream accuracy using dense fine-tuning. For inference, it shows iterative pruning can create extremely sparse models with significantly fewer FLOPs and memory footprint without loss of accuracy. The Cerebras CS-2's architecture, with its on-chip memory bandwidth and dataflow execution, is highlighted as a key enabler for accelerating unstructured sparsity.
  48. Cerebras Makes It Easy to Harness the Predictive Power of GPT-JThis post details the ease of fine-tuning the 6B parameter GPT-J model on a single Cerebras CS-2 system. It highlights the use of weight streaming execution to overcome the 96GB memory requirement for Adam optimizer states, enabling continuous training and fine-tuning on custom datasets. The post also discusses GPT-J's architectural differences, such as parallel decoder architecture and rotary position embeddings, and their implications for training.
  49. Can Sparsity Make AI Models More Accurate?Introduces Sparse Iso-FLOP Transformation (Sparse-IFT), a technique to increase AI model accuracy by increasing layer size and inducing sparsity to maintain compute requirements. Demonstrates accuracy gains of up to 3.5% on computer vision models and 0.4 perplexity improvement on language models. Highlights Cerebras hardware's native support for unstructured sparsity and its push-button software interface for applying sparsity techniques.