Cerebras

This post addresses the challenges of scaling Mixture-of-Experts (MoE) models, particularly memory limitations on GPUs and compute utilization issues on Cerebras WSE. It details the limitations of expert parallelism (EP) on GPUs, including communication overheads and load balancing issues. For the Cerebras WSE, it introduces Batch Tiling on Attention (BTA) as a solution to the compute utilization problem by decoupling batch sizes for attention and expert layers, thereby improving throughput and efficiency as MoE models become sparser.