Genomics in Unparalleled Resolution: Cerebras Wafer-Scale Cluster Trains Large Language Models on the Full COVID Genome Sequence - Cerebras

This post details the successful training of large language models (LLMs) on full genomic sequences, such as the SARS-CoV-2 genome, using Cerebras Wafer-Scale Clusters. It highlights overcoming memory and compute limitations of traditional GPU clusters, enabling training of models with billions of parameters and sequence lengths up to 10,240 tokens. The post emphasizes the ease of use of the Cerebras appliance workflow for distributed LLM training and demonstrates linear throughput scaling up to 16 CS-2 nodes, achieving significantly faster time-to-solution compared to GPU-based supercomputers for these specific genomics tasks. The work achieved a time-to-solution of less than one day for GenSLMs with 123M and 1.3B parameters, and demonstrated the ability to train GPT-J models up to 25B parameters with MSL 10,240, achieving linear throughput scaling up to 16 CS-2s.