Training mRNA Language Models Across 25 Species for $165

This post details the development of an end-to-end protein engineering pipeline, focusing on mRNA optimization. It describes the architecture exploration for codon-level language modeling, comparing CodonBERT, ModernBERT, and RoBERTa variants. CodonRoBERTa-large-v2 was selected as the best model with a perplexity of 4.10 and a Spearman CAI correlation of 0.40. The post also covers scaling to 25 species, training 4 production models in 55 GPU-hours, and building a species-conditioned system. It details the data engineering challenges, tokenization innovations, and the production deployment strategy for these models.