Jais: a New Pinnacle in Open Arabic NLP

This post introduces Jais, a 13-billion parameter Arabic-centric LLM trained on the Condor Galaxy 1 (CG-1) supercomputer. It highlights the model's strong performance on Arabic and English tasks, achieved through a bilingual vocabulary, ALiBi positional encodings, SwiGLU activation functions, and hyperparameter optimization using muP. The training dataset comprises approximately 33% Arabic data, balancing linguistic and cultural nuances with reasoning capabilities from English and code data. The post also details the technical overview of the model architecture and the pretraining process.