Falcon-Emirati: When an LLM Learns the Dialect, the Culture, and the Nuance

This post details the development of Falcon-Emirati-7B, a dialect-specialized LLM built on the Falcon-H1-Arabic model. It highlights the challenges of dialect adaptation due to the spoken nature of Emirati Arabic and the reliance on cultural context. The data pipeline involved curating authentic dialectal web data, MSA data on Emirati culture, and synthetically generated data with strict rules. The 7B parameter size was selected for its balance of quality and cost. Evaluation methodologies included manual and LLM-as-judge assessments. The underlying Falcon-H1 architecture, which fuses Mamba and Transformer attention, is also discussed as a foundation for handling morphologically rich languages and long contexts.