Digital Event Horizon
Alyah, a comprehensive benchmark for evaluating Arabic LLMs' capabilities in understanding the Emirati dialect, aims to provide a more realistic and culturally grounded assessment of these models. By addressing the gap in dialect-specific evaluations, Alyah supports the development of models better suited to local communities and institutions.
The Emirati dialect has been overlooked in the field of natural language processing (NLP) and large language models (LLMs). A new benchmark, Alyah, has been introduced to evaluate Arabic LLMs' capabilities in understanding and utilizing the Emirati dialect. The Alyah benchmark is built on a dataset of 1,173 manually curated samples from native Emirati speakers. Instruction-tuned models generally outperform base models, especially in dialect-specific language phenomena. Large instruct models perform better than smaller models in Poetry and Creative Expression categories. The Alyah benchmark represents a significant step toward more realistic and culturally grounded evaluation of Arabic LLMs.
The field of natural language processing (NLP) has witnessed significant advancements in recent years, with large language models (LLMs) emerging as a dominant force in this domain. However, a crucial aspect of LLMs' performance has long been overlooked: their ability to capture and comprehend the nuances of regional dialects. The Emirati dialect, in particular, has been a subject of interest due to its unique cultural and linguistic characteristics.
In an effort to address this gap, researchers from tiiuae have introduced Alyah, a comprehensive benchmark designed specifically for evaluating Arabic LLMs' capabilities in understanding and utilizing the Emirati dialect. This groundbreaking initiative aims to provide a more realistic and culturally grounded assessment of LLMs, thereby supporting the development of models that better serve local communities and institutions.
The Alyah benchmark is built upon a dataset comprising 1,173 samples, all collected manually from native Emirati speakers to ensure linguistic authenticity and cultural grounding. This manual curation step was essential in capturing expressions, meanings, and usages that are rarely documented in written resources and are difficult to infer from Modern Standard Arabic alone.
Each sample is formulated as a multiple-choice question with four candidate answers, exactly one of which is correct. Large language models were used to synthetically generate the distractor choices, after which they were reviewed to ensure plausibility and semantic closeness to the correct answer. To avoid positional bias during evaluation, the index of the correct answer follows a randomized distribution across the dataset.
The Alyah benchmark spans a broad spectrum of linguistic and cultural phenomena in the Emirati dialect, ranging from everyday expressions to culturally sensitive and figurative language. The distribution across categories is summarized as follows:
* Category
* Number of Samples
* Difficulty
Greetings & Daily Expressions: 61 Easy
Religious & Social Sensitivity: 78 Medium
Imagery & Figurative Meaning: 121 Medium
Etiquette & Values: 173 Medium
Poetry & Creative Expression: 32 Difficult
Historical & Heritage Knowledge: 89 Difficult
The Alyah dataset is complemented by a comprehensive model evaluation setup, which includes a total of 54 language models comprising 23 base models and 31 instruction-tuned models. These models span several architectural and training paradigms, including Arabic-native LLMs such as Jais and Allam, multilingual models with strong Arabic support such as Qwen and LLaMA, and adapted or regionally specialized models like Fanar and AceGPT.
The model evaluation setup is designed to assess the performance of these models on Alyah, using accuracy on multiple-choice questions as the primary metric. The results are intended to serve as reference measurements within the scope of Alyah, rather than absolute rankings across all Arabic benchmarks.
In terms of analysis and observed trends, several insights emerge from the evaluation. Instruction-tuned models generally outperform their base counterparts, with a particular emphasis on dialect-specific language phenomena that remain challenging for current models. The most difficult categories for the models were consistently "Language and Dialect" and "Greetings & Daily Expressions," reflecting the novelty of these dialects relative to the evaluated models.
Furthermore, it is observed that large instruct models perform marginally better than smaller models in Poetry and Creative Expression categories, indicating a clear benefit to instruction tuning in understanding dialect-specific nuances. The high variance in performance across categories highlights a multi-dimensional nature of dialectal competence, underscoring the need for more nuanced evaluations.
The Alyah benchmark represents a significant step toward more realistic and culturally grounded evaluation of Arabic language models. By focusing on the Emirati dialect, this initiative aims to support the development of models that better serve local communities, institutions, and users in the UAE. Researchers, practitioners, and the broader community are invited to use the benchmark, explore the results, and share feedback.
The Alyah dataset will be available on HuggingFace, while code implementation is hosted on GitHub. For those interested in delving deeper into this research, a comprehensive citation is provided below:
@misc{emirati_dialect_benchmark_2026,
title = {Alyah: An Emirati Dialect Benchmark for Evaluating Arabic Large Language Models},
author={Omar Alkaabi and Ahmed Alzubaidi and Hamza Alobeidli and Shaikha Alsuwaidi and Mohammed Alyafeai and Leen AlQadi and Basma El Amel Boussaha and Hakim Hacid},
year = {2026},
month = {january},
}
Related Information:
https://www.digitaleventhorizon.com/articles/Alyah-A-Breakthrough-in-Evaluating-Emirati-Dialect-Capabilities-in-Arabic-Large-Language-Models-deh.shtml
https://huggingface.co/blog/tiiuae/emirati-benchmarks
Published: Tue Jan 27 04:48:14 2026 by llama3.2 3B Q4_K_M
