Cultural Benchmarking of LLMs in Standard and Dialectal Arabic Dialogues

There is a significant gap in evaluating cultural reasoning in LLMs using conversational datasets that capture culturally rich and dialectal contexts. Most Arabic benchmarks focus on short text snippets in Modern Standard Arabic (MSA), overlooking the cultural nuances that naturally arise in dialogues. To address this gap, we introduce ArabCulture-Dialogue, a culturally grounded conversational dataset covering 13 Arabic-speaking countries, in both MSA and each country's respective dialect, spanning 12 daily-life topics and 54 finegrained subtopics. We utilize the dataset to form three benchmarking tasks: (i) multiplechoice cultural reasoning, (ii) machine translation between MSA and dialects, and (iii) dialect-steering generation. Our experiments indicate that the performance gap between MSA and Arabic dialects still exists, whereby the models perform worse on all three tasks in the dialectal setup, compared to the MSA one. Recent years have seen substantial progress in Arabic NLP, with the emergence of Arabic-centric LLMs such as Jais (Sengupta et al., 2023), SILMA (SILMA-AI, 2024), and ALLaM (Bari et al., 2025) , alongside multilingual models that increasingly support Arabic. Evaluation benchmarks also have expanded accordingly.