EMNLP Findings2024
Extrinsic Evaluation of Cultural Competence in Large Language Models
Shaily Bhatt, Fernando Diaz
Abstract
Productive interactions between diverse users and language technologies require outputs from the latter to be culturally relevant and sensitive. Prior works have evaluated models' knowledge of cultural norms, values, and artefacts, without considering how this knowledge manifests in downstream applications. In this work, we focus on extrinsic evaluation of cultural competence in two text generation tasks, open-ended question answering and story generation. We quantitatively and qualitatively evaluate model outputs when an explicit cue of culture, specifically nationality, is perturbed in the prompts.
Although we find that model outputs do vary when varying nationalities and feature culturally relevant words, we also find weak correlations between text similarity of outputs for different countries and the cultural values of these countries. Finally, we discuss important considerations in designing comprehensive evaluation of cultural competence in user-facing tasks.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
11 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- DIWALI - Diversity and Inclusivity aWare cuLture specific Items for India: Dataset and Assessment of LLMs for Cultural Text Adaptation in Indian Context cites Investigating Cultural Alignment of Large Language Models
- Distributional Open-Ended Evaluation of LLM Cultural Value Alignment Based on Value Codebook cites Investigating Cultural Alignment of Large Language Models
- Culture is Not Trivia: Sociocultural Theory for Cultural NLP cites Investigating Cultural Alignment of Large Language Models
- Culture is Not Trivia: Sociocultural Theory for Cultural NLP cites Knowledge of cultural moral norms in large language models
- TALES: A Taxonomy and Analysis of Cultural Representations in LLM-generated Stories cites Investigating Cultural Alignment of Large Language Models
- TALES: A Taxonomy and Analysis of Cultural Representations in LLM-generated Stories cites Inspecting the Geographical Representativeness of Images from Text-to-Image Models
- TALES: A Taxonomy and Analysis of Cultural Representations in LLM-generated Stories cites WildChat: 1M ChatGPT Interaction Logs in the Wild
Cited by top-tier papers11
- Culture is Not Trivia: Sociocultural Theory for Cultural NLPNaitian Zhou, David Bamman, Isaac L. BleamanACL 2025 · 33 citations
- Advancing Social Intelligence in AI Agents: Technical Challenges and Open QuestionsLeena Mathur, Paul Pu Liang, Louis-Philippe MorencyEMNLP 2024 · 6 citations
- Distributional Open-Ended Evaluation of LLM Cultural Value Alignment Based on Value CodebookJaehyeok Lee, Xiaoyuan Yi, Jing Yao, Hyunjin Hwang et al.ICML 2026 · 1 citation
- TALES: A Taxonomy and Analysis of Cultural Representations in LLM-generated StoriesKirti Bhagat, Shaily Bhatt, Athul Velagapudi, Aditya Vashistha et al.CHI 2026 · 1 citation
- DIWALI - Diversity and Inclusivity aWare cuLture specific Items for India: Dataset and Assessment of LLMs for Cultural Text Adaptation in Indian ContextPramit Sahoo, Maharaj Brahma, Maunendra Sankar DesarkarEMNLP 2025
Builds on18
- BERTScore: Evaluating Text Generation with BERTTianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger et al.ICLR 2020 · 8,443 citations
- WildChat: 1M ChatGPT Interaction Logs in the WildWenting Zhao, Xiang Ren, Jack Hessel, Claire Cardie et al.ICLR 2024 · 504 citations
- Inspecting the Geographical Representativeness of Images from Text-to-Image ModelsAbhipsa Basu, R. Venkatesh Babu, Danish PruthiICCV 2023 · 54 citations
- Knowledge of cultural moral norms in large language modelsAida Ramezani, Yang XuACL 2023 · 44 citations
- Investigating Cultural Alignment of Large Language ModelsBadr AlKhamissi, Muhammad N. ElNokrashy, Mai Alkhamissi, Mona T. DiabACL 2024 · 27 citations
Related papers
- How Hypocritical Is Your LLM judge? Listener-Speaker Asymmetries in the Pragmatic Competence of Large Language ModelsJudith Sieker, Sina ZarrießACL Findings 2026
- Can Large Language Models Be an Alternative to Human Evaluations?David Cheng-Han Chiang, Hung-yi LeeACL 2023 · 254 citations
- The Perils of Using Mechanical Turk to Evaluate Open-Ended Text GenerationMarzena Karpinska, Nader Akoury, Mohit IyyerEMNLP 2021 · 3 citations
- MAKIEval: A Multilingual Automatic WiKidata-based Framework for Cultural Awareness Evaluation for LLMsRaoyuan Zhao, Beiduo Chen, Barbara Plank, Michael A. HedderichEMNLP Findings 2025
- SocialCC: Interactive Evaluation for Cultural Competence in Language AgentsJincenzi Wu, Jianxun Lian, Dingdong Wang, Helen M. MengACL 2025 · 8 citations
