ACL Findings2023
DLAMA: A Framework for Curating Culturally Diverse Facts for Probing the Knowledge of Pretrained Language Models
摘要
A few benchmarking datasets have been released to evaluate the factual knowledge of pretrained language models. These benchmarks (e.g., LAMA, and ParaRel) are mainly developed in English and later are translated to form new multilingual versions (e.g., mLAMA, and mParaRel). Results on these multilingual benchmarks suggest that using English prompts to recall the facts from multilingual models usually yields significantly better and more consistent performance than using non-English prompts. Our analysis shows that mLAMA is biased toward facts from Western countries, which might affect the fairness of probing models. We propose a new framework for curating factual triples from Wikidata that are culturally diverse. A new benchmark DLAMA-v1 is built of factual triples from three pairs of contrasting cultures having a total of 78,259 triples from 20 relation predicates. The three pairs comprise facts representing the (Arab and Western), (Asian and Western), and (South American and Western) countries respectively. Having a more balanced benchmark (DLAMA-v1) supports that mBERT performs better on Western facts than non-Western ones, while monolingual Arabic, English, and Korean models tend to perform better on their culturally proximate facts. Moreover, both monolingual and multilingual models tend to make a prediction that is culturally or geographically relevant to the correct label, even if the prediction is wrong.
问 Lune
几分钟读透这篇 Paper,不用再花一下午。
Lune 能读完全文,也能读引用它的顶会 Paper,讲清它的结论、证据和局限,每一点都注明出自哪篇 Paper。不用硬啃,也不靠猜。
已有 15 篇顶会 Paper 引用了它。别等审稿人问起,先弄清它们发现了什么。
脉络
- Evaluating and Improving Cultural Awareness of Reward Models for LLM Alignment 引用了 Towards Measuring and Modeling "Culture" in LLMs: A Survey
- Defining Knowledge: Bridging Epistemology and Large Language Models 引用了 Factual Consistency of Multilingual Pretrained Language Models
- ARBERT & MARBERT: Deep Bidirectional Transformers for Arabic 引用了 Language Models are Few-Shot Learners
- Culture is Not Trivia: Sociocultural Theory for Cultural NLP 引用了 Towards Measuring and Modeling "Culture" in LLMs: A Survey
- Knowledgeable or Educated Guess? Revisiting Language Models as Knowledge Bases 引用了 X-FACTR: Multilingual Factual Knowledge Retrieval from Pretrained Language Models
- Knowledgeable or Educated Guess? Revisiting Language Models as Knowledge Bases 引用了 Language Models are Few-Shot Learners
- Factual Consistency of Multilingual Pretrained Language Models 引用了 X-FACTR: Multilingual Factual Knowledge Retrieval from Pretrained Language Models
引用它的顶会 Paper15
- Culture is Not Trivia: Sociocultural Theory for Cultural NLPNaitian Zhou, David Bamman, Isaac L. BleamanACL 2025 · 33 次引用
- Towards Measuring and Modeling "Culture" in LLMs: A SurveyMuhammad Farid Adilazuarda, Sagnik Mukherjee, Pradhyumna Lavania, Siddhant Singh 等EMNLP 2024 · 21 次引用
- Modular Pluralism: Pluralistic Alignment via Multi-LLM CollaborationShangbin Feng, Taylor Sorensen, Yuhan Liu, Jillian Fisher 等EMNLP 2024 · 12 次引用
- Defining Knowledge: Bridging Epistemology and Large Language ModelsConstanza Fierro, Ruchira Dhar, Filippos Stamatiou, Nicolas Garneau 等EMNLP 2024 · 5 次引用
- Evaluating and Improving Cultural Awareness of Reward Models for LLM AlignmentHongbin Zhang, Kehai Chen, Xuefeng Bai, Yang Xiang 等ICLR 2026 · 4 次引用
它引用的 Paper5
- Language Models are Few-Shot LearnersTom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah 等NeurIPS 2020 · 64,255 次引用
- X-FACTR: Multilingual Factual Knowledge Retrieval from Pretrained Language ModelsZhengbao Jiang, Antonios Anastasopoulos, Jun Araki, Haibo Ding 等EMNLP 2020 · 81 次引用
- Factual Consistency of Multilingual Pretrained Language ModelsConstanza Fierro, Anders SøgaardACL Findings 2022 · 16 次引用
- ARBERT & MARBERT: Deep Bidirectional Transformers for ArabicMuhammad Abdul-Mageed, AbdelRahim A. Elmadany, El Moatez Billah NagoudiACL 2021
- Knowledgeable or Educated Guess? Revisiting Language Models as Knowledge BasesBoxi Cao, Hongyu Lin, Xianpei Han, Le Sun 等ACL 2021
相关 Paper
- GeoMLAMA: Geo-Diverse Commonsense Probing on Multilingual Pre-Trained Language ModelsDa Yin, Hritik Bansal, Masoud Monajatipoor, Liunian Harold Li 等EMNLP 2022 · 27 次引用
- mLUKE: The Power of Entity Representations in Multilingual Pretrained Language ModelsRyokan Ri, Ikuya Yamada, Yoshimasa TsuruokaACL 2022 · 34 次引用
- MAKIEval: A Multilingual Automatic WiKidata-based Framework for Cultural Awareness Evaluation for LLMsRaoyuan Zhao, Beiduo Chen, Barbara Plank, Michael A. HedderichEMNLP Findings 2025
- Having Beer after Prayer? Measuring Cultural Bias in Large Language ModelsTarek Naous, Michael J. Ryan, Alan Ritter, Wei XuACL 2024
- Cross-Lingual Consistency of Factual Knowledge in Multilingual Language ModelsJirui Qi, Raquel Fernández, Arianna BisazzaEMNLP 2023 · 9 次引用
