Lune

ACL Findings2023

DLAMA: A Framework for Curating Culturally Diverse Facts for Probing the Knowledge of Pretrained Language Models

Amr Keleg, Walid Magdy

Abstract

A few benchmarking datasets have been released to evaluate the factual knowledge of pretrained language models. These benchmarks (e.g., LAMA, and ParaRel) are mainly developed in English and later are translated to form new multilingual versions (e.g., mLAMA, and mParaRel). Results on these multilingual benchmarks suggest that using English prompts to recall the facts from multilingual models usually yields significantly better and more consistent performance than using non-English prompts. Our analysis shows that mLAMA is biased toward facts from Western countries, which might affect the fairness of probing models. We propose a new framework for curating factual triples from Wikidata that are culturally diverse. A new benchmark DLAMA-v1 is built of factual triples from three pairs of contrasting cultures having a total of 78,259 triples from 20 relation predicates. The three pairs comprise facts representing the (Arab and Western), (Asian and Western), and (South American and Western) countries respectively. Having a more balanced benchmark (DLAMA-v1) supports that mBERT performs better on Western facts than non-Western ones, while monolingual Arabic, English, and Korean models tend to perform better on their culturally proximate facts. Moreover, both monolingual and multilingual models tend to make a prediction that is culturally or geographically relevant to the correct label, even if the prediction is wrong.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

15 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • Evaluating and Improving Cultural Awareness of Reward Models for LLM Alignment cites Towards Measuring and Modeling "Culture" in LLMs: A Survey
  • Defining Knowledge: Bridging Epistemology and Large Language Models cites Factual Consistency of Multilingual Pretrained Language Models
  • ARBERT & MARBERT: Deep Bidirectional Transformers for Arabic cites Language Models are Few-Shot Learners
  • Culture is Not Trivia: Sociocultural Theory for Cultural NLP cites Towards Measuring and Modeling "Culture" in LLMs: A Survey
  • Knowledgeable or Educated Guess? Revisiting Language Models as Knowledge Bases cites X-FACTR: Multilingual Factual Knowledge Retrieval from Pretrained Language Models
  • Knowledgeable or Educated Guess? Revisiting Language Models as Knowledge Bases cites Language Models are Few-Shot Learners
  • Factual Consistency of Multilingual Pretrained Language Models cites X-FACTR: Multilingual Factual Knowledge Retrieval from Pretrained Language Models
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers15

  1. Culture is Not Trivia: Sociocultural Theory for Cultural NLPNaitian Zhou, David Bamman, Isaac L. BleamanACL 2025 · 33 citations
  2. Towards Measuring and Modeling "Culture" in LLMs: A SurveyMuhammad Farid Adilazuarda, Sagnik Mukherjee, Pradhyumna Lavania, Siddhant Singh et al.EMNLP 2024 · 21 citations
  3. Modular Pluralism: Pluralistic Alignment via Multi-LLM CollaborationShangbin Feng, Taylor Sorensen, Yuhan Liu, Jillian Fisher et al.EMNLP 2024 · 12 citations
  4. Defining Knowledge: Bridging Epistemology and Large Language ModelsConstanza Fierro, Ruchira Dhar, Filippos Stamatiou, Nicolas Garneau et al.EMNLP 2024 · 5 citations
  5. Evaluating and Improving Cultural Awareness of Reward Models for LLM AlignmentHongbin Zhang, Kehai Chen, Xuefeng Bai, Yang Xiang et al.ICLR 2026 · 4 citations

Builds on5

  1. Language Models are Few-Shot LearnersTom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah et al.NeurIPS 2020 · 64,255 citations
  2. X-FACTR: Multilingual Factual Knowledge Retrieval from Pretrained Language ModelsZhengbao Jiang, Antonios Anastasopoulos, Jun Araki, Haibo Ding et al.EMNLP 2020 · 81 citations
  3. Factual Consistency of Multilingual Pretrained Language ModelsConstanza Fierro, Anders SøgaardACL Findings 2022 · 16 citations
  4. ARBERT & MARBERT: Deep Bidirectional Transformers for ArabicMuhammad Abdul-Mageed, AbdelRahim A. Elmadany, El Moatez Billah NagoudiACL 2021
  5. Knowledgeable or Educated Guess? Revisiting Language Models as Knowledge BasesBoxi Cao, Hongyu Lin, Xianpei Han, Le Sun et al.ACL 2021

Related papers

  1. GeoMLAMA: Geo-Diverse Commonsense Probing on Multilingual Pre-Trained Language ModelsDa Yin, Hritik Bansal, Masoud Monajatipoor, Liunian Harold Li et al.EMNLP 2022 · 27 citations
  2. mLUKE: The Power of Entity Representations in Multilingual Pretrained Language ModelsRyokan Ri, Ikuya Yamada, Yoshimasa TsuruokaACL 2022 · 34 citations
  3. MAKIEval: A Multilingual Automatic WiKidata-based Framework for Cultural Awareness Evaluation for LLMsRaoyuan Zhao, Beiduo Chen, Barbara Plank, Michael A. HedderichEMNLP Findings 2025
  4. Having Beer after Prayer? Measuring Cultural Bias in Large Language ModelsTarek Naous, Michael J. Ryan, Alan Ritter, Wei XuACL 2024
  5. Cross-Lingual Consistency of Factual Knowledge in Multilingual Language ModelsJirui Qi, Raquel Fernández, Arianna BisazzaEMNLP 2023 · 9 citations
Dusk over the sea between two cliffs drawn in fine vertical lines