ACL Findings2022
Factual Consistency of Multilingual Pretrained Language Models
Abstract
Pretrained language models can be queried for factual knowledge, with potential applications in knowledge base acquisition and tasks that require inference. However, for that, we need to know how reliable this knowledge is, and recent work has shown that monolingual English language models lack consistency when predicting factual knowledge, that is, they fill-in-theblank differently for paraphrases describing the same fact. In this paper, we extend the analysis of consistency to a multilingual setting. We introduce a resource, MPARAREL 1 , and investigate (i) whether multilingual language models such as mBERT and XLM-R are more consistent than their monolingual counterparts; and (ii) if such models are equally consistent across languages. We find that mBERT is as inconsistent as English BERT in English paraphrases, but that both mBERT and XLM-R exhibit a high degree of inconsistency in English and even more so for all the other 45 languages.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
5 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- Measuring the Effect of Disfluency in Multilingual Knowledge Probing Benchmarks cites AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated Prompts
- Measuring the Effect of Disfluency in Multilingual Knowledge Probing Benchmarks cites X-FACTR: Multilingual Factual Knowledge Retrieval from Pretrained Language Models
- Measuring the Effect of Disfluency in Multilingual Knowledge Probing Benchmarks cites Cross-Lingual Consistency of Factual Knowledge in Multilingual Language Models
- mCoT: Multilingual Instruction Tuning for Reasoning Consistency in Language Models cites Cross-Lingual Consistency of Factual Knowledge in Multilingual Language Models
- AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated Prompts cites Inducing Relational Knowledge from BERT
- X-FACTR: Multilingual Factual Knowledge Retrieval from Pretrained Language Models cites Inducing Relational Knowledge from BERT
- X-FACTR: Multilingual Factual Knowledge Retrieval from Pretrained Language Models cites Unsupervised Cross-lingual Representation Learning at Scale
- Cross-Lingual Consistency of Factual Knowledge in Multilingual Language Models cites Inducing Relational Knowledge from BERT
- Cross-Lingual Consistency of Factual Knowledge in Multilingual Language Models cites AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated Prompts
- Cross-Lingual Consistency of Factual Knowledge in Multilingual Language Models cites X-FACTR: Multilingual Factual Knowledge Retrieval from Pretrained Language Models
Cited by top-tier papers5
- Cross-Lingual Consistency of Factual Knowledge in Multilingual Language ModelsJirui Qi, Raquel Fernández, Arianna BisazzaEMNLP 2023 · 9 citations
- Defining Knowledge: Bridging Epistemology and Large Language ModelsConstanza Fierro, Ruchira Dhar, Filippos Stamatiou, Nicolas Garneau et al.EMNLP 2024 · 5 citations
- mCoT: Multilingual Instruction Tuning for Reasoning Consistency in Language ModelsHuiyuan Lai, Malvina NissimACL 2024 · 5 citations
- Measuring the Effect of Disfluency in Multilingual Knowledge Probing BenchmarksKirill Semenov, Rico SennrichEMNLP 2025 · 2 citations
- Advancing Reliable Test-Time Adaptation of Vision-Language Models under Visual VariationsYiwen Liang, Hui Chen, Yizhe Xiong, Zihan Zhou et al.ACM MM 2025 · 1 citation
Builds on8
- AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated PromptsTaylor Shin, Yasaman Razeghi, Robert L. Logan IV, Eric Wallace et al.EMNLP 2020 · 1,162 citations
- Unsupervised Cross-lingual Representation Learning at ScaleAlexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary et al.ACL 2020 · 539 citations
- Inducing Relational Knowledge from BERTZied Bouraoui, José Camacho-Collados, Steven SchockaertAAAI 2020 · 183 citations
- X-FACTR: Multilingual Factual Knowledge Retrieval from Pretrained Language ModelsZhengbao Jiang, Antonios Anastasopoulos, Jun Araki, Haibo Ding et al.EMNLP 2020 · 81 citations
- From Natural Language Processing to Neural DatabasesJames Thorne, Majid Yazdani, Marzieh Saeidi, Fabrizio Silvestri et al.VLDB 2021 · 62 citations
Related papers
- Bridging the Language Gap: Uncovering and Aligning Shared Circuits for Multi-Hop Reasoning in Multilingual LLMsChenghao Sun, Zhen Huang, Yonggang Zhang, Xinmei Tian et al.AAAI 2026
- Analogy Training Multilingual EncodersNicolas Garneau, Mareike Hartmann, Anders Sandholm, Sebastian Ruder et al.AAAI 2021 · 17 citations
- Lost in Multilinguality: Dissecting Cross-lingual Factual Inconsistency in Transformer Language ModelsMingyang Wang, Heike Adel, Lukas Lange, Yihong Liu et al.ACL 2025
- Tracing Multilingual Factual Knowledge Acquisition in PretrainingYihong Liu, Mingyang Wang, Amir Hossein Kargaran, Felicia Körner et al.EMNLP Findings 2025
- Paths Not Taken: Understanding and Mending the Multilingual Factual Recall PipelineMeng Lu, Ruochen Zhang, Carsten Eickhoff, Ellie PavlickEMNLP 2025 · 15 citations
