EMNLP Findings2024
DYNAMICQA: Tracing Internal Knowledge Conflicts in Language Models
Sara Marjanovic, Haeun Yu, Pepa Atanasova, Maria Maistro, Christina Lioma, Isabelle Augenstein
Abstract
Knowledge-intensive language understanding tasks require Language Models (LMs) to integrate relevant context, mitigating their inherent weaknesses, such as incomplete or outdated knowledge. However, conflicting knowledge can be present in the LM's parameters, termed intra-memory conflict, which can affect a model's propensity to accept contextual knowledge. To study the effect of intra-memory conflict on an LM's ability to accept relevant context, we utilize two knowledge conflict measures and a novel dataset containing inherently conflicting data, DYNAMICQA. This dataset includes facts with a temporal dynamic nature where facts can change over time and disputable dynamic facts, which can change depending on the viewpoint. DYNAMICQA is the first to include real-world knowledge conflicts and provide context to study the link between the different types of knowledge conflicts. We also evaluate several measures on their ability to reflect the presence of intra-memory conflict: semantic entropy and a novel coherent persuasion score. With our extensive experiments, we verify that LMs exhibit a greater degree of intra-memory conflict with dynamic facts compared to facts that have a single truth value. Furthermore, we reveal that facts with intra-memory conflict are harder to update with context, suggesting that retrieval-augmented generation will struggle with the most commonly adapted facts. copenlu/dynamicqa copenlu/dynamicqa
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
11 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- Adaptive Chameleon or Stubborn Sloth: Revealing the Behavior of Large Language Models in Knowledge Conflicts cites Retrieval Augmented Language Model Pre-Training
- To Trust or Not to Trust? Enhancing Large Language Models' Situated Faithfulness to External Contexts cites Adaptive Chameleon or Stubborn Sloth: Revealing the Behavior of Large Language Models in Knowledge Conflicts
- To Trust or Not to Trust? Enhancing Large Language Models' Situated Faithfulness to External Contexts cites When Not to Trust Language Models: Investigating Effectiveness of Parametric and Non-Parametric Memories
- To Trust or Not to Trust? Enhancing Large Language Models' Situated Faithfulness to External Contexts cites On the Risk of Misinformation Pollution with Large Language Models
- Whose Facts Win? LLM Source Preferences under Knowledge Conflicts cites Adaptive Chameleon or Stubborn Sloth: Revealing the Behavior of Large Language Models in Knowledge Conflicts
- Precise Information Control in Long-Form Text Generation cites Adaptive Chameleon or Stubborn Sloth: Revealing the Behavior of Large Language Models in Knowledge Conflicts
- Precise Information Control in Long-Form Text Generation cites Retrieval Augmented Language Model Pre-Training
- KScope: A Framework for Characterizing the Knowledge Status of Language Models cites Adaptive Chameleon or Stubborn Sloth: Revealing the Behavior of Large Language Models in Knowledge Conflicts
- KScope: A Framework for Characterizing the Knowledge Status of Language Models cites On the Risk of Misinformation Pollution with Large Language Models
- KScope: A Framework for Characterizing the Knowledge Status of Language Models cites Semantic Uncertainty: Linguistic Invariances for Uncertainty Estimation in Natural Language Generation
- On the Risk of Misinformation Pollution with Large Language Models cites Retrieval Augmented Language Model Pre-Training
Cited by top-tier papers11
- Show Me the Work: Fact-Checkers' Requirements for Explainable Automated Fact-CheckingGreta Warren, Irina Shklovski, Isabelle AugensteinCHI 2025 · 15 citations
- Precise Information Control in Long-Form Text GenerationJacqueline He, Howard Yen, Margaret Li, Shuyue Stella Li et al.NeurIPS 2025 · 8 citations
- Whose Facts Win? LLM Source Preferences under Knowledge ConflictsJakob Schuster, Vagrant Gautam, Katja MarkertACL 2026 · 3 citations
- KScope: A Framework for Characterizing the Knowledge Status of Language ModelsYuxin Xiao, Shan Chen, Jack Gallifant, Danielle S. Bitterman et al.NeurIPS 2025 · 3 citations
- To Trust or Not to Trust? Enhancing Large Language Models' Situated Faithfulness to External ContextsYukun Huang, Sanxing Chen, Hongyi Cai, Bhuwan DhingraICLR 2025
Builds on16
- Retrieval Augmented Language Model Pre-TrainingKelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat et al.ICML 2020 · 2,937 citations
- Adaptive Chameleon or Stubborn Sloth: Revealing the Behavior of Large Language Models in Knowledge ConflictsJian Xie, Kai Zhang, Jiangjie Chen, Renze Lou et al.ICLR 2024 · 294 citations
- When Not to Trust Language Models: Investigating Effectiveness of Parametric and Non-Parametric MemoriesAlex Mallen, Akari Asai, Victor Zhong, Rajarshi Das et al.ACL 2023 · 233 citations
- On the Risk of Misinformation Pollution with Large Language ModelsYikang Pan, Liangming Pan, Wenhu Chen, Preslav Nakov et al.EMNLP Findings 2023 · 85 citations
- Semantic Uncertainty: Linguistic Invariances for Uncertainty Estimation in Natural Language GenerationLorenz Kuhn, Yarin Gal, Sebastian FarquharICLR 2023 · 49 citations
Related papers
- Can Language Models Serve as Temporal Knowledge Bases?Ruilin Zhao, Feng Zhao, Guandong Xu, Sixiao Zhang et al.EMNLP Findings 2022 · 5 citations
- Who's Who: Large Language Models Meet Knowledge Conflicts in PracticeQuang Pham, Hoang Ngo, Anh Tuan Luu, Dat Quoc NguyenEMNLP Findings 2024 · 3 citations
- When Facts Change: Temporal Knowledge Conflict Resolution in LLMsJonas Wallat, Wolfgang Nejdl, Sandipan SikdarACL Findings 2026
- Micro-Act: Mitigate Knowledge Conflict in Question Answering via Actionable Self-ReasoningNan Huo, Jinyang Li, Bowen Qin, Ge Qu et al.ACL 2025
- Conflict-Aware Soft Prompting for Retrieval-Augmented GenerationEunseong Choi, June Park, Hyeri Lee, Jongwuk LeeEMNLP 2025 · 1 citation
