EMNLP Findings2025
MAGIC: A Multi-Hop and Graph-Based Benchmark for Inter-Context Conflicts in Retrieval-Augmented Generation
Jungyeon Lee, Kangmin Lee, Taeuk Kim
Abstract
Knowledge conflict often arises in retrievalaugmented generation (RAG) systems, where retrieved documents may be inconsistent with one another or contradict the model's parametric knowledge. Existing benchmarks for investigating the phenomenon have notable limitations, including a narrow focus on the question answering setup, heavy reliance on entity substitution techniques, and a restricted range of conflict types. To address these issues, we propose a knowledge graph (KG)-based framework that generates varied and subtle conflicts between two similar yet distinct contexts, while ensuring interpretability through the explicit relational structure of KGs. Experimental results on our benchmark, MAGIC, provide intriguing insights into the inner workings of LLMs regarding knowledge conflict: both open-source and proprietary models struggle with conflict detection-especially when multi-hop reasoning is required-and often fail to pinpoint the exact source of contradictions. Finally, we present in-depth analyses that serve as a foundation for improving LLMs in integrating diverse, sometimes even conflicting, information.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
Lineage
- Adaptive Chameleon or Stubborn Sloth: Revealing the Behavior of Large Language Models in Knowledge Conflicts cites Locating and Editing Factual Associations in GPT
- Knowledge Conflicts for LLMs: A Survey cites Locating and Editing Factual Associations in GPT
- Knowledge Conflicts for LLMs: A Survey cites Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
Builds on12
- Retrieval-Augmented Generation for Knowledge-Intensive NLP TasksPatrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni et al.NeurIPS 2020 · 19,162 citations
- Locating and Editing Factual Associations in GPTKevin Meng, David Bau, Alex Andonian, Yonatan BelinkovNeurIPS 2022 · 3,415 citations
- Adaptive Chameleon or Stubborn Sloth: Revealing the Behavior of Large Language Models in Knowledge ConflictsJian Xie, Kai Zhang, Jiangjie Chen, Renze Lou et al.ICLR 2024 · 294 citations
- KG-GPT: A General Framework for Reasoning on Knowledge Graphs Using Large Language ModelsJiho Kim, Yeonsu Kwon, Yohan Jo, Edward ChoiEMNLP Findings 2023 · 47 citations
- Knowledge Conflicts for LLMs: A SurveyRongwu Xu, Zehan Qi, Zhijiang Guo, Cunxiang Wang et al.EMNLP 2024 · 38 citations
Related papers
- Exploring Knowledge Conflicts for Faithful LLM Reasoning: Benchmark and MethodTianzhe Zhao, Jiaoyan Chen, Shuxiu Zhang, Haiping Zhu et al.SIGIR 2026
- TruthfulRAG: Resolving Factual-level Conflicts in Retrieval-Augmented Generation with Knowledge GraphsShuyi Liu, Yu-Ming Shang, Xi ZhangAAAI 2026 · 2 citations
- Benchmarking Multimodal Knowledge Conflict for Large Multimodal ModelsYifan Jia, Yuntao Du, Kailin Jiang, Yuyang Liang et al.AAAI 2026
- Micro-Act: Mitigate Knowledge Conflict in Question Answering via Actionable Self-ReasoningNan Huo, Jinyang Li, Bowen Qin, Ge Qu et al.ACL 2025
- FaithfulRAG: Fact-Level Conflict Modeling for Context-Faithful Retrieval-Augmented GenerationQinggang Zhang, Zhishang Xiang, Yilin Xiao, Le Wang et al.ACL 2025 · 18 citations
