ACL Findings2025
COMPKE: Complex Question Answering under Knowledge Editing
Keyuan Cheng, Zijian Kan, Zhuoran Zhang, Muhammad Asif Ali, Lijie Hu, Di Wang
Abstract
Knowledge Editing-Efficiently modifying the knowledge in large language models has gathered great attention.
Current benchmarks primarily use multi-hop question answering to assess and analyze newly injected or updated knowledge. However, we argue that these benchmarks fail to effectively evaluate how well the updated models apply this knowledge in real-life scenarios, particularly when questions require complex reasoning, involving one-to-many relationships or multi-step logical intersections. To fill in this gap, we introduce a new benchmark, COMPKE: Complex Question Answering under Knowledge Editing, which includes 11,924 complex questions that reflect real-life situations. We conduct an extensive evaluation of four knowledge editing methods on COMPKE, revealing that their effectiveness varies notably across different models. For instance, MeLLo attains an accuracy of 39.47 on GPT-4O-MINI, but this drops sharply to 3.83 on QWEN2.5-3B. We further investigate the underlying causes of these disparities from both methodological and model-specific perspectives. The datasets are available at https://github.com/ kzjkzj666/CompKE.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
1 top-tier paper has cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- Aging with GRACE: Lifelong Model Editing with Discrete Key-Value Adaptors cites LoRA: Low-Rank Adaptation of Large Language Models
- Aging with GRACE: Lifelong Model Editing with Discrete Key-Value Adaptors cites Memory-Based Model Editing at Scale
- Aging with GRACE: Lifelong Model Editing with Discrete Key-Value Adaptors cites Does Localization Inform Editing? Surprising Differences in Causality-Based Localization vs. Knowledge Editing in Language Models
- Does Localization Inform Editing? Surprising Differences in Causality-Based Localization vs. Knowledge Editing in Language Models cites Memory-Based Model Editing at Scale
Cited by top-tier papers1
Builds on27
- LoRA: Low-Rank Adaptation of Large Language ModelsEdward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu et al.ICLR 2022 · 18,833 citations
- Memory-Based Model Editing at ScaleEric Mitchell, Charles Lin, Antoine Bosselut, Christopher D. Manning et al.ICML 2022 · 520 citations
- Aging with GRACE: Lifelong Model Editing with Discrete Key-Value AdaptorsTom Hartvigsen, Swami Sankaranarayanan, Hamid Palangi, Yoon Kim et al.NeurIPS 2023 · 349 citations
- Does Localization Inform Editing? Surprising Differences in Causality-Based Localization vs. Knowledge Editing in Language ModelsPeter Hase, Mohit Bansal, Been Kim, Asma GhandehariounNeurIPS 2023 · 307 citations
- DetectLLM: Leveraging Log Rank Information for Zero-Shot Detection of Machine-Generated TextJinyan Su, Terry Yue Zhuo, Di Wang, Preslav NakovEMNLP Findings 2023 · 80 citations
Related papers
- MQuAKE: Assessing Knowledge Editing in Language Models via Multi-Hop QuestionsZexuan Zhong, Zhengxuan Wu, Christopher D. Manning, Christopher Potts et al.EMNLP 2023 · 36 citations
- MMKE-Bench: A Multimodal Editing Benchmark for Diverse Visual KnowledgeYuntao Du, Kailin Jiang, Zhi Gao, Chenrui Shi et al.ICLR 2025
- ScEdit: Script-based Assessment of Knowledge EditingXinye Li, Zunwen Zheng, Qian Zhang, Dekai Zhuang et al.ACL Findings 2025
- MIKE: A New Benchmark for Fine-grained Multimodal Entity Knowledge EditingJiaqi Li, Miaozeng Du, Chuanyi Zhang, Yongrui Chen et al.ACL Findings 2024 · 5 citations
- MQuAKE-Remastered: Multi-Hop Knowledge Editing Can Only Be Advanced with Reliable EvaluationsShaochen (Henry) Zhong, Yifan Lu, Lize Shao, Bhargav Bhushanam et al.ICLR 2025
