Lune

ICLR2025Top-tier venue

MMKE-Bench: A Multimodal Editing Benchmark for Diverse Visual Knowledge

Yuntao Du, Kailin Jiang, Zhi Gao, Chenrui Shi, Zilong Zheng, Siyuan Qi, Qing Li

Abstract

Knowledge editing techniques have emerged as essential tools for updating the factual knowledge of large language models (LLMs) and multimodal models (LMMs), allowing them to correct outdated or inaccurate information without retraining from scratch. However, existing benchmarks for multimodal knowledge editing primarily focus on entity-level knowledge represented as simple triplets, which fail to capture the complexity of real-world multimodal information. To address this issue, we introduce MMKE-Bench, a comprehensive MultiModal Knowledge Editing Benchmark, designed to evaluate the ability of LMMs to edit diverse visual knowledge in real-world scenarios. MMKE-Bench addresses these limitations by incorporating three types of editing tasks: visual entity editing, visual semantic editing, and user-specific editing. Besides, MMKE-Bench uses free-form natural language to represent and edit knowledge, offering a more flexible and effective format. The benchmark consists of 2,940 pieces of knowledge and 8,363 images across 33 broad categories, with evaluation questions automatically generated and human-verified. We assess five state-of-the-art knowledge editing methods on three prominent LMMs, revealing that no method excels across all criteria, and that visual and user-specific edits are particularly challenging. MMKE-Bench sets a new standard for evaluating the robustness of multimodal knowledge editing techniques, driving progress in this rapidly evolving field. Project Page: https://mmke-bench-iclr.github.io/

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

11 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • Locating and Editing Factual Associations in GPT cites Language Models are Few-Shot Learners
  • InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning cites BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models
  • InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning cites Language Models are Few-Shot Learners
  • InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning cites MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Models
  • Correct When Paired, Wrong When Split: Decoupling and Editing Modality-Specific Neurons in MLLMs cites InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning
  • Correct When Paired, Wrong When Split: Decoupling and Editing Modality-Specific Neurons in MLLMs cites BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models
  • MedMKEB: A Comprehensive Knowledge Editing Benchmark for Medical Multimodal Large Language Models cites Locating and Editing Factual Associations in GPT
  • MedMKEB: A Comprehensive Knowledge Editing Benchmark for Medical Multimodal Large Language Models cites BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models
  • MedMKEB: A Comprehensive Knowledge Editing Benchmark for Medical Multimodal Large Language Models cites MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Models
  • BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models cites Language Models are Few-Shot Learners
  • When Large Multimodal Models Confront Evolving Knowledge: Challenges and Explorations cites Locating and Editing Factual Associations in GPT
  • When Large Multimodal Models Confront Evolving Knowledge: Challenges and Explorations cites InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning
  • When Large Multimodal Models Confront Evolving Knowledge: Challenges and Explorations cites BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models
  • When Large Multimodal Models Confront Evolving Knowledge: Challenges and Explorations cites Language Models are Few-Shot Learners
  • When Large Multimodal Models Confront Evolving Knowledge: Challenges and Explorations cites MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Models
  • MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Models cites BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models
  • MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Models cites Language Models are Few-Shot Learners
  • MMKU-Bench: A Multimodal Update Benchmark for Diverse Visual Knowledge cites Locating and Editing Factual Associations in GPT
  • MMKU-Bench: A Multimodal Update Benchmark for Diverse Visual Knowledge cites BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers11

  1. When Large Multimodal Models Confront Evolving Knowledge: Challenges and ExplorationsKailin Jiang, Yuntao Du, Yukai Ding, Yuchen Ren et al.ICLR 2026 · 7 citations
  2. MMKU-Bench: A Multimodal Update Benchmark for Diverse Visual KnowledgeBaochen Fu, Yuntao Du, Cheng Chang, Baihao Jin et al.ICML 2026 · 7 citations
  3. MedMKEB: A Comprehensive Knowledge Editing Benchmark for Medical Multimodal Large Language ModelsDexuan Xu, Jieyi Wang, Zhongyan Chai, Yongzhi Cao et al.AAAI 2026 · 1 citation
  4. Editing the Moving World: Model Editing for Video LLMsQian Zhang, Xinye Li, Xiaokai Wu, Junhao Xu et al.ACL 2026
  5. Correct When Paired, Wrong When Split: Decoupling and Editing Modality-Specific Neurons in MLLMsTingchao Fu, Wenkai Wang, Fanxiao Li, Huadong Zhang et al.ACL 2026

Builds on20

  1. Language Models are Few-Shot LearnersTom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah et al.NeurIPS 2020 · 64,255 citations
  2. BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language ModelsJunnan Li, Dongxu Li, Silvio Savarese, Steven C. H. HoiICML 2023 · 7,873 citations
  3. InstructBLIP: Towards General-purpose Vision-Language Models with Instruction TuningWenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong et al.NeurIPS 2023 · 4,013 citations
  4. Locating and Editing Factual Associations in GPTKevin Meng, David Bau, Alex Andonian, Yonatan BelinkovNeurIPS 2022 · 3,415 citations
  5. MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language ModelsDeyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li et al.ICLR 2024 · 3,079 citations

Related papers

  1. MC-MKE: A Fine-Grained Multimodal Knowledge Editing Benchmark Emphasizing Modality ConsistencyJunzhe Zhang, Huixuan Zhang, Xunjian Yin, Baizhou Huang et al.ACL Findings 2025 · 3 citations
  2. Visual-Oriented Fine-Grained Knowledge Editing for MultiModal Large Language ModelsZhen Zeng, Leijiang Gu, Xun Yang, Zhangling Duan et al.ICCV 2025 · 3 citations
  3. MultiMedBench: A Scenario-Aware Benchmark for Evaluating Knowledge Editing in Medical VQAShengtao Wen, Haodong Chen, Yadong Wang, Zhongying Pan et al.AAAI 2026
  4. Can Knowledge be Transferred from Unimodal to Multimodal? Investigating the Transitivity of Multimodal Knowledge EditingLingyong Fang, Xinzhong Wang, Depeng Wang, Zongru Wu et al.ICCV 2025 · 4 citations
  5. MIKE: A New Benchmark for Fine-grained Multimodal Entity Knowledge EditingJiaqi Li, Miaozeng Du, Chuanyi Zhang, Yongrui Chen et al.ACL Findings 2024 · 5 citations
Dusk over the sea between two cliffs drawn in fine vertical lines