Lune

ACL Findings2024

Trust in Internal or External Knowledge? Generative Multi-Modal Entity Linking with Knowledge Retriever

Xinwei Long, Jiali Zeng, Fandong Meng, Jie Zhou, Bowen Zhou

Abstract

Multi-modal entity linking (MEL) is a challenging task that requires accurate prediction of entities within extensive search spaces, utilizing multi-modal contexts. Existing generative approaches struggle with the knowledge gap between visual entity information and the intrinsic parametric knowledge of LLMs. To address this knowledge gap, we introduce a novel approach called GELR, which incorporates a knowledge retriever to enhance visual entity information by leveraging external sources. Additionally, we devise a prioritization scheme that effectively handles noisy retrieval results and manages conflicts arising from the integration of external and internal knowledge. Moreover, we propose a noise-aware instruction tuning technique during training to finely adjust the model's ability to leverage retrieved information effectively. Through extensive experiments conducted on three benchmarks, our approach showcases remarkable improvements, ranging from 2.0% to 6.5%, across all evaluation metrics compared to strong baselines. These results demonstrate the effectiveness and superiority of our proposed method in tackling the complexities of multi-modal entity linking.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

3 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • Collaborative Cognitive Diagnosis with Disentangled Representation Learning for Learner Modeling cites RCD: Relation Map Driven Cognitive Diagnosis for Intelligent Education Systems
  • Multi-level Mixture of Experts for Multimodal Entity Linking cites Autoregressive Entity Retrieval
  • Multi-level Mixture of Experts for Multimodal Entity Linking cites Learning Transferable Visual Models From Natural Language Supervision
  • EventVAD: Training-Free Event-Aware Video Anomaly Detection cites Learning Transferable Visual Models From Natural Language Supervision
  • Learning Transferable Visual Models From Natural Language Supervision cites A Simple Framework for Contrastive Learning of Visual Representations
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers3

  1. EventVAD: Training-Free Event-Aware Video Anomaly DetectionYihua Shao, Haojin He, Sijie Li, Siyu Chen et al.ACM MM 2025 · 19 citations
  2. Collaborative Cognitive Diagnosis with Disentangled Representation Learning for Learner ModelingWeibo Gao, Qi Liu, Linan Yue, Fangzhou Yao et al.NeurIPS 2024 · 12 citations
  3. Multi-level Mixture of Experts for Multimodal Entity LinkingZhiwei Hu, Víctor Gutiérrez-Basulto, Zhiliang Xiang, Ru Li et al.KDD 2025 · 1 citation

Builds on23

  1. Learning Transferable Visual Models From Natural Language SupervisionAlec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh et al.ICML 2021 · 47,906 citations
  2. A Simple Framework for Contrastive Learning of Visual RepresentationsTing Chen, Simon Kornblith, Mohammad Norouzi, Geoffrey E. HintonICML 2020 · 24,064 citations
  3. LoRA: Low-Rank Adaptation of Large Language ModelsEdward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu et al.ICLR 2022 · 18,833 citations
  4. Autoregressive Entity RetrievalNicola De Cao, Gautier Izacard, Sebastian Riedel, Fabio PetroniICLR 2021 · 200 citations
  5. RCD: Relation Map Driven Cognitive Diagnosis for Intelligent Education SystemsWeibo Gao, Qi Liu, Zhenya Huang, Yu Yin et al.SIGIR 2021 · 168 citations

Related papers

  1. OpenMEL: Unsupervised Multimodal Entity Linking Using Noise-Free Expanded Queries and Global CoherenceXinyi Zhu, Yongqi Zhang, Lei ChenVLDB 2025 · 2 citations
  2. Breaking the Noise Barrier: LLM-Guided Semantic Filtering and Enhancement for Multi-Modal Entity AlignmentChenglong Lu, Chenxiao Li, Jingwei Cheng, Yongquan Ji et al.EMNLP 2025
  3. VP-MEL: Visual Prompts Guided Multimodal Entity LinkingHongze Mi, Jinyuan Li, Xuying Zhang, Haoran Cheng et al.ACL Findings 2025 · 3 citations
  4. Instructed Language Models with Retrievers Are Powerful Entity LinkersZilin Xiao, Ming Gong, Jie Wu, Xingyao Zhang et al.EMNLP 2023 · 7 citations
  5. Multi-level Matching Network for Multimodal Entity LinkingZhiwei Hu, Víctor Gutiérrez-Basulto, Ru Li, Jeff Z. PanKDD 2025 · 1 citation
Dusk over the sea between two cliffs drawn in fine vertical lines