Lune

EMNLP Findings2022

Multimodal Knowledge Learning for Named Entity Disambiguation

Dongjie Zhang, Longtao Huang

Abstract

With the popularity of online social media, massive-scale multimodal information has brought new challenges to traditional Named Entity Disambiguation (NED) tasks. Recently, Multimodal Named Entity Disambiguation (MNED) has been proposed to link ambiguous mentions with the textual and visual contexts to a predefined knowledge graph. Existing attempts usually perform MNED by annotating multimodal mentions and adding multimodal features to traditional NED models. However, these studies may suffer from 1) failing to model multimodal information at the knowledge level, and 2) lacking multimodal annotation data against the large-scale unlabeled corpus. In this paper, we explore a pioneer study on leveraging multimodal knowledge learning to address the MNED task. Specifically, we first harvest multimodal knowledge in the Meta-Learning way, which is much easier than collecting ambiguous mention corpus. Then we design a knowledge-guided transfer learning strategy to extract unified representation from different modalities. Finally, we propose an Interactive Multimodal Learning Network (IMN) to fully utilize the multimodal information on both the mention and knowledge sides. Extensive experiments conducted on two public MNED datasets demonstrate that the proposed method achieves improvements over the stateof-the-art multimodal methods.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

3 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • I2CR: Intra- and Inter-modal Collaborative Reflections for Multimodal Entity Linking cites DRIN: Dynamic Relation Interactive Network for Multimodal Entity Linking
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers3

  1. DRIN: Dynamic Relation Interactive Network for Multimodal Entity LinkingShangyu Xing, Fei Zhao, Zhen Wu, Chunhui Li et al.ACM MM 2023 · 21 citations
  2. I2CR: Intra- and Inter-modal Collaborative Reflections for Multimodal Entity LinkingZiyan Liu, Junwen Li, Kaiwen Li, Tong Ruan et al.ACM MM 2025 · 2 citations
  3. Multi-level Mixture of Experts for Multimodal Entity LinkingZhiwei Hu, Víctor Gutiérrez-Basulto, Zhiliang Xiang, Ru Li et al.KDD 2025 · 1 citation

Builds on5

  1. VideoBERT: A Joint Model for Video and Language Representation LearningChen Sun, Austin Myers, Carl Vondrick, Kevin Murphy et al.ICCV 2019 · 1,396 citations
  2. ERNIE-ViL: Knowledge Enhanced Vision-Language Representations through Scene GraphsFei Yu, Jiji Tang, Weichong Yin, Yu Sun et al.AAAI 2021 · 414 citations
  3. Progressive Semantic Matching for Video-Text RetrievalHongying Liu, Ruyi Luo, Fanhua Shang, Mantang Niu et al.ACM MM 2021 · 20 citations
  4. Evaluating Entity Disambiguation and the Role of Popularity in Retrieval-Based NLPAnthony Chen, Pallavi Gudipati, Shayne Longpre, Xiao Ling et al.ACL 2021
  5. 12-in-1: Multi-Task Vision and Language Representation LearningJiasen Lu, Vedanuj Goswami, Marcus Rohrbach, Devi Parikh et al.CVPR 2020

Related papers

  1. Multi-Grained Multimodal Interaction Network for Entity LinkingPengfei Luo, Tong Xu, Shiwei Wu, Chen Zhu et al.KDD 2023 · 21 citations
  2. Multi-level Matching Network for Multimodal Entity LinkingZhiwei Hu, Víctor Gutiérrez-Basulto, Ru Li, Jeff Z. PanKDD 2025 · 1 citation
  3. Grounded Multimodal Named Entity Recognition on Social MediaJianfei Yu, Ziyan Li, Jieming Wang, Rui XiaACL 2023 · 32 citations
  4. Multimodal Entity Linking: A New Dataset and A BaselineJingru Gan, Jinchang Luo, Haiwei Wang, Shuhui Wang et al.ACM MM 2021 · 41 citations
  5. MELOV: Multimodal Entity Linking with Optimized Visual Features in Latent SpaceXuhui Sui, Ying Zhang, Yu Zhao, Kehui Song et al.ACL Findings 2024 · 6 citations
Dusk over the sea between two cliffs drawn in fine vertical lines